[sims] Database errors and slow performance

[NickDay85]

Hi all,

 

Is anyone else experiencing issues with database timeout errors and very slow performance in SIMS since the summer release?

 

Doing tasks which would seem simple (admitting year 7s, adding photos etc). Are now causing the client to hang before throwing them out with errors.

 

Any advice would be greatly appreciated.

 

Many thanks

Nick

[PhilNeal]

Did you upgrade to SQL2014 at the same time?

[NickDay85]

No, still running SQL2012

[andyheth]

I too have been having untold amounts of issues - I wasn't sure if it was something our end or not but am leaning towards the Summer update being the culprit.......we have had issues with Cover not displaying for certain days - crashes with Transaction Roll back . Single Sign on not working and then the next day it works. Staff complaining SIMs is slow when it never used to be. I wonder if other institutions are experiencing the same issues?

[ticker]

We are having issues like this, just running a basic report its bringing the whole system to a grinding halt. It stopped staff registering in PARS why I don't know. We have done all the usual methods people have recommend on here, past it up to our SIMS support company.

[PhilNeal]

Please log cases. We have not been getting feedback that the summer release has performance problems.

[minimoo]

mm, I was thinking this evening that sims was being slow adding classes to templates for marksheets. Whilst I would not say I've seen issues as severe as described above, nor had staff reporting slownesss - the regular users (office staff) have definitely reported seeing more crashes recently.

 

We applied the summer release early, and shortly afterwards I logged 1507-3354788 when sims was crashing - and got told to run the DB maintenance patch.

 

I logged 1508-3368857 with output from windbg showing a handle leak (sims.net leaked 10,000 handles, hit the .net limit of 10,000 handles and then crashes) if left alone for a period of time. That case is currently needing me to send a copy of our sql db off - bit surprised as i would have thought the windbg output of the leaked objects would be more useful to the developer on its own. I've received this one and a 2 other types of crashes about about 10 times in the last 2 weeks of coming back.

@PhilNeal In terms of 'crashes', do you feel they are too rare, or what about using the solus3 infrastructure to collect "errors"? i.e. when sims crashes locally, store stack trace (maybe not crashdump as that might contain personal data) - upload that locally into solus3. Use the guid/stack trace there to report back the areas/errors that schools are receiving, and number of times.

[matt40k]

They should already have the crash dumps - they are signed up and registered with Microsoft so when it crashes and sends a crash report to Microsoft, Capita can log into Microsoft portal and download them, wheither anyone does however - interestingly Microsoft is saying SIMS .net compatible with Windows 10 - https://www.microsoft.com/en-us/windows/compatibility/CompatCenter/ProductDetailsViewer?Type=Software&Name=SIMS&ModelOrVersion=7&Vendor=The+Capita+Group&Locale=1033%2C2057%2C3081%2C4105%2C16393&LastSearchTerm=Capita%2BSIMS&BreadcrumbPath=Capita+SIMS&TempOsid=Windows+10

 

I would guess they need the DB just to test the fix as it doesn't happen on their training database.

[bossman]

I too have had certain admin staff (not many) having crashes which some relate to a null reference not related to an object!

 

This is from our exams officer whilst doing just basic tasks like entering ULN numbers or accessing marksheets which she used to have no problems with, we migrated during the Summer to Server 2012R2 with SQL2014 and all is patched to the latest revision,

 

Monitoring it at the moment as it is not that regular, not had any issues with speed yet even though it is virtualised (Hyper-V) 4 CPUs and 10GB RAM (only used 9.2GB to date) Login times 6 - 8 secs average 150 clients.

Edited September 17, 2015 by bossman

[tech-man]

I gave up with a FMS speed issue with support.

[minimoo]

They should already have the crash dumps -[/url]

 

If it's kicking in their custom exception error - it's not "crashing" as such, so unless they've got something clever going on, I doubt MS get the report. ?

[PhilNeal]

I don't think we do get crash reports through automatically so please report them if they are affecting confidence.

 

In the case of our newer software Agora, Teacher App etc we do have a means of monitoring and have an active programme of fixing issues often almost before they are noticed.

[ozydave]

Slow here too. Didn't seem to actually crash just takes a long time to do basic things. Attendance module for one

[Tefters]

mm, I was thinking this evening that sims was being slow adding classes to templates for marksheets. Whilst I would not say I've seen issues as severe as described above, nor had staff reporting slownesss - the regular users (office staff) have definitely reported seeing more crashes recently.

 

We applied the summer release early, and shortly afterwards I logged 1507-3354788 when sims was crashing - and got told to run the DB maintenance patch.

 

I logged 1508-3368857 with output from windbg showing a handle leak (sims.net leaked 10,000 handles, hit the .net limit of 10,000 handles and then crashes) if left alone for a period of time. That case is currently needing me to send a copy of our sql db off - bit surprised as i would have thought the windbg output of the leaked objects would be more useful to the developer on its own. I've received this one and a 2 other types of crashes about about 10 times in the last 2 weeks of coming back.

@PhilNeal In terms of 'crashes', do you feel they are too rare, or what about using the solus3 infrastructure to collect "errors"? i.e. when sims crashes locally, store stack trace (maybe not crashdump as that might contain personal data) - upload that locally into solus3. Use the guid/stack trace there to report back the areas/errors that schools are receiving, and number of times.

@minimoo - By any chance were your users were receiving the "Error Creating Windows Handle Win32Exception" error?

I have had a good few admin staff complaining of this recently and its happening once or twice a day and coincidentally enough only since coming back in September (SIMS Summer patch installed during Summer and now also on Autumn patch as of yesterday).

The machines they are running on are i3's with 4gb ram which is more than plenty for the tasks they do so cannot see how the application is bombing out unless there is a handles leak somewhere.

 

Unfortunately I know F'all when it comes to WinDBG and SIMS support have been no use thus far other than the usual DB maintenance tasks however the DB speed and performance is fine overall and teachers can happily skip in and out of registers etc... between classes without issue.

[RichP]

Yep, slow here too. Often reports time out, and ended up increasing the commandtimeout to ridiculous number just to get information out!

We did speed up attendance a bit by archiving old attendance marks. Turns out all attendance marks go in the same table and stay there forever. Reduced 8 million rows down to 1 million...

[mm3]

may not help anyone but I noticed deleting "My SIMS Documents" and "tempSimsRpt" with sims closed reopen sims.

 

increased our report speed.

 

we had some old redirected profiles so it may not help others

[minimoo]

@Tefters - *sigh* let me see if typing @PhilNeal can get him to get an answer on whether capita require more information from the sims core product manager... - Whilst I don't expect an immediate fix - it's a real shame that a crash bug in the summer release looks to have missed the autumn release. It shouldn't be that hard to identify if sims is the issue in this case and they came back to me last week suggesting I close google chrome as it had one of the highest handles (once the application fault occurs, the handle count drops to ~0 as it closes the main sims screen and garbage collects), so i'm not even sure if we've got to the point that a fix is in progress.

 

I'm not quite sure how capita have got the ticket I submitted set up (1508-3368857) as it shows as 'closed' on their my account website but the support guys are still updating it/asking questions, and it's linked to an problem report of PRB11652 which has status of "closed(inactive)"

 

When I submitted the original case on the 25th August, I spent the time to load up windbg, catch the ~10,000 objects and send off about 10MB's of logs, which I thought would help illustrate the problem to the developers, and allow a speedy fix for an issue that's likely to affect more customers then just me.

 

When I did this, I dumped the list of handles/objects:

 

Handles:

Strong Handles: 50

Pinned Handles: 14

Ref Count Handles: 152

Weak Long Handles: 627

Weak Short Handles: 3182

 

A few hours later:

 

54cbc988 9995 319840 System.Windows.Forms.NativeMethods+WndProc

54cbbc04 10397 582232 System.Windows.Forms.Control+ControlNativeWindow

Total 22994 objects

 

Handles:

Strong Handles: 50

Pinned Handles: 14

Ref Count Handles: 109

Weak Long Handles: 10218

Weak Short Handles: 12603

 

On a simpler level, if you watch task manager/process monitor - and add in the "user objects" and "GDI objects" column - what you will probably find is that just before it crashes the user objects has leaked to around 10,000. Mark Russinovich has a blog post @ Blogs - Mark's Blog - Site Home - TechNet Blogs has some links that basically outlines there's a 10,000 max object limit in windows. On the basis that it'll crash on it's own (if left overnight), and during the day (if admin staff using), i'm inclined to think that as the homepage is refreshed and people move around within sims, there's a few places where a object isn't getting released completely - to allow it to be garbage collected, and therefore it's creeping up to 10,000.

 

I'd be inclined to guess it's probably the home page causing it (and therefore if you were to close the initial home page you might find you don't see the issue) - and as it's new - wonder if it's got anything to do with the message/inbox changes that happened in the last release.

[minimoo]

@Tefters - just realized a really easy way to see /diagnose what's going on - especially on the basis that it's the message window - you can just click refresh

 

Step 1: Take a memory snapshot

Step 2: Hit refresh multiple times on the "my messages" window

Step 3: take a 2nd snapshot

 

If you look the below, can see the memory in "other classes" has increased:

 

 

if we do a compare between the two memory snapshots, for new objects:

 

 

Ok, so we've got 1000 more "NativeWindows" controls - which if you look at the crash you were getting you might recognise

 

So.. where did they come from:

 

 

To go any further, you'd need either the source code, or to look at the interpreted stuff that dotnet generates - which I'd expect that would break the license agreement, so not going to do that.

 

In any case, I'd take the guess that what's happening here is - the autumn release updated the my messages box to show those new icons to let you tick/categorize the messages, and there's probably a missing .Dispose() call on something within the MyMessages widget - so each time a refresh occurs, the messages get removed from the list so they are no longer visible (or even in the list), but the memory is never free'd().

 

This is something i'd hope/think would take about an hour of time to diagnose and fix - it's taken me about 15 minutes to come up with the above analysis - and in theory, we now know what object we are not cleaning up.

[ITGuyWestMidlands]

Dotpeek?

[minimoo]

screenshot's were from red-gate's tools, and yea, something like dotpeek could be used to see what's going on behind the scene's, but there's no point spending the time really

[PhilNeal]

@minimoo - it's worked, I've noticed! Have asked our architect to look at your observations.

[Steve21]

@minimoo - By any chance were your users were receiving the "Error Creating Windows Handle Win32Exception" error?.

 

Funnily enough had the same issues raised this week, as apparently it keeps crashing on those errors.

 

Hadn't had a chance to look into it yet, but guess if it's a known issue it's not my side woot woot

 

Steve

[ITGURU]

SIMS is a resource hog full stop . Whenever we've had slow issues they've always told me to run the indexing patches. Our database is getting on for 9GB.

Only solution we've really had is testing with SSDs improve things dramatically so generally it's down to workstation performance rather than Server.

 

However, we cannot change things like school dates, terms etc during the school day and this grinds everything to a holt - even though there is minimal use on the server when it's on a dual 6 core server with 24GB RAM SQL usage is minimal.

[minimoo]

Out of interest, given the 9GB size, do you ever delete anything ? (aka old students). We haven't, but i'm wondering if we should under DPA - should probably log a case for guidance as another school raised about them deleting students on the "my-account" forums.

 

We've never deleted leavers (As you need to keep them for a while, and as I've never seen Capita publish guidance on when schools should remove them, *and* there's no bulk routine to remove leavers at the moment). This must have a knock-on effect - i.e. I can raise a case that part X of sims is slow - to which sims support will give indexing patches or/and otherwise spend time diagnosing - but is the reason it's slow in the first place that they've never had time to give schools a method/guidance to tidy it up.

[ITGURU]

The majority of issues occurred when we merged with another school.

 

I don't delete anything within SIMS - my job is to ensure SIMS is running and set up new users and remove old users. I have no dealings with the modules within SIMS. There are specific members of staff who are trained in the relevant modules.