SIMS.net - at my wits end with performance problems

[ForgotMyPassword]

Hi all,

 

We have a brand new Dell Hypervisor running only one VM, the SIMS server. When it was commissioned, it was completely installed from new, not done via a P->V process. The specs are excellent and the all the server metrics indicate it never breaks a sweat! Plenty of RAM (18Gb)

 

However SIMS is painfully slow at points - for example opening a students' SEN details takes 40 seconds, this is killing our admin team

 

So, what I've tried:

 

1). The server was clustered, we've moved it out onto its own hypervisor as above

2). Moved the VHD to it's own RAID array (RAID10 SAS discs)

3). Applied the reindex patch 14265

4). Updated the OS (Server 2012R2) and SQL (2012 full) fully

5). Database Diagnostics in SIMS

6). Validate Memberships

7). Archived attendance marks

8). Capita have been on site for a 'health check' - no issues found

9). ShrinkDBLog run on the LDFs, they are now small (MDF is about 5Gb)

 

Lastly I should point out the issues are the same whether run from the application on the server, a top of the line i7 machine connected to the core switch or one of the older admin machines further away. No different at any times of day either.

 

We are running Autumn 2.

 

So, as I said before the server resource monitor indicates it's NEVER busy, likewise SQL Activity Monitor - nothing seems to be causing it stress yet the problems persist.

 

Anyone any ideas? I'm 99% certain we are looking at a SIMS issue as opposed to hardware/network but I feel I've exhausted all ideas so far!

 

Local SIMS support haven't been able to help much further than sending the usual articles, all of which have been worked through.

 

Thanks,

James

Edited March 28, 2015 by ForgotMyPassword

[IrritableTech]

Have you limited how much ram SQL can use? There are cases of SQL hogging all the ram leaving nothing for the system.

 

It could just be sims though - it's not the most efficient set of programs.

[Scotbynature]

We had a similar problem a few weeks ago, we updated our server to 2012 r2 and SQL 2014 and SIMS basically ground to a halt. Almost a minute to log on, constantly unresponsive when teachers wanted to take their register. The sql service was hogging between 99 and 100 % of the cpu almost constantly. Our support team found a patch had been released from capita and that seemed to fix it. Something about making it run in comparability mode. Not sure if that will help or not but worth a shot.

[IrritableTech]

Also what compatibility level is the sql server on?

[3s-gtech]

I shouldn't be any issue with your spec or broad configuration issue. We have a similar setup - single VM on its own hypervisor (it was P2V'd though). Host has 32GB of RAM, VM allocated 30GB, SQL Server given hard allocation of 26GB (it'll load as much into RAM as possible). It runs fine for us - some expected slowness with very large reports but general operation fine (Server 2008 R2, SQL 2012).

 

I'm guessing you have some nice fat network connections into it?

It's worth checking your SIMS.ini and connect.ini files to double check that the configuration is optimal, combined with a scour through your DNS to check everything is perfect (long shot here).

 

First time I built a virtualised SIMS server it behaved like yours, was very poor performance wise but I couldn't isolate the reason (P2V'd on Server 2003). Built it fresh on 2008 R2, and later P2V'd that but without the issues. Could never isolate the cause. From my experience (and due to how exasperated you sound) it may be worth spinning up a new VM (if you have the free resources), copying the DB onto it, and testing it from there.

[MatthewL]

Not a user of SIMS but used other systems in similar process before and had similar issues, was caused by the AV and needed certain paths and file types in the exceptions. Worth looking at that, disable it first and see if it makes any difference.

[ForgotMyPassword]

At the moment it's limited to 10Gb (having played with different values) but I'm not seeing ANYTHING like the amount of RAM usage via resource manager that I would expect. SQL seems to use a couple of gig of system RAM at most, not like my previous experience where it gobbled up as much as it felt like!

 

EDIT: is anyone setting a minimum value? I'm not at present.

Edited March 28, 2015 by ForgotMyPassword

[ForgotMyPassword]

Thanks - that's quite a well known issue with 2014 but we are running 2012 (compatibility mode 2012)

 

Like I say, there's not a single metric or graph that shows ANYTHING struggling - in fact the whole server is never stressed, by SQL or anything else.

 

Antivirus is a good call but I've even tried removing it completely and it made no odds. I've excluded the SQL and SIMS folders and processes.

 

Thanks for all the help so far folks!

[eddyc]

I found after a migration that SIMS was slow and there were two main issues;

 

a) SQL 2014 as mentioned above which was resolved by compatibility mode.

b) SIMS.ini in C:\Windows and all of the users home folders in a Windows folder was pointing to the wrong server. I just deleted the file from all of these locations.

 

Made a massive difference and things went from minutes to seconds.

[FN-GM]

Try deleting the "My SIMS Documents" in the My Documents folder (Or home area depending on your folder redirection). Solved performance issues for myself in the past.

[ForgotMyPassword]

We have the 'Clear my documents folder on log off' flag set in User Options but I will give that a go in case!

[ForgotMyPassword]

Nope. Made no difference

[jinnantonnixx]

If you're on SQL 2012, the compatability 'feature' won't be a problem for you, so ignore that.

 

Check out @MartinT 's discovery:

http://www.edugeek.net/forums/mis-systems/147490-sims-slow-after-upgrade-sql-2014-a-2.html#post1289447

 

I've had some success by rebuilding the indexes (not the SIMS reindex patch, which doesn't update the stats).

http://www.edugeek.net/forums/mis-systems/128123-sims-gradually-getting-slower-slower.html#post1097603

Edited March 28, 2015 by jinnantonnixx

[FN-GM]

We have the 'Clear my documents folder on log off' flag set in User Options but I will give that a go in case!

 

Try doing it manually by deleting the folder in the users documents.

[ForgotMyPassword]

Yep, tried deleting the folder too - no difference

 

I'm looking into the rebuilding indexes stuff now.

[ForgotMyPassword]

So I created a maintenance plan tonight to reindex via SQL Management aaaaaaaaand? No better.

 

I've also tried:

1). Disabling NUMA Spanning (no different)

2). Reducing the amount of CPU Cores and RAM (no different)

 

Gah.

[3s-gtech]

Tried another VM to host it? I know it's work, but may help to isolate the issues.

[ForgotMyPassword]

That's today. It'll have to be on the same hypervisoras the clustered ones are pretty full but will try a fresh VM.

 

I don't have a SIMS migration tool so need to try and find a guide for how to move from somewhere.

[jinnantonnixx]

You could try IoMeter against your OS, data and log drives to see what sort of throughput your system is offering.

Iometer project

[vikpaw]

Is it the same for an admin user on the client? What about the main admin / a domain admin account?

 

What are your clients running? Win 7? Okay performance there. Client performance issues can manifest during intensive tasks such as opening and using SIMS, even though you may not hit any peaks.

My machine frequently dies because the svchost.exe is killing my machine, and it always seems to be when i'm doing MIS tasks which require more resources.

 

As mentioned by others I'd defo check AV issues on both client and server. Firewall could also be an issue.

 

If you're setting up a new test server, you'll find a few threads here, basically, you could do a rough and ready one by restoring your database to a test sql setup and copying over the sims folder. i think that's enough for you to run the client install on the same box and give it a go. I wouldn't use that for anything more than an immediate test. If it doesn't work you'll need to install SIMS using the ISO / DVD, on to a test box (VM) then restore a backup to it.

[jinnantonnixx]

You can highlight (or discount) CPU pressure by checking the 'runnable_tasks_count'.

A task is 'runnable' when it's waiting for a share of time on the scheduler. In other words, it's queued up waiting for some CPU time.

 

Ideally, the runnable_tasks_count should be 0 for each sceduler. It's not the end of the world if you see low figures for busy bursts, but constant non-zero values is a sign of CPU pressure.

 

Note that this query is a snapshot. Run it several times, noting your results, to get a more general picture.

 

SELECT scheduler_id, current_tasks_count, runnable_tasks_count
FROM sys.dm_os_schedulers
WHERE scheduler_id < 255

Edited March 30, 2015 by jinnantonnixx

[DMcCoy]

Power management on the server set to full power? HV host/vm set to performance in windows?

[badtooth]

Did you get anywhere - having very similar problems myself on a VM with similar setup (albeit ESXi 5.5).

[ForgotMyPassword]

Yep! Fixed!

 

I moved the SIMS storage out of the cluster and onto a local RAID array with SSDs

 

Now we are back in business and it takes 5-10 seconds to access parts of SIKS that took upwards of a minute.

 

I still feel it could be faster but it's much more useable.