Slow SIMS server

[ozydave]

Recently moved sims to a new server 2022 on SQL 2022.

Its a VM on vmware. Its has 50gb Ram and 16 CPUs allocated & vmxnet3 Ethernet Adapter 10GB

 

Its slow, simple reports take a while. Users reported random crashes.

 

I've run the patches 20647, 22573, archived attendance marks, run database diagnostics, validated memberships.

 

Performance monitor on sims server itself runs about 30% CPU. Memory 38%

VMware monitoring coincides, the actual vm host is not under any strain at all.

 

Lots of research, iperf etc. AI suggest it could be down to the vmxnet3 Ethernet Adapter. (As below)

Apparently it might be a know issue to vmware.

 

My question is have you: Disabled Large Send Offload (LSO) & Disabled TSO (TCP segmentation offload)?

To improve performance

 

VMXNET3 with LSO enabled = known issue

VMXNET3 + Windows Server 2019/2022 + LSO/TCP Offload on ESXi 7/8 frequently causes:

• Half-speed outbound throughput
• High CPU usage in vmxnet3 virtual NIC driver
• Choppy SQL response times
• Slow SMB and slow database transfers
• Zero NIC errors (stats remain clean), which matches your report

VMware has published multiple KBs about this.

The fix is always the same:

❗ Disable Large Send Offload (LSO)

❗ Disable TSO (TCP segmentation offload)

[5tu]

It has been a while, but does SIMS still require SQL to be run in 2012 compatibility mode?  If so, what mode is your new SQL server running?

[Homer]

10 minutes ago, 5tu said:

It has been a while, but does SIMS still require SQL to be run in 2012 compatibility mode?  If so, what mode is your new SQL server running?

Yeah, it does 🤦‍♂️

[MancTech]

How does SIMS run on the server itself?  If its still sluggish then might not be network adapter related.

[ozydave]

Yep my sql is in compatibility mode.

 

@manatech it is a bit more zippy running things directly on the sims server

[3s-gtech]

How big is your SIMS DB, and have you set SQL Server RAM limits. It always used to be that you wanted the whole DB to fit in RAM, and force it to do so, otherwise it'd slow down appreciably.

[bjeffery]

23GB DB - make sure log file is not too big - also lower CPU cores - we run 7 as we found this to be optimal. 

Give SIMS 100 CPUs and it would make no difference

[mthomas08]

Ah yes, the SIMs performance problems.
I spent a lot of time troubleshooting that and you listed all the things I remember.

 

Tried to do the reports via RDP just to see how it goes?

 

Over the years we had report issues due to printers and microsoft office.

[ozydave]

@3s-gtech

Our DB is 16GB

[3s-gtech]

How much RAM is SQL Server using?

 

What type of disks are in the host?

 

Have you tried offering it part of your soul? SIMS likes souls.

[psydii]

How long has the server been up?

 

How big is TempDB?

 

What recovery mode are you running? Are the log files able to grow quickly enough?

 

Have you tried giving it a reboot?  It clears out ram, and the tempdb, which can help.

 

You might also want to look at tweaking TempDB autogrowth 

[ozydave]

53 minutes ago, 3s-gtech said:

How much RAM is SQL Server using?

 

What type of disks are in the host?

 

Have you tried offering it part of your soul? SIMS likes souls.

Its already had my soul. I am a husk of my former self

[chazzy2501]

SIMS is a pig, I found that my ESET AV was dogging it, even though it knows to leave SQL stuff alone. I found that running simple SQL queries would yield 1000ms delays.. According to the chat bot the way SIMS it queries things is super duper inefficient. Still all above anything I can do. 

 

The sims database is so small it could fit into system ram so goodness knows why it is so slow.   

[ozydave]

On 14/11/2025 at 14:26, psydii said:

How long has the server been up?

 

How big is TempDB?

 

What recovery mode are you running? Are the log files able to grow quickly enough?

 

Have you tried giving it a reboot?  It clears out ram, and the tempdb, which can help.

 

You might also want to look at tweaking TempDB autogrowth 

TempDB is 100mb, autogrowth is unlimited.

 

SIMS just locked up yesterday morning, Random people kicked out, if users were still in SIMS none of the registers would load.

Process monitor on the server showed nothing untoward. CPU sitting about 38%, RAM usage about 30%

VMware vsphere performance graphs showed ruffly the same 

 

I'm done, holding up the white flag, given up.

Handing it over the ParentPay now

 

 

[bjeffery]

did you lower the CPU threads? - try an odd number.

[ozydave]

18 minutes ago, bjeffery said:

did you lower the CPU threads? - try an odd number.

I'll do this tonight ready for the morning. Be ready for a phone call 🙂

[ozydave]

So reduced the CPUs from 16 to 10 last night. Low and behold, today no one was kicked out and overall the users reported it as being 'better'.

Going to drop it to 7 cpus tonight as @bjeffery suggested to gauge the effect.

 

 

 

[bjeffery]

I have had this with many SQL servers and reducing the CORES has always fixed this - never known why odd numbers are seem to be better - just confirmed and our SIMS DB is 23GB and i run on 7 VCPUs. We run on 55GB of RAM.

 

We use Hyper-V and read a couple of years ago about CPU queuing on VM machines and the overhead of too many cores so I never go overkill with CPU as realistically machines don't need the processing power! 

[john]

On VMWare you need to look at cpu wait / read states along with your actual physical underlying CPU structure. If you have 1 cpu with 8 cores and HT on them and giving every VM loads it'll queue up to get on a CPU to do the job so right sizing your VMs sensible can help. 

 

https://blogs.vmware.com/cloud-foundation/2017/03/09/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb/ gives very sensible direction and advice, admittedly for an old ESXi and I don't have a new version to hand to see how it still looks today I doubt the logic had changed dramatically. 

 

HyperV works and behaves differently to ESXi on vCPU in my experience. 

[psydii]

We run on a six core server. 

 

ParentPay has a support article for this (which you've probably read judging by the things you tried at the beginning of the thread) here is it for reference of others in the future: KB0012791

 

You can instead of those patches mentioned int he article (and your initial post) set up a maintenance plan: KB0013523

 

I got fed up one year and paid Microsoft to go splunking. Their recommendations were to put the database and log files on separate performant disk subsystems (but only because it is best practice - they observed that the underlying disk subsystem we had, was not breaching performance baselines) they also observed that the six cores with the 32gb of ram was not a constraint.

 

However they did feel that the indexes were suboptimal (not much I was prepared to do about that), as was  MaxDOP of 0 . Their recommendation here was to increase MaxDOP to 4 while setting "cost threshold for parallelism" to 25.

 

We passed this over to capita, who then asked for our database, did some poking and then issued some "bespoke" patches. Things have been reasonably fine since.

 

I have no idea whether they did touch MaxDOP in the end. It might be worth a go?

 

A quick google for those two settings suggests these are reasonable starting figures when optimising on a busy-ish, small-ish database/server - and that ignoring the cost threshold figure is a common cause of performance issues where the MDOP value has been tweaked.

 

EDIT: as per John's post, we had previously weighted the sql server vm to get higher priority on the vcores and make sure only a small number lightweight servers were sharing the host.

Edited November 19, 2025 by psydii

[ozydave]

So, Reduced the CPU's to 7 last night. 

Users so far report it is much zipper. No one kicked out, small reports run ok.

I plan to change MAXDOP tonight and see what tomorrow brings

 

Attached are some base lines i uploaded to Grok AI (in case anyone is interested).

 

Cheers for all your pointers 

 

SlowSIMS_Grok_report.pdf

[psydii]

Hmm. Grok has definitely been trained on emails from MS SQL Server Support team.

 

The tempdb thing... yep forgot we did that - it needs to start off big enough so you aren't continually pausing for it to grow during large queries (timetable changes and census were the common killers for us).

 

I might need to go check on that on our server, its been rebuilt since I last went performance chasing.

[psydii]

I don't mean to alarm anyone, but I just asked copilot about optimising sims.net performance and it cited this thread. Grok is also known to be a bit contrarian (like father like son), so please check your sources, people!

[MancTech]

On 20/11/2025 at 13:11, ozydave said:

So, Reduced the CPU's to 7 last night. 

Users so far report it is much zipper. No one kicked out, small reports run ok.

I plan to change MAXDOP tonight and see what tomorrow brings

 

Attached are some base lines i uploaded to Grok AI (in case anyone is interested).

 

Cheers for all your pointers 

 

SlowSIMS_Grok_report.pdf 913.96 kB · 25 downloads

 

Any update on your findings @ozydave?

[ozydave]

30 minutes ago, MancTech said:

 

Any update on your findings @ozydave?

Still very much an ongoing battle.

ParentPay had a remote session, did a bit of poking around, and suggested disabling all third-party applications like Wonde, GroupCall, Edulink, and SIMS services manager for a few days to see if that helps. They recommended enabling them one by one to identify the resource hog. I wasn’t keen on being chased around the building by angry users with pitchforks, so I didn’t follow through with that. They also weren’t too thrilled about me running SQL statements on my own SQL server, even though it’s just for reporting on what the SQL server is doing and not altering the SIMS database in any way. Oh and maybe report the server!!!!!

Wonde turned out to be a bit of a hog. I contacted them to ensure I was on a "default schedule, regular reporting." Wonde doesn’t do any writebacks for us. I’m pretty confident our server and SQL installation are in good health and configured correctly after setting MAXDOP to 4 and parallelism to 80. 7 CPUs seemed to be the sweet spot as well.

We have a SIMS report that outputs to a CSV, and it takes up to three hours to run. The resulting CSV is only 1.09 MB in size, so it’s not an extensive report by any means. Clearly, something is off here. Even when kicking off the report at midnight when everyone’s tucked up in bed, it can still take three hours.

I’m asking ParentPay if they’ll take the report and our database to see what they come up with. I’m not completely focusing on this report; it just stands out. We’ve had SIMS hangs and people getting kicked out even when this report isn’t running.