Laggy VMs

[Gongalong]

Hi folks,

 

We had our cluster rebuilt with Server 2019 by a 3rd party consultancy (on Proliant DL380 Gen 9 servers). As part of that they rebuilt our VM DCs.

 

Unlike the previous DCs, the rebuilt servers are just slow to use e.g. opening Active Directory is just laggy, along with using it. The VMs have plenty of resources and don't appear to be overloaded i.e. low CPU use, memory not maxing out.

 

We have some legacy VMs that were migrated across e.g. file servers, mostly Server 2016, and they're fine.

 

Any ideas what could be causing the laggy behaviour?

 

Thanks

[FishCustard]

If you create a new (non-migrated) VM - not a DC, just a regular Server 2019 install - does that lag?

[Gongalong]

I'll give it a go, although these DCs were built from scratch, not migrated (sorry, might not have been clear about that!). The old 2016 DCs were demoted and removed.

[mgs1990]

This could be the network drivers, The HPE Embedded adapters are Broadcom based. which are known to have latency issues when the VMQ feature is enabled. i usually turn it off if a drive update doesn't fix it or im pressed for time.

[mavhc]

Don't pay the consultant and complain to them until they fix it, that's the entire reason to use a consultant.

[Gongalong]

@mavhc They've been paid, and I'm flagging it up with them, but I wanted to get some other views as to what might be causing it.

[FishCustard]

I'll give it a go, although these DCs were built from scratch, not migrated (sorry, might not have been clear about that!). The old 2016 DCs were demoted and removed.

 

Oh I got that - I just want to see if it's the fact they're new VMs/Server 2019 VMs or domain controllers that's causing the lag.

[Domino]

How are the VMs sized compared to the underlying hardware?

[Gongalong]

Not sure what you mean by "sized"? Virtual hardware spec?

[Domino]

Yep, what do the VMs have, and whats available for them to run on with regards to CPU and RAM

[Gongalong]

The cluster is two DL380 Gen 9s, each have 2 * Intel Xeon E5-2667 v3 3.2GHz. 192GB RAM. There are around 15 VMs running on the cluster, typically split across both hosts. (This is Hyper-V, just in case that wasn't clear)

 

The DC I use most and that feels slow* has 8 processors and 4GB of RAM. I have just noticed that RAM usage on this as 85%, so I'm going to up it to 8GB and see if it makes any difference.

 

*Having just used it a bit, it's things like the start menu, searching in the start menu, opening up AD.

[mavhc]

If you help them find a solution ask for a discount. Are they all SSD? Monitor the disk queue, ram usage, cpu usage on the host for starters.

[Domino]

So that single VM is using one whole processors worth of Physical cores?

 

Is it just a DC? I'd at least reduce it to 4, and honestly if it's just a DC consider 2.

 

How are the other VMs specced? It could just be contention. For every VM to be able to occupy CPU slots simultaneously there needs to be enough physical CPU Cores to accommodate each VM.

 

The more they fight over resource the worse it'll be - these VMs are all in an ecosystem so just right-sizing a single VM won't bring you the full picture (and the lower CPU utilization you'll see - they literally can't work enough at the same time to take advantage).

 

As a rule of thumb start low and work up till performance is acceptable. We start at either two or four vCPUs and 4/8 GB of RAM. One of the great thing about VMs is you can adjust their allocation if needed.

[Gongalong]

If you help them find a solution ask for a discount. Are they all SSD? Monitor the disk queue, ram usage, cpu usage on the host for starters.

No SSDs, the cluster uses an MSA 2040 SAN with RAID10 and 17 * 1.2TB 10k SAS drives.

[Domino]

Here's a couple of other threads on the subject (VMware based, but the concept remains the same)

 

http://www.edugeek.net/forums/enterprise-software/185012-want-compare-my-esxi-ram-allocation-others.html

http://www.edugeek.net/forums/enterprise-software/198522-vmware-esx-server-cpu-resource-allocations-confused.html

[Gongalong]

So that single VM is using one whole processors worth of Physical cores?

Not quite sure what you mean. The hosts each have 32 cores.

Is it just a DC? I'd at least reduce it to 4, and honestly if it's just a DC consider 2.

No, it also runs the NetSupport DNA software. CPU load is definitely low, it's just the RAM usage that's in question on a revisit.

How are the other VMs specced? It could just be contention. For every VM to be able to occupy CPU slots simultaneously there needs to be enough physical CPU Cores to accommodate each VM.

Mostly specced lower than this. Only our SIMS server has a higher spec, as requested by County.

[Domino]

Not quite sure what you mean. The hosts each have 32 cores.

 

No, it also runs the NetSupport DNA software. CPU load is definitely low, it's just the RAM usage that's in question on a revisit.

 

Mostly specced lower than this. Only our SIMS server has a higher spec, as requested by County.

 

You said each host had 2x e5-2667 v3? That's an 8 core processor. it has 32 cores if you take hyperthreading into account - but you can't really utilize those for full performance VM Cores.

[Gongalong]

Ah OK, I'm just going by the cores I see in Task Manager.

[Domino]

I'd definitely try it with 4 vCPUs - it'll be much easier for the host to schedule onto the physical hardware, and faster scheduling = snappier response.

[mavhc]

HDDs are for photos and videos, who's speccing hdd based systems in 2019? Also same question about SANs.

 

1. Is the physical server overloaded at all, disk queue, network/san queue.

2. How are the resources of the physical server distributed, how many Virtual CPUs for each, how many total? How much ram each, how much total?

[FishCustard]

I'd definitely try it with 4 vCPUs - it'll be much easier for the host to schedule onto the physical hardware, and faster scheduling = snappier response.

 

This^

 

It's not the case that more vCPUs = better. It's aaaaall about scheduling.

[foofighterjim]

Have you recently updated DNA to 4.75? If so I believe there may be a slight issue with it. Our application server started dropping the DNA server service after the update and the whole server was lagging slightly, NetSupport have been on today to change one of the .dll's and things have been better this afternoon.

[Gongalong]

HDDs are for photos and videos, who's speccing hdd based systems in 2019? Also same question about SANs.

The servers themselves are 4 years old, it's just that they've been rebuilt over the Summer with Server 2019.

1. Is the physical server overloaded at all, disk queue, network/san queue.

RAM was potentially the only issue - it is running SQL Server with NetSupport DNA - so I've upped that to 8GB. Currently it's sat at 40% usage for RAM.

2. How are the resources of the physical server distributed, how many Virtual CPUs for each, how many total? How much ram each, how much total?

We have around 15-20 VMs. Most only have 2 cores assigned, with the exception of the SIMS server that has 8, and our heavy usage file server that has 8 (although sounds like 4 might be better). Total RAM usage is around 200GB across all VMs, but we normally run half the servers on each host (each have 192GB of RAM).

Have you recently updated DNA to 4.75? If so I believe there may be a slight issue with it. Our application server started dropping the DNA server service after the update and the whole server was lagging slightly, NetSupport have been on today to change one of the .dll's and things have been better this afternoon.

Only just done that, but it was bad before 4.75, and bad ever since the 2019 update.

 

It is much better since the RAM increase, and change to 4 cores.

[mavhc]

Do you have more vCPU cores assigned than there are physical cores in total? (And does that include hyperthreading fake cores?)

 

I meant total virtual ram assigned vs total physical ram in the physical server

[Smokebomb]

Check these links out. The default scheduler has changed in Server 2019 Hyper-V and this may have an impact on any existing VM's that are not upgraded to the latest VM version:

https://kevinholman.com/2019/07/22/windows-server-2019-hyper-v-might-only-use-half-the-available-cpus/

https://www.altaro.com/hyper-v/hyper-v-core-scheduler/

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types