File Server Performance Woes

[Frodo_Baggins]

Bit stuck on this one, any thoughts on where to look next?

 

End user (21H1) reports latency when opening files from and navigating to network shares (When seeing the issue in person it looks they are re-indexing every time they go to a sub folder? both shares with issues are on the same server, other shares on other servers/hosts are fine.)

 

General performance on the VM is not great and laggy but not seeing high usage or spikes in resource usage when users report and performance on the host is good (seemingly no pattern in reporting time.)

 

Lots of Event ID: 1020 which relate to a delay in opening the files and most mornings this week Event ID: 1032 but didn't collect a log/dmp due to "Stalled I/O"

 

Uninstalled the last batch of Windows updates on the server and after a long restart the issue remains

 

I've got Sophos Intercept Advanced X installed but not had any issues on any other servers...

 

Found this thread but not had chance to look through it yet as it looks like a bit of a minefield Server 2019 network performance - Windows Server - Spiceworks

 

Any thoughts or rough ideas on where to look outside of moving the VM to another host?

[TwistedHelixis]

is the issue with just one user?

 

what are those shares like if the users logs in from a different laptop?

[Frodo_Baggins]

Thanks, All users on site are effected, performance is the same on both laptops and PC's.

Wont admit to being an expert on Hyper V but VMQ and RSC is enabled on the VM (can see quite a few threads online saying to turn these off on in 2019 due to performance issues)

[pantscat]

Might seem odd - but check the power plan is not set to Balanced. I've experienced disk latency issues with VMs that weren't set to "High Performance".

[foofighterjim]

Thanks, All users on site are effected, performance is the same on both laptops and PC's.

Wont admit to being an expert on Hyper V but VMQ and RSC is enabled on the VM (can see quite a few threads online saying to turn these off on in 2019 due to performance issues)

 

We had something broadly similar a few years ago and it was our AV causing the issue (or rather my poor configuration of it). Basically we had on demand file scanning active on the clients and the server policies. This was obviously fine when we were testing, but once everyone logged on and the server was scanning every file being accessed this obviously caused quite a queue for disk read / write attempts.

[Frodo_Baggins]

We had something broadly similar a few years ago and it was our AV causing the issue (or rather my poor configuration of it). Basically we had on demand file scanning active on the clients and the server policies. This was obviously fine when we were testing, but once everyone logged on and the server was scanning every file being accessed this obviously caused quite a queue for disk read / write attempts.

Coming from Sophos and having the same before I immediately thought the same, scan on read and write are turned on so will keep an eye on that - Thanks!

[Frodo_Baggins]

Might seem odd - but check the power plan is not set to Balanced. I've experienced disk latency issues with VMs that weren't set to "High Performance".

Just changed it as it was set to balanced, will monitor. Cheers!

[Frodo_Baggins]

More reports today of the same issue, will keep updating this thread in case anyone else has the same issue! Have just logged a ticket with Sophos

[Jawloms]

I had this a few years ago and it was simply the driver for the disks/RAID. It had put a default one in, when I installed HP's proper one it was fine.

[Oaktech]

Is the network card a Broadcom one? You may need to disable VMQ

[Frodo_Baggins]

Is the network card a Broadcom one? You may need to disable VMQ

Thanks, It's HP/Intel - VMQ & RCS are disabled

[Oaktech]

Thanks, It's HP/Intel - VMQ & RCS are disabled

 

https://www.intel.co.uk/content/www/uk/en/support/articles/000005593/ethernet-products.html

 

I had problems with an HP server's performance that turned out to be needing to turn off IPV4 Large Send offload, RSS and fine tuning other Offloading options. Can't remember now exactly what the combination was now!

[Frodo_Baggins]

https://www.intel.co.uk/content/www/uk/en/support/articles/000005593/ethernet-products.html

 

I had problems with an HP server's performance that turned out to be needing to turn off IPV4 Large Send offload, RSS and fine tuning other Offloading options. Can't remember now exactly what the combination was now!

Thanks will give this a look, banging my head against the wall a bit If you dont mind and still have access could you DM the settings you have on the network adapter - Just disabled IPV4 Large Send Offload (I enabled a similar setting called "Large SMB MTU" when testing)

Edited March 3, 2022 by Frodo_Baggins

[Oaktech]

Thanks will give this a look, banging my head against the wall a bit If you dont mind and still have access could you DM the settings you have on the network adapter - Just disabled IPV4 Large Send Offload (I enabled a similar setting called "Large SMB MTU" when testing)

 

Unfortunately I don't have access to that server any more... Let me check the settings on an HP VM Host I do have access to.

 

*Edit* I've just checked - the only HP I have access to has a Broadcom chipset.

Edited March 3, 2022 by Oaktech

[Frodo_Baggins]

Looks like it could be a failing disk on the san, a lot of 149 & 147 events

[Oaktech]

Any logging/errors on the SAN itself? My EMC web console gave a load of SMART warnings when one of the disks failed, but I had to log into it directly.

[Frodo_Baggins]

Any logging/errors on the SAN itself? My EMC web console gave a load of SMART warnings when one of the disks failed, but I had to log into it directly.

No warnings, only alerts for firmware updates