Major issue with a Hyper V Server

[abillybob]

Hello All

 

I've been working on this all week staying until sometimes 1am trying to sort it all out and as soon as I think I'm getting somewhere it appears now.

 

We basically have 2 Hyper V Servers that replicate to each other. They run 5 Virtualised Servers, Terminal Server, Another Terminal Server, SQL Server, A clocking in Server and a WSUS/MDT Server.

 

For some strange reason this week the SQL and servers running on HyperV - B are running intermittently slow, at some stages they're really fast and then it'll just run un-bearably slow again. At first I thought it was a recourse problem so I moved over all the other servers to HyperV - A with just the SQL running on HyperV - B but still the problem persists.

 

I've looked through event viewer and get a lot of instances of this

 

 

along with this

 

 

So I thought maybe it's a problem with the RAID but Intel Rapid Storage isn't showing me anything to be worried about and looking on the server itself doesn't show any red flashing lights to say a HDD has gone bang.

 

I'm in the process of moving over the SQL server to HyperV - A and see if that helps at all but I'm not convinced. I really don't know why this is having problems, I've stopped backups just in case it was this also.

 

Can anyone help me in this time of need, a lot of pressure is being put on me as I'm going away tomorrow on holiday and need to ensure it's all working before I go but I really don't know what else to try or do. Chdsk also doesn't report any errors on the HDD's.

 

I'm sure I'll be here until early hours of the morning again so if anyone can help I'd really appreciate it!

[kmount]

Hi,

 

I presume you only see these messages in the event logs on one of the two servers? (i.e. server B)

 

Moving the SQL Server onto A is a good call, if the errors go away you know Server B does have a problem.

 

I presume they don't share storage etc if you are replicating them in a pair so no reason to believe a common shared item is the culprit.

 

See how it goes on A, if still slow - look for those errors to see if they have followed.

[sluggster66]

Out of interest what types of NICs are you using?

[yac2016]

We had an issue with NIC's on Hyper-V Broadcoms I believe, driver issues.

[sluggster66]

We had an issue with NIC's on Hyper-V Broadcoms I believe, driver issues.

 

Yep same here, had to change a setting on the Nic

[nathan3388]

have you tried disabling the virtual machine queue on the hyper v Network card that was a major issue with us with performance issues on the network.

[PotNoodleTech]

Intel Rapid Storage? What kind of specification server is this?

[localzuk]

My first thought with this would be backup software - what backup software are you using? We experienced something like this with BackupAssist a while back. Veeam has done it too though.

[MartinT]

Try pausing all replication and see how the performance changes, if at all?

[Oaktech]

+1 for the broadcom issue.

 

https://www.dell.com/support/article/uk/en/ukbsdt1/sln132131/windows-server-slow-network-performance-on-hyper-v-virtual-machines-with-virtual-machine-queue-vmq-enabled?lang=en

[HPlum78]

My initial thought was around spectre/ meltdown patching, have you applied the relevant updates to your SQL environment? has this also been done on the hyper v hosts and VMs? as well as being enabled?

[abillybob]

Sorry all for the late reply but I appreciate all your help I went to Barcelona last week but was able to sort it out last Monday.

 

So a while back we had a problem where the SQL server was full on it's hard drive so I needed to allocate some space but in order to do this I had to move around some storage from other servers and shrink their hard drives before I could successfully add more storage to the SQL.

 

It seems that for some reason after doing this it was this that caused the problem, I deleted the servers manually from HyperV A and re-replicated each server to Hyper V A. Started it back up and they've been really quick ever since. I need to now do the same on Hyper V B. Seems it somehow effected the performance of the Virtual Hard Drives and deleting/re-creating them fixed the issue.