[Hyper-V] Poor performance of VMs on new server

[FragglePete]

We implemented our first Hyper-V server with a iSCSI SAN last year and it has been pretty good so far. The plan was to introduce an additional server this year and we done that last week; so we now have the two hosts running as a cluster with failover. Took great joy in wowing at the ability to live migrate servers back and forth and all seems good. As Hyper-V and iSCSI / SANs was a new area for me, both these setups I arranged to be installed by the suppliers and asked specifically for the same engineer who did the original install to do the install of the 2nd server which he did while patiently answering all my questions, etc.

 

However, I've noticed an issue.... the performance of the VMs on the new host is appalling. I created a new VM on the new Host and built Server 2012 on to it, when I tried copying over what was a small file from another server on the network it seem to take a long time. Tried the same file on a VM on the original Host and the file copied over straight away. Odd; so I copied over this small file again onto the new Host server itself and again the file copied over straight away no issues - but the VM hosted on this host was slow. Ok, let me just migrate the new VM to the first Host and see what happens.... this time copying over the file takes no time at all to the speed that I would expect. So, I migrate over a different VM server from the original host to the new host and try copying over the file - with the VM running on that node (ie. the new host server) the performance was terrible, with a 50mb file taking well over 10 seconds to copy over sometime stalling and taking much longer. I tried copying a larger (1Gb) file over from one server to different VMs running on each of the two Hosts simultaneously.... the VM on the old host showed a speed of 86.1 MB/s whereas the VM on the new host shows a speed of 5.54Mb/s !!!!! WTF????

 

As I said, copy the file over on the host itself and the speed is fine, just the performance of the VMs themselves on this host is terrible.

 

I've e-mailed the engineer and I'm waiting a response, but was wondering if anybody else had encountered anything like this and may offer some guidance to as what is going on!

 

Thanks

 

Pete

[andydis]

hi Pete

 

is the new hyper-v host 2012 r2 ?

how many physcial nics?

what speed are these nics?

what are each NIC plugged into ( and vlan'd?)

can you post a screenshot of your hyper-v virtual switch manager?

 

my initial suspicion is that the "allow management operating system to share this adapter" is ticked

[Steve21]

VMQs on the NICs have been causing a lot of issues for us here (including speed/disconnects) until we disabled them. (Specifically Broadcom adapters)

 

Steve

[FragglePete]

Just done a Windows Update and allowed drivers down - there was an update for the Intel NICS that I've allowed to apply and after a restart I do appear to have the correct performance on the NIC. The 4 x Intel NIC are being used for the iSCSI network to the SAN, and I also have 4 X Broadcomm NICS teamed to the Core Network switch. But, not going to celebrate just yet as I'll need to monitor it a bit more closely but the driver update may have solved it.

 

Thanks for the replies; will look into those suggestions also.

 

Pete

[FragglePete]

 

The tick box is indeed ticked as per the screenshot - but does appear to be working as expected presently. Will monitor and decide tomorrow if it's worth opening up the EduTeaBags in celebration.

 

Pete

[FragglePete]

Oh dear - just restarted the VM and again performance is slow again.

 

Pete

[andydis]

have you compared the hyper v settings from old server to new server?

 

i never allow the physcial NIC to share the adaptor , its best practice to have a dedicated NIC ( or teamed) for the host and a dedicated ( or teamed ) for the virtual switch.

but i notice you have 2 virtual switches, whats the private virtual switch used for?

[FragglePete]

Would appear that @Steve21 may have hit the nail on the head. The engineer confirmed a Bug in Broadcom adapters where VMQ needs to be disabled - initial setup by the engineer had only two adapters in the Team and I added the additional two later on but didn't know about the VMQ issue. Disabled it and things seem happier now.

 

Pete

[detjo]

I had similar issues with two 'identical' HP servers, where the server that had HP NICs was fine and the one with Broadcom NICs was ridiculously slow. Turns out the drivers installed for the Broadcom NICs were supplied by Microsoft. I downloaded drivers from Broadcom website and it fixed the problem and ran at full speed. Edited April 9, 2015 by detjo

[Steve21]

Would appear that @Steve21 may have hit the nail on the head. The engineer confirmed a Bug in Broadcom adapters where VMQ needs to be disabled - initial setup by the engineer had only two adapters in the Team and I added the additional two later on but didn't know about the VMQ issue. Disabled it and things seem happier now.

 

Pete

 

Good to hear Had the exact same issue on 2 new HOSTS that were put in just before I joined here, but even worse VMs dropping off network, refusing to talk to eachother, slow speeds etc haha

 

Seems to be major issues with the Broadcom NetXtreme (1Gigs) and HyperV that's been there for a while now, as this started for us back in July/Aug~ (8 ports in our hosts so they weren't working properly at all haha)

 

Although not exact same issues not had a single issue since disabling VMQs so hopefully all good on your end too

 

Steve

[Marshall_IT]

Does any one know if this affects intel NICs too?

 

Will it have any negative effect if i disable it on a NIC that is otherwise performing alright?

[Steve21]

Does any one know if this affects intel NICs too?

 

Will it have any negative effect if i disable it on a NIC that is otherwise performing alright?

 

The issue we had is specific to Broadcom (and only then a specific set of drivers/cards).

 

https://support.microsoft.com/en-us/kb/2986895/

 

This is a known issue with Broadcom NetXtreme 1-gigabit network adapters that use the b57nd60a.sys driver when VMQ is enabled on the network adapter. (By default, VMQ is enabled.)

 

The latest versions of the driver are 16.2 and 16.4, depending on which OEM version that you are using or whether you are using the Broadcom driver version. Broadcom designates these driver versions as 57xx-based chipsets. They include 5714, 5715, 5717, 5718, 5719, 5720, 5721, 5722, 5723, and 5780.

 

These drivers are also sold under different model numbers by some server OEMs. HP sells these drivers under model numbers NC1xx, NC3xx, and NC7xx.

 

Not had any issues on Intel yet cards that I've seen at least. But if you disable the VMQs it'll just cause additional work on the HOST for routing packets etc.

 

Steve