Strange Hyper-V issue

[Jawloms]

I've done some Googling and am having difficulty finding anything one this one;

 

SpiceWorks emailed me on Saturday and told me that one of my Hyper-V (2012 R2) servers, and a server it hosts, were offline. I RDP'd in (the server which I RDP in to is hosted on the server which is apparently offline) and it worked so I just ignored it and thought I'd investigate Monday morning.

 

On coming in this morning, the host and one of the servers it hosts are completely uncontactable over the network, but the other servers it hosts are all working fine. The host has 4 NICs which are all in one team so how can the host and one server not respond to a ping, or be able to ping out either, when all the other servers hosted on it work fine? This happened around three weeks ago too and a reboot sorted it, but it's hosting SIMS so I'd rather not reboot unless I have to.

 

Any thoughts?

 

Thank you

 

Stuart

[Steve21]

Are VMQs all disabled? As that's first thing I'd always check, so so many problems caused by them.

 

Steve

[Steve21]

One random example built from our original cases: https://support.microsoft.com/en-us/kb/2986895

 

Virtual machines may randomly lose network connectivity. The network adapter seems to be working in the virtual machine. However, you cannot ping or access network resources from the virtual machine. Restarting the virtual machine does not resolve the issue.

 

You cannot ping or connect to a virtual machine from a remote computer.

 

It happens on a lot of cards even not Broadcom too

 

Steve

[localzuk]

Yup, I've had this issue too. Disabling VMQs made a huge difference but hasn't completely resolved it. Updating NIC driver software helped a bit too.

[sted]

Yup, I've had this issue too. Disabling VMQs made a huge difference but hasn't completely resolved it. Updating NIC driver software helped a bit too.

 

what are they even supposed to do all they seem to do is slow network cards to wet string speed

[Steve21]

what are they even supposed to do all they seem to do is slow network cards to wet string speed

 

On 1GB cards nothing at all As it's disabled by default. 10GB enabled.

 

But it's supposed to make routing etc on them more effecient

 

Steve

[sted]

On 1GB cards nothing at all As it's disabled by default. 10GB enabled.

 

But it's supposed to make routing etc on them more effecient

 

Steve

its not disabled by default at least not on any 2012 install ive seen

[Steve21]

its not disabled by default at least not on any 2012 install ive seen

 

I'll rephrase. VMQ is enabled on card, but not enabled to do anything while on card.

 

Unless you have BelowTenGigVmqEnabled enabled in the registry it doesn't work on cards below 10GB even if enabled on the card.

 

Unless it's changed, but certainly never did

 

VMQ and 1G NICs

 

The second issue that is reported frequently is the implementation of VMQ on 1G NICs. By default, we do not enable VMQ on 1G NICs because a single processor is usually more than sufficient to handle the networking traffic generated. If your workload requires that you use VMQ on a 1G card you will need to enable it by setting a registry key.

 

Steve

[Jawloms]

The registry key doesn't exist so I'm assuming that VMQs are not enabled as they are all 1gig cards. Also, the driver version is the newest which is supposed to mend this problem, and the problem exists on the host as well as one of the hosted servers and these articles all talk about this affecting the hosted servers. None of them mention the host.

 

Thank you so far.

 

Stuart

[Steve21]

The registry key doesn't exist so I'm assuming that VMQs are not enabled as they are all 1gig cards. Also, the driver version is the newest which is supposed to mend this problem, and the problem exists on the host as well as one of the hosted servers and these articles all talk about this affecting the hosted servers. None of them mention the host.

 

Thank you so far.

 

Stuart

 

That key isn't linked to your part Jawloms, you'd need to do it on the card properties, as it's still causing problems overall, and will affect HOST too. Always disable them first as the test I'd say

 

Steve

[Jawloms]

Checked the properties of all four NICs, and it's disabled on them all.

[Steve21]

On the team too?

 

Steve

[Jawloms]

Ah, on the team's properties I have "Virtual Machine Queues", "Virtual Machine Queues - Shared M...." and "Virtual Machine Queues - VLAN Id...." and they are all enabled. Do I disable on all three, and if I do will it make anything blip on the network as if it will I'll do it at break when less people are using things hosted on that server?

 

Thanks again.

[Steve21]

"Virtual Machine Queues" should do it (we left others enabled as by Microsoft's advice as shouldn't do anything anyway).

 

For us it took the host offline for a split second as it's updating the NIC/Team details so loses connection, so would advise waiting and trying it at least Could be a few other things but certainly would test that first and see as some of the others are hotfixes and obviously better not to hotfix unless needed haha

 

Steve

[Jawloms]

Brilliant, thanks a lot. I'll give it a go in a bit.

[Jawloms]

I disabled it and it made no difference, so I ran the connectivity troubleshooter to reset stuff and it still didn't work. I've now rebooted the server as I need it to work and that has solved it. I'll just have to see if it does it again in the future. Thanks for all your help though.

[localzuk]

Another thing that helped - disabling anti-virus on-access scanning on the Hyper-V nodes, and on the storage drive for the virtual hard drives on the storage server.

[localzuk]

Well, after a period of quiet, our servers have started doing this again. I have a question - are you using Veeam as your backup software?

[Jawloms]

Yes, the free one.

[PR-UK]

I've had this problem since rebuilding our Hyper-V cluster over the summer holidays. Tried disabling VMQs, removing TCP offloading, and other NIC hardware acceleration stuff, none of it seemed to fix it. The only VMs we had problems with were our email server and our file guest cluster, other VMs on the same hosts worked without issue. The file and email servers are the most heavily utilised servers on the network and 75% of the time it would happen when the backup server was taking it's hourly backup but not always.

 

Once the connection is down nothing can restore it, if you disable the VM adapter it will become stuck and couldn't be re-enabled. If we did a live migration of the VM to another Host the connection was restored, until the next time it happened.

 

A few weeks ago we went through the networking again and enabled Flow Control on our switches and server NICs, that seemed to improve things. However we had another issue with the file cluster last week so I went in to our backup software DPM and turned off all the QOS throttling. It's now been over a week since we had an issue with the file cluster and nearly 3 weeks since we had an issue with the email server.

 

I'm still not convinced we've fixed it but we will have to wait and see...

 

Another potential cause was mentioned on this site: https://blog.workinghardinit.work/2015/09/18/the-mysterious-case-of-infrequent-network-connectivity-issues-on-2-hyper-v-vms-out-of-40-guests/ all of our VMs were configured as Access VLAN 0 so I set them to Untagged, but again I'm not sure if this really made any difference or not.

[localzuk]

As far as I can tell, it appears to be down to load. Our restarts are now back, and are like clockwork - always when a backup is running.

 

When Xmas holidays arrive, I'm taking the cluster down and updating as much of it to 2012 R2 as possible, to ensure all the servers are communicating with the latest SMB protocol.

 

After that, the only other option is to see if using iscsi would be a better, if more complex, option.

[PR-UK]

We rebuilt our entire domain over the summer holidays, all our servers are 2012 R2, our storage is a multi-node SAN connected via iSCSI.

 

On our file guest cluster it's only the main production adapter (which is based on two nics teamed on the host) that would disconnect, the separate virtual adapter for the heartbeat (which is based on a single nic on the host) was unaffected.

 

We did look at our old Hyper-V cluster and compare any differences and when it came down to the production adapter the only difference I could see was that under Get-VMSwitch | fl the BandwidthReservationMode was set to 'Absolute' on the old servers and 'Weight' on the new ones, I'm not entirely sure what this that means though...

[Steve21]

As far as I can tell, it appears to be down to load. Our restarts are now back, and are like clockwork - always when a backup is running.

 

When Xmas holidays arrive, I'm taking the cluster down and updating as much of it to 2012 R2 as possible, to ensure all the servers are communicating with the latest SMB protocol.

 

After that, the only other option is to see if using iscsi would be a better, if more complex, option.

 

Have you tried any of the hotfixes in regards to the restart? Assuming you're clustered via SMB from your comment?

 

e.g.

https://support.microsoft.com/en-us/kb/3072380

https://support.microsoft.com/en-us/kb/3068445

 

Steve

[localzuk]

Yup, both installed.

 

I've noticed today that only one of our Hyper-V nodes has suffered with this at all for months. So, I've drained that node for now, and am working through to see what the differences between it and the working one are.

 

If I find a smoking gun, I'll report back!

[Jawloms]

This has just done it's strange thing again. The host is completely uncontactable over the network and so is one of the servers it is hosting. The other five it hosts are all working perfectly. If the host isn't contactable over the network, why and how are the hosted working?