Hyper-V on 2012 R2 keeps resetting VMs

[localzuk]

We have an issue with one of our Hyper-V hosts. Overnight it is resetting all the hosted VMs on the machine for what appears to be no reason.

 

I have gone though handfuls of hotfixes that apparently resolve this issue. I've updated NIC drivers, disabled VMQs (as the card is broadcom based and this has been known to cause issues like this).

 

All I get, in terms of errors, is an event 1069 for each machine saying "Cluster Resource "Virtual Machine $name" of type "Virtual Machine" in clustered role "$name" failed. Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."

 

I'm starting to pull my hair out a little over this! The storage is up and running, and the servers hosted on the other Hyper-V node don't restart, so it isn't the storage. Windows is fully up to date on the server, and the report for validating the cluster comes up clean.

 

Any ideas?

[localzuk]

Ooh, just spotted a new error message - it lost connection to the Witness share somehow. Even though it was up and running and the other server didn't lost connection to it.

[hallb15]

Clutching at straws here but are the time and date settings exactly the same on all servers/storage?

[localzuk]

Clutching at straws here but are the time and date settings exactly the same on all servers/storage?

 

Yup. All set via time server.

[hallb15]

Yup. All set via time server.

OK. Was worth checking though - I've had this before with the time on a NAS box drifting to the point where the drive maps just disappear without warning.

It will be something simple but elusive - these annoying things always are...

[localzuk]

Well, thought I'd got it cracked. Nope. Just did it again. 18 VMs all reset at once.

 

I think I'm just going to have to just kill the cluster and build again from scratch.