Failover cluster name failure

[gdrinkall]

I am currently going crazy beating my head against a brick wall with my Server 2012 failover cluster. The whole system fell over (for reasons I cannot find) in the middle of the night last weekend. Cluster service had failed, all resources offline. Managed to get the services back up and the storage back online but the cluster name is in a failed state (Ip address is online) Everything DNS looks OK, cluster name resolved OK and looks to be accessible.

 

I get 3 Event IDs in the cluster events 1254, 1205 and 1069. Under server manager the cluster name is giving me a Kerberos Security Error. In event viewer I am getting EventID4 - Security-Kerberos -The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server virthost1$. The target name used was HTTP/cluster.domain.internal.

 

I am also unable to run live migrations. If I try to live migrate I get an error saying "Virtual machine migration operation for 'VM1' failed at migration source 'VIRTHOST2'". If I take the machine offline and run a quick migration it works OK.

 

(I think my servers are getting mardy with me as Im currently costing up to replace them)

 

Any suggestions would be appreciated

 

Thanks

 

Garry Drinkall

[sister_annex]

First things I would check is Network Comms - can your hosts talk to each other over the cluster network? Do you have separate networks for Cluster Coms and LMs - if not you may get timeouts on your Cluster due to the traffic you're LM generates.

 

Are you using Broadcom NICs? There was something a while ago that I saw that some Broadcom cards can cause issues

 

If you're getting Kerberos errors i would also check the time is correct on your servers too

[sister_annex]

Oh yeah... your cluster name gets registered as a computer object in AD so I would check that's not been disabled/deleted too

 

SA

[HPlum78]

Test-computersecurechannel -verbose powershell from the nodes....

[HPlum78]

If it comes back as anything other than ............. Is in good condition then that needs fixing.

[HPlum78]

Oh and if that ain't the issue dump out the cluster logs.... Get-clusterlog -destination . (I think!)

[gdrinkall]

Thanks for the replies.

 

Comms is working between hosts on the cluster network and the LM network. Time and date is accurate on both hosts.

They are broadcom NICs so will look at that but they have been working fine for 5 years and nothing has changed, not even any windows updates.

AD account is still there and active and secure channel came back as good.

 

Just tried another live migration and got the following event details

"The Virtual Machine Management Service failed to establish a connection for a Virtual Machine migration with host 'VIRTHOST1': The logon attempt failed (0x8009030C). The Virtual Machine Management Service failed to authenticate the connection for a Virtual Machine migration at the source host: The logon attempt failed (0x8009030C)."

[HPlum78]

So my understanding is that there are 3 things that can cause that error, duplicate IP's, duplicate DNS records or using the wrong network to do the migrations.

[HPlum78]

Just stumbled across this https://charbelnemnom.com/2016/03/virtual-machine-migration-operation-failed-to-authenticate-the-connection-at-the-source-host-the-target-principal-name-is-incorrect-0x80090322-hyperv/

 

Maybe worth a look see...

[gdrinkall]

Just followed this through, DNS all looks fine on both my DNS servers. No duplicate IPs showing and the only network set to allow live migrations is the live migration network

 

Im slowing going crazy

 

Garry

[PotNoodleTech]

Is this anything to do with the "RDS patch" that went on a few months ago, servers that were patched, were not allowed to connect to servers that were not patched, so if you have some servers that are not up to date you can get some weird stuff going on that may well be affecting more than just remote desktop connections?

[3s-gtech]

https://blogs.technet.microsoft.com/mckittrick/unable-to-rdp-to-virtual-machine-credssp-encryption-oracle-remediation/

[gdrinkall]

I can RDP onto all the servers fine. Quick migrations work fine

[cscc]

Can you run a cluster validation report?

[gdrinkall]

ClusterValidation.zip

 

Attached is the cluster validation report

[cscc]

https://social.technet.microsoft.com/Forums/windows/en-US/0dbec820-69cf-4bad-91fd-5ddb09c06a3b/issue-with-setting-up-a-file-server-role-error-12541205-and-1069?forum=winserverClustering

 

Similar but not exact, points to AD permissions for the named computer account. Maybe worth a look.

[3s-gtech]

The error message suggest some issue with the passwords used by the services for the cluster. You may need to reset the secure channel password used by the cluster or the computers - this doesn't necessarily mean dropping them off the domain and rejoining but that may be an option.

[gdrinkall]

Will look at trying this over the xmas break, just in case

[HPlum78]

Did you use the Get-clusterlog? (do that!) Also if you have used the Test-computersecurechannel command then this verifies the passwords being used for the computer accounts.