garethEds Posted October 24, 2017 Posted October 24, 2017 On our cluster I am getting the following error: Clustered role 'CAUGwyrCyp9' has exceeded its failover threshold. It has exhausted the configured number of failover attempts within the failover period of time allotted to it and will be left in a failed state. No additional attempts will be made to bring the role online or fail it over to another node in the cluster. Please check the events associated with the failure. After the issues causing the failure are resolved the role can be brought online manually or the cluster may attempt to bring it online again after the restart delay period. Now I didn't set this stuff up, but this error has been there for a while and it appears on both HyperV nodes. I cannot find any reference to CAUGwyrCyp9 at all - anywhere. I've looked under Roles and it's not there. There is nothing in the AD, there is nothing on either server. Yet HyperV still seems to be looking for it. The error above is preceeded by: The Cluster service failed to bring clustered role 'CAUGwyrCyp9' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered role. Where and what is it? There is another error related to it: Cluster resource 'GwyrCAU' of type 'Distributed Network Name' in clustered role 'CAUGwyrCyp9' failed.Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet Any help for this HyperV amateur would be appreciated. Thanks all Gareth
NB9457 Posted October 24, 2017 Posted October 24, 2017 It sounds like it could be an old VM. Do all the other VMs failover as they should? There's nothing in the actual HyperV console on the servers that you don't recognise is there? (Not the failover cluster manager)
HPlum78 Posted October 24, 2017 Posted October 24, 2017 I seem to remember there was an issue with this way back in server 2012 but I thought it had been resolved in R2, but I suppose it could still be an issue if the servers were originally 2012 and then in-place upgraded, I will have a dig around to see if I can find anything and if my thoughts are correct....
garethEds Posted October 24, 2017 Author Posted October 24, 2017 It sounds like it could be an old VM. Do all the other VMs failover as they should? There's nothing in the actual HyperV console on the servers that you don't recognise is there? (Not the failover cluster manager) Nothing in the HyperV consoles on either nodes. Everything looks how it should look. Gareth - - - Updated - - - CAU is Cluster Aware Updating Ok - thanks @HPlum78 - I shall take a look in there. Gareth
HPlum78 Posted October 24, 2017 Posted October 24, 2017 Can you take a look at the logs and see if just before these errors an update cycle is being initiated.
garethEds Posted October 24, 2017 Author Posted October 24, 2017 I seem to remember there was an issue with this way back in server 2012 but I thought it had been resolved in R2, but I suppose it could still be an issue if the servers were originally 2012 and then in-place upgraded, I will have a dig around to see if I can find anything and if my thoughts are correct.... The two HyperV machines were built as 2012R2. In fact I was just wondering whether upgrading them to 2016 would help but thought better of it. Gareth
HPlum78 Posted October 24, 2017 Posted October 24, 2017 https://social.technet.microsoft.com/wiki/contents/articles/13590.known-issues-windows-server-2012-failover-clustering.aspx May help.
Steve21 Posted October 24, 2017 Posted October 24, 2017 Go into failover cluster manager and check under the role name for the fist one that's the VM - CAUGwyrCyp9, and check in networks for the second one (Though it might just be a reference in the VM settings) Don't do it through HyperV GUI Steve
HPlum78 Posted October 24, 2017 Posted October 24, 2017 (edited) Screen Grab of the CAU AD computer in my Dev Environment. This is nothing to do with a VM, In AD you should have a Cluster Aware Updating Resource with that name. If that is not in AD then for what could be a number of reasons it has gone. Need to reconfigure the CAU probably will need to use PowerShell and forcibly remove the current Cluster Aware Updating Configuration and then recreate. Oh and I will add this, even if the CAU resource is in AD I would guess that some of the supporting configuration for the CAU is not there! so ether way you are going to be removing the current CAU setup and recreating it from the ground up.... Edited October 24, 2017 by HPlum78
Steve21 Posted October 24, 2017 Posted October 24, 2017 Yes Which should show under the Roles/VM list too. But going back a step here, you used to have CAUGwyrCyp9 and CAUGwyrCyp93DQU in AD already, did you delete them? If you run get-Cauclusterrole in powershell what's it return? Steve
HPlum78 Posted October 24, 2017 Posted October 24, 2017 Sorry @Steve21 miss read what you put there and yes you are correct.
garethEds Posted October 24, 2017 Author Posted October 24, 2017 Yes Which should show under the Roles/VM list too. But going back a step here, you used to have CAUGwyrCyp9 and CAUGwyrCyp93DQU in AD already, did you delete them? If you run get-Cauclusterrole in powershell what's it return? Steve Hi @Steve21 I do have a user in AD called CAUGwyrCyp93dqu but not CAUGwyrCyp9 - don't recall every seeing one there. Would you advise setting up the cluster aware updating again? As mentioned - I didn't do this and the CAU seems to be reporting as working fine. Cheers Gareth
garethEds Posted October 24, 2017 Author Posted October 24, 2017 [ATTACH=CONFIG]45908[/ATTACH] Screen Grab of the CAU AD computer in my Dev Environment. This is nothing to do with a VM, In AD you should have a Cluster Aware Updating Resource with that name. If that is not in AD then for what could be a number of reasons it has gone. Need to reconfigure the CAU probably will need to use PowerShell and forcibly remove the current Cluster Aware Updating Configuration and then recreate. Oh and I will add this, even if the CAU resource is in AD I would guess that some of the supporting configuration for the CAU is not there! so ether way you are going to be removing the current CAU setup and recreating it from the ground up.... Ok - I've searched the whole domain and that name isn't there. Would you have a link on how to re-create the CAU Config and start it again? Makes sense to backup the VMs - which we do and then re-do the CAU setup. If you have any links then that would be great. If not - Google here I come... Thanks for all your advice, Gareth
HPlum78 Posted October 25, 2017 Posted October 25, 2017 Sorry was in meetings yesterday, did this get sorted? if not something along these lines:- Get-Cluster | Get-ClusterParameter Note down the values of any parameters (if they are there they will look like CAUGwyrCyp9/GwyrCAU) but its always worth a look. You say that the cluster resources for the CAU are already missing from AD but if not delete them. Then run the following (PowerShell needs to be running as admin) Get-Cluster | Set-ClusterParameter "CauResourceName" -Delete Get-Cluster | Set-ClusterParameter "CauVCOName" -Delete then you should be able to run the cluster aware updating setup again.
garethEds Posted October 25, 2017 Author Posted October 25, 2017 Sorry was in meetings yesterday, did this get sorted? if not something along these lines:- Get-Cluster | Get-ClusterParameter Note down the values of any parameters (if they are there they will look like CAUGwyrCyp9/GwyrCAU) but its always worth a look. You say that the cluster resources for the CAU are already missing from AD but if not delete them. Then run the following (PowerShell needs to be running as admin) Get-Cluster | Set-ClusterParameter "CauResourceName" -Delete Get-Cluster | Set-ClusterParameter "CauVCOName" -Delete then you should be able to run the cluster aware updating setup again. Hiya - and this will not break anything else? Cheers Gareth
HPlum78 Posted October 25, 2017 Posted October 25, 2017 Clearly all of the usual caveats apply here - make sure you have a backup of your current config and the like and just in case you need some background on backing up a failover cluster environment https://technet.microsoft.com/en-us/library/cc771973(v=ws.11).aspx this will help, but apart from that no you should be good to remove and recreate the CAU in your cluster using those commands. Its worth calling out that the preferred method of removing the CAU is via the remove-CauClusterRole https://technet.microsoft.com/en-us/library/hh847218(v=wps.630).aspx but I expect you are beyond the point of being able to use this command and it will throw an exception (win32...... what ever the error message is).
garethEds Posted October 25, 2017 Author Posted October 25, 2017 Thanks @HPlum78 - I'm actually quite scared of doing it in case everything buggers up. Gareth
HPlum78 Posted October 25, 2017 Posted October 25, 2017 Yeah Fail Over Clustering every time I hear that I brake out into a cold sweat! to be fair in 2012r2 it has improved and is less of a pain. Not so long ago something like this would have been cluster not starting!
garethEds Posted October 25, 2017 Author Posted October 25, 2017 The failover bit is working fine. It's just the updating we think - from the error messages. Gareth
HPlum78 Posted October 25, 2017 Posted October 25, 2017 Yeah from what you have posted that's my assessment.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now