HyperV errors after maintance mode

[RabbieBurns]

I put a hyperV host into maintenance mode, ran some updates (including a BIOS and Firmware update for the system) however now I cannot migrate any Virtual machines back to this host.

 

The error I get is this:

 

Log Name:      System
Source:        Microsoft-Windows-Hyper-V-High-Availability
Date:          31/08/2012 3:05:18 PM
Event ID:      21502
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      HYPERV1.domain.au
Description:
'Virtual Machine TypeQuick' live migration did not succeed at the destination.

'Virtual Machine TypeQuick' failed to start.

Live migration of 'TypeQuick' did not succeed. (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to restore with Error 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick': IDE/ATAPI Account does not have sufficient privilege to open attachment 'C:\ClusterStorage\Virtual Machine C Drives\VirtualMachines\TypeQuick-1\TypeQuick.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick': Hyper-V Virtual Machine Management service Account does not have sufficient privilege to open attachment 'C:\ClusterStorage\Virtual Machine C Drives\VirtualMachines\TypeQuick-1\TypeQuick.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)
Event Xml:

 
   
   21502
   0
   2
   0
   0
   0x2000000000000000
   
   2885993
   
   
   System
   HyperV1.domain.au
   
 
 
   Virtual Machine TypeQuick
   'Virtual Machine TypeQuick' live migration did not succeed at the destination.

'Virtual Machine TypeQuick' failed to start.

Live migration of 'TypeQuick' did not succeed. (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to restore with Error 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick': IDE/ATAPI Account does not have sufficient privilege to open attachment 'C:\ClusterStorage\Virtual Machine C Drives\VirtualMachines\TypeQuick-1\TypeQuick.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)

'TypeQuick': Hyper-V Virtual Machine Management service Account does not have sufficient privilege to open attachment 'C:\ClusterStorage\Virtual Machine C Drives\VirtualMachines\TypeQuick-1\TypeQuick.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID 7C955DEA-71F1-4799-A4B7-8D44E1548AB1)
 

 

I have looked online and found this page: http://support.microsoft.com/kb/2249906

 

I have tried that and get this response:

NT VIRTRUAL MACHINE\7C955DEA-71F1-4799-A4B7-8D44E1548AB1: No mapping between acc
ount names and security IDs was done.
Successfully processed 0 files; Failed processing 1 files

 

Im not sure what else to try, or why this is happening...

 

(I have been having lock-ups with this host in question which is what prompted me to put it into maintenance mode and run updates etc)

[RabbieBurns]

Also, the cluster validation passes everything except for a couple of windows updates which are not yet installed on the other host.

[RabbieBurns]

Ive just done a couple of reboots and now I'm seeing this in the event logs each time which might explain it

 

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          31/08/2012 4:41:15 PM
Event ID:      1069
Task Category: Resource Control Manager
Level:         Error
Keywords:      
User:          SYSTEM
Computer:     HYPERV1.domain.au
Description:
Cluster resource 'LUN 0003 (Witness Disk)' in clustered service or application 'Cluster Group' failed.
Event Xml:

 
   
   1069
   0
   2
   3
   0
   0x8000000000000000
   
   2886506
   
   
   System
   HYPERV1.domain.au
   
 
 
   LUN 0003 (Witness Disk)
   Cluster Group
 

even though the LUN003 (witness disk) is shown as Online under the Cluster MAnager, Storage, in the Disk Witness in Quorum...

[RabbieBurns]

and now the other host has gone down and i cant seem to turn it back on via remote console.. so all my VMs are offline.... FUCK!!

[Achandler]

I assume you have a failover cluster, since you talk about the witness disks etc.

 

Can you get either host online and actually login to it to open the failover cluster manager? You need a DC to be working to login and open the failover cluster (which I assume you have).

[RabbieBurns]

yes the server mentioned in the first post is up, has the VMs local to it, but cannot start them.

[Achandler]

So the VMs aren't stored on a SAN?

 

Or do you mean you can see the .vdi files within C:\ClusterStorage etc but cannot start them.

 

If the second problem is the case, then it is likely a permissions issue on the actual virtual machines or on the cluster disk. There is definitely a command you can run to see who has permissions to run a Virtual machine and you can add a server's permissions manually using the GUID (I think I remember). Another way of testing what the problem coudl be is create a new VM on the functioning server, then run it, this will prove the issue is purely with the already created VMs.

 

If you can create new ones fine, you can even test if the problem is with the "hard drive" or the configuration part of the VM by creating a new VM and attaching the .vdi of a dead VM checking if it boots (it probably won't) because it is likely the vdi that is locked. Do this test on a none essiential VM just in case.

 

The problem with a permissiosn issue is you can eb checkign things for hours to find where the problem lies.

[RabbieBurns]

VMs are on cluster shared storage (SAN) can see them but not run them. Tried the command about permissions (see a previous post) and it hasnt fixed them.

 

Can create a new virtual machine, but cannot create a VHD for it. I get an error similar to this:

 

[Achandler]

So you have read only permissions to the disk. VMs need modify to launch. and obviously can't create.

 

VMM and HyperV are probably fine, the problem will be the resource being locked through Failover Manager, I have had this before when a server failed. You could Try evicting the dead host from the failover cluster, so effectively you have a cluster of none, then bring the disks up one by one and see if you get an error. There is also a command to see permissions to the disks (as in which hosts). if the evicted hsot still has permissions you coudl remove them there to.

 

I have had a similar problem before caused by a dead host, and evicting it sorted my problem, I then had to remove a permission to the cluster resource as my disk was showing as redirected.

[RabbieBurns]

This page seems to have fixed my problem for me

 

(0xC03A0014) Failed to add device ‘Microsoft Virtual Hard Disk’ | User Answer To Computing Errors

 

I have made “Network Service” member of Administrative group and that worked for me

 

This seems to have worked for me too...

[Achandler]

I read that link earlier (the technet stuff), it also mentions Intel motherboards and problems on them.

 

There is a post from someone mentioned that the Network Service should never need to be in the Administrators Group as well.

 

But at least you have your VMs working now, and can look at other issues later.

[RabbieBurns]

I read the intel stuff, and tried the reg fix it suggested, but that didnt work.

 

No idea why this fix works or what caused it (im assuming it was the BIOS / Firmware update... )

 

and the other question remains is what has happened to the other host that Ive done no maintenance on.. why did that suddenly die.. I guess Ill find out when I can get to the server room in the morning...

[RabbieBurns]

Virtual Machines Fail to Start, cannot add disks

 

Looks like that might be closer to what caused the issue, and a more permanent fix.. ?

 

Will try this once I can get the 2nd host back up..