Hyper-V Host Problems

[Darylrese]

Dear all,

 

I'm having a bit of a nightmare with one of my Hyper-V hosts and wondered if anyone could help.

 

All of a sudden today, my second Hyper-V Host displayed 'No virtual machines installed' . All virtual machines on that Host ARE working currently.

 

Did a bit of research and on TechNet they suggesed restarting the 'Windows Management Instrumentation' service which saves rebooting to fix this issue.

 

Unfortunately, I went to do this but the service was already in a 'Stopping' state and has got stuck. This has resolved in all Hyper-v management services stopping too.

 

I would just reboot the Host, however both of my DC's are on this host too so I want to avoid it unless 100% necessary. This is something I am looking at sorting out ASAP as it's been a concern of mine since starting at my new school a month ago.

 

If I do need to restart the Host, how should I go about this baring in mind BOTH of the DC's are on this host?

 

I cant migrate machines because none of the management tools are working on the host in question! All I can think is safely shutdown the servers on HOST 2, reboot and hope for the best!

 

Any ideas?

Edited December 8, 2013 by Darylrese

[kmount]

Yeah, that's what I'd do, shut them down cleanly and spin the host.

 

If they are on shared storage even if the host doesn't come back you can at least fire them up on the other host.

 

If on local storage, check your backups work first.

[ChrisH]

What Hyper-V version are you using ? If it's 2008 R2 it sounds like you are aware of the chicken and the egg situation of clustering with no contactable DC, unless your running 2012 as this is supposed to work better now. If I was you I would make another DC on the other host, even if you remove it later. If you don't then you need to get a copy of the domain controllers VHD and stick it into a new VM on the other machine, I have done this quite a few times now and all you need to do is set the IP address again. Of course the risk you run is having the original DC starting up on the other host.

[Bueno]

What backup method are you using on your DC's? The reason I ask is that you should only be using system state backups for your DC's. I wouldn't recommend moving a DC VHD to another host and turning it on, this will cause a few issues - mainly replication problems.

 

I would turn off one of the DC's on the problem host and set it so the VM doesn't start when the host is started. Restore the same DC to another host using system recovery, configure networking and check replication/DC health

 

Once you move the DC via this method I would just power off the other DC on the problem host and then power cycle the problem host.

Edited December 8, 2013 by Bueno

[ChrisH]

What backup method are you using on your DC's? The reason I ask is that you should only be using system state backups for your DC's. I wouldn't recommend moving a DC VHD to another host and turning it on, this will cause a few issues - mainly replication problems.

 

I would turn off one of the DC's on the problem host and set it so the VM doesn't start when the host is started. Restore the same DC to another host using system recovery, configure networking and check replication/DC health

 

Once you move the DC via this method I would just power off the other DC on the problem host and then power cycle the problem host.

 

The OP stated that the Hyper-V management services had already stopped as part of the problem so they will not be able to change any of the configuration. If both the DCs VM are shutdown via remote desktop if possible then copying across the HD will be fine, there will be no issue with replication as they will all obey the latest revision. I've had to do this a few times and there were no issues but you have to be careful of ending up with the same machine running on both hosts If you can, rename the VHD to stop it starting.

Your best bet is another temporary DC on the other host though, make it outside the cluster on some none shared storage in case the other host suddenly has an issue with the lack of a contactable DC, you will be able to start that DC without clustering started up.

[Darylrese]

Yep they are on shared storage. So if host fails to come back up for any reason I can just make a new vm for my dc's on host 1 and reattach the storage?

 

Is there anyway of getting the services to start without a reboot?

 

Yep it's Hyper-v server 2008 R2.

 

Thanks for help so far guys. I'm pretty sure I set the dc's to automatically start on host restart so provided it comes back up ok should be alright.

 

As DC's are not started until host is booted, will I be able to get onto my PC to remote into the host or will I need to attach a monitor to the host itself?

Edited December 8, 2013 by Darylrese

[kmount]

Are the hosts joined to the domain that the virtual domain controllers manage? If so that could cause you some chicken and egg problems for sure though if they are standalone workgroup members it should be less frightening. If RDP is enabled you should be able to connect to the host from your PC assuming you can log onto it without the domain being around (make sure you know the local admin password if it is domain joined just incase!)

 

I don't know as much about 2008R2 Hyper-V but certainly on 2012 it's quite flexible when it comes to host failure so there's always a way forward and having it on shared storage helps here too. Backups will be crucial here just incase it does go wrong though.

 

Inclined to agree with ChrisH though that another DC on the working host would be sensible if for nothing more than to protect you from shenanigans with no DCs should the problematic host have issues getting the DCs back up.

[Darylrese]

Yep the hosts are joined to the domain!

 

I don't have much confidence in the backups either as DPM doesn't seem to let me take a full backup of a VM only partitions on them? I plan on doing an overhaul of Hyper-V and backup's but like I say have only been here a month or so.

 

I will probably have to copy the VHD's of the DC and start up a new VM on the first host just incase.

 

I would have thought there would be a way to force Windows Management Instrumentation service to restart?

 

I have already tried killtask /pid /f without any luck.

Edited December 8, 2013 by Darylrese

[kmount]

Phew, yikes

 

I'd get yourself a new DC up on the other host per ChrisH advice so you can at least have something you have control of.

 

I don't know a lot about DPM sorry, but if I was you, I'd fish out a copy of Veeam Free Edition from Veeam: Modern Data Protection too ( Veeam Backup Free Edition : Free Backup For Hyper-V | Veeam Software Official Blog ) so you can get backup your 'new' DC to give you some comfort!

 

If the service is hung on 'stopping' you might find something like procmon may help show you what's happening or at least try and point you in the direction if a process is hung that needs to be killed to let your stuck process go.

 

Sometimes though, a reboot is unavoidable

[adamf]

Spinning up a new DC on the working host is definitely the way to go. If you don't and you restart the host with the problem and both DCs on it, you will loose access to the cluster, as others have pointed out. What's worse is, if you make use of Cluster Shared Volumes and you loose both DCs, you won't be able to get to the ClusterStorage folder get the VHDs (since this requires AD access) then you'll have the attach the storage to another machine to get the VHD of a DC so you can start it on another machine.

 

Been there, done that!

[Darylrese]

Thank-you guys!

 

Can someone just run through the steps to take to get the DC up and running on the working host please?

 

I came from a school with VMware and I'm fairly new to Hyper-v...don't like it much!#

 

Do I call the DC the same as the current one on the new host?

 

I had a look at Veeam a few weeks ago at a seminar, it looked amazing! Going to get a price for it!

Edited December 8, 2013 by Darylrese

[adamf]

Yep..

 

Build a new VM as you normally would, install whatever OS the other DCs are using and join to the domain as a member server. Give it a different name, since you will have 3 running when the other 2 are back up.

 

Install AD (either add/remove roles or dcpromo - depending on the OS), follow through the wizard and allow it to replicate. I would make it a DNS server also as I am guessing the other DCs are DNS server and without them you will have a few problems whilst they are unavailable. I would also set the primary DNS on the host you are about to restart to be the newly created DC/DNS server so there is something to contact when it starts up.

[Darylrese]

Ok great, thank-you. So I shouldn't use the VHD's of the current DC to build the new server?

 

Our current DC is on Server 2012

[adamf]

Nope... Build it from scratch. Everything will replicate across to the new DC.

[Darylrese]

Thank-you Adam and kmount! I have PM'd you Adam for more info.

 

I will try and build a new DC this week.

 

Regards

[kmount]

Good luck fella!

 

What doesn't kill us makes us stronger, good learning experience too.

 

You'll get through this and be stronger for it.

[Darylrese]

haha indeed it does!

 

I am currently building a third DC as we speak on the working host, will let you know when its up and replicated and where i go next!

 

Many thanks again

[Darylrese]

Morning All,

 

I have created DC3 this morning on the working host and have replicated data across from the existing DC. AD and DNS are now fully populated.

 

Just applying updates to server now and will reboot host later in the week or at weekend.

 

Im guessing i don't need to do anything else? Aslong as DC3 is up when the second host is rebooted we should be good to go?

[ChrisH]

From a cluster starting view, you should be okay, you may want to consider the plan of action if the host doesn't reboot properly and the hosted VMs fail to failover but that is unlikely.

[Achandler]

Make sure your online DC is pointed to by your servers for DNS. Otherwise they won't work on the reboot as they won't see the domain even though it is there.

[zag]

You really shouldn't have DC's virtualized.

 

I keep both of mine on physical boxes.

 

EDIT: sorry didnt realize you were already building another DC

Edited December 9, 2013 by zag

[CHR1S]

I have 3 DC's two virtual on two different hosts and 1 physical.

 

Out of interest, why didnt you just RDP into one, shut it down then add the VHD on a different host and start it back up?

Also, could you manage the VM's from the hyper-v console on the other host or does the dead service stop management full stop?

[Darylrese]

I am certainly going to add a physical DC to the infrastructure, just haven't had chance yet as only been here a month.

 

If i shut down the DC in question to do that, i can't start it back up due to the error so if it didnt work, id have no DC.

 

The dead services stop the management full stop, heres what i get:

 

 

[CHR1S]

Your only shutting down 1 DC, the second is still running right?

 

If the move fails you still have a DC working while you create a physical DC. Its a 10 minute job, rdp > shutdown > connect VHD to another VM on the working host > start VM

[Darylrese]

The second one is on that host too Chris. Bit of a nightmare!

 

However, i just spoke to the company that put the hosts in and they have said there IS another DC installed on the DPM box as a VM (physical hardware off the cluster) which is already the primary DNS server so we should be fine to shutdown machines, reboot the host and bring everything back up!

 

Sorry i didn't know this being new to this network, maybe problem solved!

Edited December 9, 2013 by Darylrese