All Virtual Servers and a Powercut

[Achandler]

Now firstly, this hasn't happened to me fully. But I have experienced a similar problem to what this would cause and was quite lucky to solve it quickly, but there must be a way of doing it.

 

The scenario all your servers are virtualised, you have 4/5 physical hosts connected through switches via iSCSI or Fiber to a SAN (with all your Virtual HDDS on). The power goes off for about 30 minutes, meaning all your servers and switches and SAN have to be turned off. How do you get everything running again?

 

 

I did similar to below:

Switches

SAN

Hosts

 

I then went to a Host, but couldnt login as there was No Domain Controller, or DNS or DHCP. I logged in locally to check the HyperV Cluster status, but I couldnt as I was in on a local account, Windows gives you an error message saying Domain Accounts only basically. I couldn't login with a Domain Account because there was no Domain, no Domain Controllers had come online because the failover cluster had totally failed and now couldn't be restarted due to the above errror.

 

I looked to see if I could see the SAN through C:\ClusterStorage but it is inaccessible.

 

What is the process of getting a fully virtualised cluster back online? If your cannot login on the domain, you cannot access the cluster setup or the SAN to start vritual servers and if you don't start virtual servers then you cannot connect to your cluster.

 

All my virtual servers are set to auto-start, and I could ping between hosts, ping the SAN and ping the switches. I tried rebooting an extra time (cleanly) but this had no affect.

 

Anyone got the answer to this, I assume there is a set method (which I will be googling soon) because this siituation is realistic and nearly happened to me. I had a physical DC available, although I was planning to Virtualise this tomorrow! Which i now won't be.

[DMcCoy]

The order I did everything.

 

Switches,

Physical VM management box (vCenter, vSphere client etc, hyper-v console)

SAN (wait for it to fully boot so you know if there are any errors to fix).

All VM hosts

First DC VM - Wait untill booted (will take a few mins extra due to no dns)

Second DC VM - will boot normally (assuming dns pointing at first)

Restart First DC - will boot normally (assuiming dns pointed at second)

Power on all other VMs.

[Achandler]

The problem lies with the Failover Cluster Manager, it would not run without you being logged into a Domain Account, so no DC meant to Management Tools.

[Dos_Box]

I would keep a physical DC, or failing that (and please feel free to correct me if this is not possible for whatever reason) have at least one of your hyper-v servers with a DC on as a stand-alone server to enable local logon.

[CyberNerd]

 

 

Restart First DC - will boot normally (assuiming dns pointed at second)

 

why do you restart teh first dc?

[Achandler]

Indeed Dos Box, I think a seperate physical DC, also providing DNS with all the Virtual machine Tools on is definitely the answer. I was wondering if there was a clever way of solving this with no physical boxes as DCs

[sparkeh]

I would keep a physical DC, or failing that (and please feel free to correct me if this is not possible for whatever reason) have at least one of your hyper-v servers with a DC on as a stand-alone server to enable local logon.

Bingo!

I thought that was just good practice, surely it doesn't make sense to have VM hosts on the domain for this very reason?

My hosts are all standalone machines.

[DMcCoy]

why do you restart teh first dc?

 

Because they get quite upset when booting without a domain, although with a single DC it tends to get the services going in a better order. It's just quicker than waiting for everything to retry later (replication etc).

[mb2k01]

We have an almost identical set up and were forced to put in a physical DC when doing some early disaster recovery checks. (It became evident that although we technically could get up and running again, it just took far too long / required a lot more knowledge going forwards etc).

Now we have a physical DC in the mix in a power loss situation we would power up switches, boot up the physica DC and check all services had started, then begin booting up everything needed for the virtual side of things.

[pcstru]

Depends on the versions you are running. It's possible to log into an individual host in a cluster using the vSphere client. You need to have left root able to manage the server. If you are using ESX, you could also use the console or secure shell in and use vmware-cmd to start guest machines.

[mb2k01]

Just an add-on to my earlier post....

As I mentioned, we are almost identical in terms of setup etc. What we did with our "Physical DC" is to actually have the physical box run the Hyper-V role, and then have a virtual DC running on it! This additional DC is still technically virutal, but because the physical server runs the bog standard Hyper-V role outside of the failover cluster manager, it is simply a local service. The reason I did this was to give a bit more flexibility in future. If the hardware died I just have a VHD to plug in somewhere - or import in to the cluster while I replaced harware etc. Might be worth considering?

[Avalon]

Depends on the versions you are running. It's possible to log into an individual host in a cluster using the vSphere client. You need to have left root able to manage the server. If you are using ESX, you could also use the console or secure shell in and use vmware-cmd to start guest machines.

 

thats what we do. another +1 to vmware in the vsphere v hyper-v dogfight as far as I'm concerned!

[mrbios]

If it's only 30 minutes i wouldnt need to shut them down at all Got a backup generator powering the server room and office with the UPS's providing failover while the generator kicks in which provides a fair few hours of electrical down time.

 

But for the sake of argument i'll play along

 

Switches on first, iSCSI on second then the important VMs (DCs + Vcenter server) are all set to power on first automatically when the host powers up along with the linux gateway and proxy servers. I've then got 5 minute delay after the last DC is powered on before other critical servers start coming up, again automatically. Less important non-system critical servers can then be powered up manually.

 

I have 2 DCs on seperate physical hosts for the very reason if one fails to automatic boot the other should, if for any reason both didn't boot i would have to find other ways around that but i can't see why vSphere wouldn't work from a laptop connected directly or something.

[Steven_Cleaver]

Hi

 

I have had the power go a couple of times now as the Power is a bit ropey in the area I work in, but deffinately has helped having a physical server in the virtual environment as a couple of other people have said, I also sort of use this for backups so it isn't just sitting their doing nothing. I am using Xen as my virtualisation solution and below is the order I use to get everything back up.

 

UPS

Switch

SAN

Physical Server (Access to Xen Management Console)

Physical Servers Hosts

DC VM's (DHCP,DNS etc ( although these are set to startup automatically when Physical Host is available so they start first)

Other VM's

 

To be honest it usually takes about 15 to 20 Minutes to get everything back up and only one memeber of staff has noticed the two times it has happened as we are both in pretty early, we have also docmented this and I have this sat ontop of the Server with IP etc just in case I am not in and one of the other Techs has to do it.

[TheScarfedOne]

The guidance from Microsoft is to keep one Physical DC to protect against this. Exactly my setup - I have two further virtual DCs. 3 x Hosts, with 35 total virtual servers