Windows Server 2008 R2 Thread, All Virtual Servers and a Powercut in Technical; Now firstly, this hasn't happened to me fully. But I have experienced a similar problem to what this would cause ...
26th October 2011, 01:06 PM #1
All Virtual Servers and a Powercut
Now firstly, this hasn't happened to me fully. But I have experienced a similar problem to what this would cause and was quite lucky to solve it quickly, but there must be a way of doing it.
The scenario all your servers are virtualised, you have 4/5 physical hosts connected through switches via iSCSI or Fiber to a SAN (with all your Virtual HDDS on). The power goes off for about 30 minutes, meaning all your servers and switches and SAN have to be turned off. How do you get everything running again?
I did similar to below:
I then went to a Host, but couldnt login as there was No Domain Controller, or DNS or DHCP. I logged in locally to check the HyperV Cluster status, but I couldnt as I was in on a local account, Windows gives you an error message saying Domain Accounts only basically. I couldn't login with a Domain Account because there was no Domain, no Domain Controllers had come online because the failover cluster had totally failed and now couldn't be restarted due to the above errror.
I looked to see if I could see the SAN through C:\ClusterStorage but it is inaccessible.
What is the process of getting a fully virtualised cluster back online? If your cannot login on the domain, you cannot access the cluster setup or the SAN to start vritual servers and if you don't start virtual servers then you cannot connect to your cluster.
All my virtual servers are set to auto-start, and I could ping between hosts, ping the SAN and ping the switches. I tried rebooting an extra time (cleanly) but this had no affect.
Anyone got the answer to this, I assume there is a set method (which I will be googling soon) because this siituation is realistic and nearly happened to me. I had a physical DC available, although I was planning to Virtualise this tomorrow! Which i now won't be.
26th October 2011, 01:17 PM #2
The order I did everything.
Physical VM management box (vCenter, vSphere client etc, hyper-v console)
SAN (wait for it to fully boot so you know if there are any errors to fix).
All VM hosts
First DC VM - Wait untill booted (will take a few mins extra due to no dns)
Second DC VM - will boot normally (assuming dns pointing at first)
Restart First DC - will boot normally (assuiming dns pointed at second)
Power on all other VMs.
26th October 2011, 01:19 PM #3
The problem lies with the Failover Cluster Manager, it would not run without you being logged into a Domain Account, so no DC meant to Management Tools.
26th October 2011, 01:20 PM #4
I would keep a physical DC, or failing that (and please feel free to correct me if this is not possible for whatever reason) have at least one of your hyper-v servers with a DC on as a stand-alone server to enable local logon.
26th October 2011, 01:20 PM #5
why do you restart teh first dc?
Originally Posted by DMcCoy
26th October 2011, 01:22 PM #6
Indeed Dos Box, I think a seperate physical DC, also providing DNS with all the Virtual machine Tools on is definitely the answer. I was wondering if there was a clever way of solving this with no physical boxes as DCs
26th October 2011, 01:28 PM #7
Originally Posted by Dos_Box
I thought that was just good practice, surely it doesn't make sense to have VM hosts on the domain for this very reason?
My hosts are all standalone machines.
26th October 2011, 01:28 PM #8
Because they get quite upset when booting without a domain, although with a single DC it tends to get the services going in a better order. It's just quicker than waiting for everything to retry later (replication etc).
Originally Posted by CyberNerd
Thanks to DMcCoy from:
CyberNerd (26th October 2011)
26th October 2011, 01:32 PM #9
We have an almost identical set up and were forced to put in a physical DC when doing some early disaster recovery checks. (It became evident that although we technically could get up and running again, it just took far too long / required a lot more knowledge going forwards etc).
Now we have a physical DC in the mix in a power loss situation we would power up switches, boot up the physica DC and check all services had started, then begin booting up everything needed for the virtual side of things.
26th October 2011, 01:43 PM #10
Depends on the versions you are running. It's possible to log into an individual host in a cluster using the vSphere client. You need to have left root able to manage the server. If you are using ESX, you could also use the console or secure shell in and use vmware-cmd to start guest machines.
26th October 2011, 01:58 PM #11
Just an add-on to my earlier post....
As I mentioned, we are almost identical in terms of setup etc. What we did with our "Physical DC" is to actually have the physical box run the Hyper-V role, and then have a virtual DC running on it! This additional DC is still technically virutal, but because the physical server runs the bog standard Hyper-V role outside of the failover cluster manager, it is simply a local service. The reason I did this was to give a bit more flexibility in future. If the hardware died I just have a VHD to plug in somewhere - or import in to the cluster while I replaced harware etc. Might be worth considering?
26th October 2011, 02:00 PM #12
- Rep Power
thats what we do. another +1 to vmware in the vsphere v hyper-v dogfight as far as I'm concerned!
Originally Posted by pcstru
26th October 2011, 02:07 PM #13
If it's only 30 minutes i wouldnt need to shut them down at all Got a backup generator powering the server room and office with the UPS's providing failover while the generator kicks in which provides a fair few hours of electrical down time.
But for the sake of argument i'll play along
Switches on first, iSCSI on second then the important VMs (DCs + Vcenter server) are all set to power on first automatically when the host powers up along with the linux gateway and proxy servers. I've then got 5 minute delay after the last DC is powered on before other critical servers start coming up, again automatically. Less important non-system critical servers can then be powered up manually.
I have 2 DCs on seperate physical hosts for the very reason if one fails to automatic boot the other should, if for any reason both didn't boot i would have to find other ways around that but i can't see why vSphere wouldn't work from a laptop connected directly or something.
26th October 2011, 07:18 PM #14
I have had the power go a couple of times now as the Power is a bit ropey in the area I work in, but deffinately has helped having a physical server in the virtual environment as a couple of other people have said, I also sort of use this for backups so it isn't just sitting their doing nothing. I am using Xen as my virtualisation solution and below is the order I use to get everything back up.
Physical Server (Access to Xen Management Console)
Physical Servers Hosts
DC VM's (DHCP,DNS etc ( although these are set to startup automatically when Physical Host is available so they start first)
To be honest it usually takes about 15 to 20 Minutes to get everything back up and only one memeber of staff has noticed the two times it has happened as we are both in pretty early, we have also docmented this and I have this sat ontop of the Server with IP etc just in case I am not in and one of the other Techs has to do it.
26th October 2011, 10:17 PM #15
The guidance from Microsoft is to keep one Physical DC to protect against this. Exactly my setup - I have two further virtual DCs. 3 x Hosts, with 35 total virtual servers
By jgcracknell in forum Virtual Learning Platforms
Last Post: 24th February 2011, 01:52 PM
By pete in forum Licensing Questions
Last Post: 24th November 2009, 03:58 PM
By stu1892 in forum Windows Server 2000/2003
Last Post: 7th August 2009, 01:46 PM
By Kyle in forum Thin Client and Virtual Machines
Last Post: 7th March 2007, 12:01 AM
By Cyber-Dude in forum Windows
Last Post: 26th May 2006, 11:01 AM
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)