DO NOT SNAPSHOT HYPER-V DC!!!!!!!

[AWicher]

I just wanted to let others know my recent experience with our school DC's and hyper-v, and hope this will help save some one the horror of almost losing AD.

 

I will keep it short, (parts and steps may have been missed)

 

We are running on 2 blade servers each one has a DC, replicating... The problem was when for a reason or other we needed to restore a backup. Now any one who uses VMWare, Hyper-v or indeed was at the last educonf will know you can take a snapshot of the server in its current state and then have the option to rollback to a previous state if you have a problem.

 

Such a time came, so with out hesitation a rollback was done. This is when the problems started.

 

Our first problem was we got an error stating netlogon was paused, and replication failed

After several min of looking at logs... we decided another rollback (Still unaware this was the problem) and still not fixed the netlogon issue.

 

We decided to isolate, demote and remove our backup DC then rebuild it. Having done so we rebooted both servers only to find on our DC had no users, workstations. Both of us (network manager n' me) in order then then

 

At this point we found a few posts from other forums (Yes, shock to me that others are out there) but Edugeek FTW) that having a V-DC and using snapshots, rollbacks... would make issues like the one we where having.

 

As we did not have a "Normal" backup of our DC another rollback was done on both to a time that where only a few min apart from each other, in the mind set well it cant get much worse then loosing ones AD.

 

Thankfully, this worked. we where still in the same spot of not having a working netlogon and replication but all the users... where back.

 

more digging and we found a simple command to fix the connection for replication.

 

repadmin /options -DISABLE_INBOUND_REPL

and

repadmin /options -DISABLE_OUTBOUND_REPL

 

both restored the connection and forcing the netlogon service to run, so users where back and they could log on

 

now we are still having issues with the server and NM is in the proses of working out if a rebuild of the DC or some other option is out there, but what i want to tell you is definitely go for some sort of V-DC just to read this **LINK** before you attempt it.

 

 

I hope i have helped some one buy sending out this warning, that or every one already knows the problem and it was an oversight on our part.

[DMcCoy]

Be careful with snapshots for backups too. Certainly for vmware keeping a snapshot for anything other than a short time is a bad idea due to the massive redo disks that can be generated which can then take a very long time or fail to merge.

 

At minimum you want to do a system state backup of DCs even when using other imaging products for backup.

 

As you have found, you never want to snapshot DCs if there are multiple, with a single you can just about get away with it.

[ZeroHour]

Thanks for the info, I knew about the general snapshot limitations with DC's but the info about export etc is v useful.

 

Cheers

[azrael78]

Thanks for this - while we don't have any V-DC's yet, it's only a case of time before management decides to cut back on our server farm and request we virtualise even more... great idea, nice technology - but on some things, virtualising adds a few nice things (like snapshotting), great for workstations but I'm all for taking a full backup of a server and not using snapshot technology.

 

However I will be sure to keep this article in mind when the day of V-DCs and someones' 'playing around with said DCs' happens...

 

Az

[srochford]

Microsoft have always said that you should never image a domain controller as a backup means. In the past, I'd guess it wasn't easy to do this (would have had to boot into WinPE or whatever to run Ghost etc) but snapshotting makes it much easier so this is a timely reminder that you mustn't ever try and use a snapshot/image type backup.

 

I'd guess the only exception might be if you've had a total disaster and lost everything - bringing back one image could work. (It's also OK if you've only got one DC but no-one runs a real network like that :-))

[PiqueABoo]

Microsoft have always said

 

Yes they have but not very loudly... this precise topic has cropped up a couple of times on here to my knowledge.. suspect searching for USN will find them, but it's *definitely* worth repeating as it's clearly not widely understood and more and more folk will be getting involved with VMs.

 

I reckon (YMMV) the safe way to do DC snapshots is to shut down all your DCs and then snapshot them - and if you do want to roll back and it's within the tombstone period, shut them down again and revert them all back to that set of snapshots, then of course start them up. Not sure how useful it is, but it's the kind of thing I might consider doing immediately before a round of serious upgrading.

 

 

The other issue with snapshots is of course performance - can't comment on how much it hurts but I've been assured by a serious expert that it does (and the hit obviously increases with the number of snapshots).

[cookie_monster]

Couple of handy links here.

 

Deployment Considerations for Virtualized Domain Controllers

 

Virtualized Domain Controllers and Replication Issues

http://technet.microsoft.com/fr-fr/library/dd348479(WS.10).aspx

 

How to detect and recover from a USN rollback in Windows Server 2003

 

It's also worth remembering tombstone lifetime issues if restoring an old image or snapshot. As DMcCoy said a good system state backup is essential.

Edited August 18, 2009 by cookie_monster

[dhicks]

It's also OK if you've only got one DC but no-one runs a real network like that :-))

 

Why not? Is there some performance issue with domain controllers? Surely all they're doing is checking whether a given username and password combination matches okay? Is there a rough limit to the number of clients you should have per DC?

 

--

David Hicks

[powdarrmonkey]

Why not? Is there some performance issue with domain controllers? Surely all they're doing is checking whether a given username and password combination matches okay? Is there a rough limit to the number of clients you should have per DC?

 

Performance is a consideration, but usually:

 

- maintain continuous service by failing over gracefully

- protect you by sharing out the Master roles and keeping replicated copies

- localise DCs to subnets to reduce backbone traffic, make DFS lookups sensible, etc

[SYNACK]

Why not? Is there some performance issue with domain controllers? Surely all they're doing is checking whether a given username and password combination matches okay? Is there a rough limit to the number of clients you should have per DC?

 

--

David Hicks

 

Its not so much the performance that is the concern rather than the lack of duplication, having two means you have two integrated DNS servers and two copies of the database along with the ability to split the roles. In this way if something major happens to one of your DCs you can seize the roles to the other, keep all of your user accounts and computer accounts that are still completely up to date.

 

You can restore snapshots and system state backups but that means more downtime and also many more possibilities for issues with users who have changed their passwords or been added since the last backup. Worse if some of the stations have aumotaticly refreshed their machine passwords in the background during that time kicking them unglamourously off the network (modern Windows OSs do this with some regularity in the background to increase security).

 

AD itself although being a database is infact comparitivly light it is the the concern of consistancy and up to the minute data preservation that is a factor in many smaller environments.

[uk101man]

Hi,

 

I just would like to add that you should never take a snapshot then try increase the vhd. This breaks the snapshot.

 

Lucky for me I had a back up and could restore the VM.

 

I must of been working without having a cup of tea that day as I thought I had merged the snapshot.

[apaton]

The guys over at Veeam have a white paper about VSS backups for ESX based MS Windows environments, importantly they cover the issues of consistent restores for AD.

_files_whitepapers__protected_vmware_and_vss_aplication_backup_and_recovery.pdf

[cookie_monster]

Originally Posted by srochford

It's also OK if you've only got one DC but no-one runs a real network like that :-))

 

I would say that no one would run a large network like that I'm certain there are many thousands (possibly millions) of Windows SBS single server setups out there.

[SYNACK]

I would say that no one would run a large network like that I'm certain there are many thousands (possibly millions) of Windows SBS single server setups out there.

 

To be fair SBS networks are not real networks, each one is a conveniently disguised gateway to hell through which limitless evil flows.

[cookie_monster]

I take it you're not an SBS fan then? (the server software I mean not the crazy boat people)

[SYNACK]

I take it you're not an SBS fan then? (the server software I mean not the crazy boat people)

 

Correct , I have been in car accidents less painful than dealing with that specific piece of software. It does have its place, just so long as that place is very far away from me

[cookie_monster]

I remember the one rule with SBS was always use the wizards or you will pay.

[SYNACK]

I remember the one rule with SBS was always use the wizards or you will pay.

 

Even that does not always save you, some of them are dark wizards with awuful powers of destruction and a definite sence of indifference.

[cookie_monster]

some of them are dark wizards with awuful powers of destruction and a definite sence of indifference.

 

Free Sauron with every licence

[PiqueABoo]

there are many thousands (possibly millions) of Windows SBS single server setups out there.

 

There are quite a few primary schools, and I suppose that might even mean the majority of all UK schools, which one way or another have a single DC+jack-of-all-IT server too.

[ZeroHour]

Free Sauron with every licence

 

LMAO