Lost PDC :'(

[MkII]

Ran some Windows updates on the primary domain controller, and it then wouldn't boot.

Restored the VM and what's come back is pre September last year.

We have a secondary DC.

I think all machines have lost their trust relationship. This year's pupil intake is missing.

Is there any way to recover data from the secondary DC?

[chazzy2501]

if the secondary DC was up to date it may now have to fight with the restored out of date PDC, you should have left the PDC dead and built a new one. The secondary DC should have kept you going just fine.

 

EDIT: I'd recommend you turn the restored PDC off and see the secondary domain controller stays happy. the SDC should have the up to date active directory database, the DNS and group policy settings. I suspect your really just missing a DHCP host at that point.

Edited May 30, 2018 by chazzy2501

[FN-GM]

What he said ^^

 

You should have seized the roles so your last becomes the PDC emulator (and other roles) then built a new second DC.

 

What shape is that second DC? Can you still see the accounts etc?

[3s-gtech]

Erk, oh no! You can't restore a DC like that - you can restore it offline to retrieve anything that may have been on it, but it can't go back to its old role. Shut the old FSMO role holder down. Restart the second DC. See what comes up. Then, you may have to re-do any changes to your AD that have been lost, and possibly go round re-joining your PCs to the domain. You'll need to seize the FSMO roles on the second DC, rebuild the old role holder, then transfer them back. Change your backup strategy too.

[MkII]

Backup strategy - check

I wasn't thinking re the restore - came in off holiday to sort quickly. Bad liaise fair attitude on my part. Job has gone crazy this year.

 

Secondary DC has this years users and appears to be up to date. PDC is turned off meantime.

[3s-gtech]

Now it's off, seize the roles, wait for them to transfer, relax for five minutes, and build another one. I had to rebuild one of our DCs a few weeks ago - it wasn't the role holder but did handle DHCP. It was extremely simple to get it all back up and running, without backups. Just be thankful you have your second DC!

[MkII]

fsmo roles transferred successfully. Now need to point clients to 2ndDNS...

[3s-gtech]

DHCP should take care of that, for the most part. Make sure you always use the second server option rather than just leaving it blank. Same for statics. If DHCP popped (and wasn't replicated), take your time with rebuilding it - this is where having some documentation on the ranges helps.

[MkII]

Duff DNS settings. Always suspected something wasn't right there lol. New server built. Updates going on. Will install roles and promote tomorrow.

Kicked off encryption on a few laptops.

All in all a fun, productive day. [emoji16]

 

Thanks for the assists [emoji106]

[AlanD]

Restored the VM and what's come back is pre September last year.

?

 

Restored the VM? From what? There is a time limit for which trusts remain valid - I think its 30 days....(...also thinking....why are there not any recent backups?...)

[MkII]

Restored the VM? From what? There is a time limit for which trusts remain valid - I think its 30 days....(...also thinking....why are there not any recent backups?...)

Yeah quiet you

 

Hyper V snapshot. A critical oversight. Yes isn't there a tombstone period.. I think we passed it, luck would have it!

Edited May 31, 2018 by MkII

[AlanD]

Snapshots should never be considered as backups. In fact...I think I'm correct in saying that general advice is never to use use/maintain snapshots in a production server. There are lots of examples - particularly if the server runs databases - where a simple restore from a snapshot is not going to restore working versions of databases - or at least not without data loss.

 

My mantra - is that backups are the ONLY important thing that needs to happen. Servers can be slow, internet can be unavailable clients not being able to connect, etc, etc.... but there can't be an excuse for not having timely backups, even if it means running windows server backup onto a USB drive. The trouble is - because no-one sees or knows if backups happen - its far too tempting to push them down the priority table when everyone is screaming at you..

 

If I were head/manager of a school - maintaining current backups would probably be at the top of the list of demands. I'd be asking every week (possibly every day)....are backups working? Ransomware is thought to bring down at least schools every week....despite latest patches and AV installed and up to date. Off line backups are the only real protection against these attacks (yes, user training is really important) ....and investment in redundant PSUs, SANS, controllers or whatever are no substitute for having a backup.

 

I guess its too late to ask why the server wouldn't reboot? Assume it wouldn't start in safe mode...I guess its a wake up call to all of us - that a perfectly working server can just break when it restarts.

 

...wondering if the server restore was only a restore of the OS partition....and the data is still there.

[Domino]

I'd say never backup a DC. backup the AD state using whatever method your backup tool has.

 

That way you're never in a position to restore it.

 

The tombstone period has been 180 days since 2003ish IIRC. However DCs will be marked as "do not sync" in their system properties before that.

[MkII]

What's the best way to backup AD? We use Veeam for user areas etc.. I'm not a fan of the windows tool, but if needs must I guess..

[Domino]

Veeam have some guidance: https://www.veeam.com/blog/backing-up-domain-controller-best-practices-for-ad-protection.html

 

We use Networker, it has an AD system state option when choosing a DC.

[MartinT]

We replicate all our servers in Hyper-V now, which gives me comfort that I can failover if I need to.

 

All VMs are backed up to 8TB caddies kept in our cars on rotation, plus one in the fire safe.

[MkII]

I'm having more fun today. Print server, also a Hyper-VM, wouldn't boot as the CD Drive was set as 1st. No problem.

 

Next.. File Server host (attached disks) can't find file to boot from. Bit stuck with this one. A simple rebuild but wonder what's going on.

[MkII]

Rebuilt the file server and re-attached the vhxd. All happy. Corrupt config file I fear. Nearly time to go home!

[3s-gtech]

You sure a rogue Windows Update didn't sneak it, break all your stuff, then sit there innocently shrugging? That's a lot to break. Almost points to failing host disk array (had that one happen, very weird things happened before it died completely!)

[MkII]

Well it does make you wonder. We're favouring several conspiracy theories.