Disaster Recovery Horror Stories - Have you got one?

[L3OUGH]

We have been reviewing our DRP over the last day or two and we wondered if anyone knows or has heard of any stories of the network manager/techs basically walking in to the worst possible situation imaginable (such as a fire that ruins EVERYTHING, backups included) and walks away telling their bosses to sort it without them etc. Basically walking away from the job??

 

Just curious to see if anything like this has been witnessed by anyone here.

[unixman_again]

You can read about one of mine here http://www.edugeek.net/forums/general-chat/168291-today-world-backup-day-dont-forget-your-backups.html

[L3OUGH]

You can read about one of mine here http://www.edugeek.net/forums/general-chat/168291-today-world-backup-day-dont-forget-your-backups.html

 

Impressive you managed to get as much as you did back from just the paper records!!

[Joanne]

I imagine the Network Manager from this school has a tale to tell: UPDATED: Leyland school fire was ?catastrophic? says judge - Lancashire Evening Post

 

Leyland school fire: Five boys held on suspicion of arson - BBC News

 

Five boys arrested after Lancashire school fire | UK news | The Guardian

[SteveB_NI]

DR is something I should really think more about... I'm using Windows Server Backup on my DC's to back up to external hard drive each night, which is regularly swapped / exchanged and stored elsewhere.

 

I have the backup set to backup the following items: Bare metal recovery, system state, and then the hard drives. Using VSS copy backup.

 

 

My worry is now I have read that bare metal recovery only works on identical hardware? As in, if the worst comes to the worst & I need a new server prompt this could be an issue?

[mwnci]

My worry is now I have read that bare metal recovery only works on identical hardware? As in, if the worst comes to the worst & I need a new server prompt this could be an issue?

 

We were doing some testing regarding this a couple of weeks ago. We restored the backup to different hardware but of course the OS would not boot. We then inserted the relevant windows DVD and ran a windows repair and it worked leaving us with a fully functioning backup on entirely different hardware.

 

Test it out on your backups and see what happens.

[smurfomatic]

Not quite catastrophic (in fact, all data was recovered) but a tale to tell...

 

August 2013 - a couple of days before the start of term. Main server (DC, Exchange, email & MIS host) falls over. Seems to be related to a HD failure so that is swapped and a restore began. For some unfathomable reason, restore reports it'll take around 100 hours to complete! No way to figure out why (as pressure was on to get things up and running) so left it running over the weekend.

 

Backup restore reaches 99% then fails. System still not running. Turns out - after further investigation - it's not a hard drive problem but the controller. Frantic order for a new server (old one was about seven years old) and system back up and running a few days later. Two weeks I could have done without!

 

New backup procedures will be implemented this summer, alongside a change around of classrooms - single physical server hosting our two machines as virtual servers. Backing up to NAS device in separate building. If we have to use any further DR procedures, data recovery will be the least of our problems - two of the three school buildings will have been destroyed...

[Nixon77]

I heard of a guy losing everything.

 

Server raid failed, and when they went to the backup, it was useless, all the data was corrupt. Think it was a robo copy job.

 

He kept his job.

Edited April 6, 2016 by Nixon77

[davethetekkie]

 

This is a 250GB IDE Hitachi Deathstar that was the sole data drive in a server called "DATA-1"; a Pentium 4 PC pressed into service as a Server 2003 machine. Everyone at the company had ASSUMED that there were daily backups going to another server. At time of failure the server was 7 years old (approx 2004 build); the drive was not rated for 24 hour operation but had been pulling it anyway. All it took was a power cut (no UPS) and they were reduced to pulling together remnants of their data from individual workstations. When I did a post mortem on the server the main board was covered in failed caps and the fans hadn't seen compressed air in 7 years.

 

I keep it on my desk at home as a reminder and a warning. Crazy that this stuff happens... sad but true.

[pete]

It's not quite full-blown site outage, but definitely fulfills the "disaster" and "recovery" components....

 

A few years ago (I probably mentioned it on here*) approximately half our site flooded. Then, while assessing the situation reception got a call from my neighbour of "tell Pete his house has flooded".

 

So factor in that if your IT people live locally and you're affected by flooding, they may actually be doing N+1 disaster recovery plans and will be tired and annoyed.

 

* yup. http://www.edugeek.net/forums/general-chat/97493-ive-not-seen-rain-like-2.html#post847131

http://www.edugeek.net/forums/general-chat/97493-ive-not-seen-rain-like-3.html#post847177

http://www.edugeek.net/forums/general-chat/97493-ive-not-seen-rain-like-3.html#post847244