Disaster recovery plans - have you got some?

[localzuk]

So, in light of the arson attack on the council building in Oxfordshire, it got me thinking about disaster recovery plans.

 

Do you all have up to date disaster recovery plans that would work properly should your site be lost in a disaster?

[strawberry]

Nope, but no one else does either!.

 

I have setup RDS and have decent backups so if we needed to decamp we could.

[penfold]

We are currently trying to update ours as it's a bit out of date. We have a general outline which gives no real technical guidance but does provide and outline to what actions will be taken. The actual "step-by-step" procedure for getting systems back up and running was created years ago and to be honest I would guess its very out of date. It's one of the things we are trying to improve so people are aware (and acknowledge) the Data retention and recovery time scales so if something does go wrong it's not IT's fault. Edited January 15, 2015 by penfold

[DGardiner]

i have multiple buildings on the same network connected to a core building, so worst case the core goes up in flames and everyone is without internet but i replicate all important files to every site. this works nice for failover - one schools server failed and they only noticed when dhcp stopped being served

[localzuk]

Can everyone vote in the poll?

[zag]

Ours is very out of date

[Sagima]

It's on my list of things to bring to the attention of whomever ends up in charge of us next month

[elsiegee40]

At the governor school, this has been a positive thing to come out of the change to a multi academy trust. The Trust's Audit and Risk committee has driven forward Disaster Recovery Plans for each school and its IT provision. Having someone cracking the whip insisting that there is one and that it is up to date and workable has focussed minds!

[JRowley]

I voted Yes. In truth it is more or less up to date with only a handful of minor changes that need to be made to it, just need to find some time!

[synaesthesia]

We do, it's short, sweet and to the point so I've put in "other" - mostly as I've seen some plans spanning over a dozen full pages of text.

Although they also seem to include policy etc, there was little actual planning for disasters contained therein. Having seen a couple like that, I can assume ours is not as full as it should be. However, my mind on this is set thus; if there's a disaster, nobody in their right mind would want to spend an entire day reading up on policies when they could be getting on with things with a nice easy to follow set of instructions/flow processes and easy reference guides for password locations, IP addresses etc.

[Sunnyknight]

At our school, our disaster recovery was:- 'Assess the damage, and salvage what we can'.

 

Brilliant plan that. At the hospital i work at it's:-

 

'Hope everything burns down so that we can start over'.

 

That's an even better one!

[Boredguy]

My disaster plan is "Refil the tea bags!!!"

 

To be honest we do have a reasonable Disaster plan, but it assume that we've not had a complete total decimation of the site that has taken out both server locations in two buildings that are as far apart as possible (and for an Arsonist to at least do the decent thing of setting fire to the old buildings we want to replace!)

[localzuk]

We do, it's short, sweet and to the point so I've put in "other" - mostly as I've seen some plans spanning over a dozen full pages of text.

Although they also seem to include policy etc, there was little actual planning for disasters contained therein. Having seen a couple like that, I can assume ours is not as full as it should be. However, my mind on this is set thus; if there's a disaster, nobody in their right mind would want to spend an entire day reading up on policies when they could be getting on with things with a nice easy to follow set of instructions/flow processes and easy reference guides for password locations, IP addresses etc.

 

Ours is 18 pages long - covering the immediate accessing of SIMS and FMS, then intermediate required services, then eventual full recovery. It includes contact numbers, serial numbers, backup locations, server explanations, download links, hardware requirements, what server is required and when, what level of technical competence is needed for different things, a point by point recovery procedure for SIMS and FMS (which can be followed by anyone who can read), then progressively less instructions for the other bits (as full recovery requires technically competent staff).

 

I don't know how a disaster recovery document can be so short - would that not mean that you're making the assumption that existing staff knowledge still exists?

[clareq]

Ours exists, but I wrote it when we were an RM school, and we're vanilla now, so I need to re-address it. We have a strong back up though, backing up to a location nearly a mile down the road, so if our virtualisation host (site 1), DR host (site 2) and back up server (site 2) all fail, we probably have a bigger problem than just the network!

[JRowley]

My disaster plan is "Refil the tea bags!!!"

 

To be honest we do have a reasonable Disaster plan, but it assume that we've not had a complete total decimation of the site that has taken out both server locations in two buildings that are as far apart as possible (and for an Arsonist to at least do the decent thing of setting fire to the old buildings we want to replace!)

 

I would say that a complete loss of the entire site would render most disaster recovery plans pointless because you would claim on insurance and then start over and do things differently. I suppose you could plan for that eventuality too but it then becomes very dependant on the decisions being made as to building layout, money, things like that.

[Domino]

I'm of a mind that as soon as it's committed to a document and approved it's out of date - certainly with the speed of changes here.

 

So it's an evolving thing; BCP and DR split out into different sections, and constantly review.

[localzuk]

I'm of a mind that as soon as it's committed to a document and approved it's out of date - certainly with the speed of changes here.

 

So it's an evolving thing; BCP and DR split out into different sections, and constantly review.

 

Difficult to do when your governors want a document to approve which contains the procedure etc...

[Domino]

Surely the doc should contain the fact that changes in infrastructure would demand changes in DR? Or the point made when the document is presented?

 

Either that or the whole thing is just: "OFFSITE BACKUPS. Luvz, Localzuk."

[Oaktech]

We have some, and if they are followed, they'll work. I'm in the process of updating them to reflect some changes that I've made that will make life easier should the worst happen.

 

Our backups are held on a waterproof, fireproof HDD, and also replicated to another external HDD in another part of the building (soon to be another site)

 

We have instructions in the fireproof safe, in the safe of the other school in the federation, on an encrypted stick in the posession of the head, and in ordinary document backups.

 

We will shortly also have the facility to simply become an extension of another schools network (and they of us) should the absolute worst happen.

[TechMonkey]

At our school, our disaster recovery was:- 'Assess the damage, and salvage what we can'.

 

That's effectively ours, with some paperwork because they had to. When you point out the major flaws you are told that you needn't worry. Those in the position to make policy feel they can direct in cases of disaster so that is their plan, they'll wing it.

 

My feeling is procedures won't change that much if done correctly, but a yearly review should be carried out. Yes the first casualty of any event is the plans but if everyone knows what should be done then they can all pull together towards that goal & there will be a "in case of change these people need to know and this will be done to allow them to know". Otherwise everyone does what they feel is best which is all well and good but doesn't lend it self to a coherent strategy!

[itgeek]

Veeam Backup and Replication

[howartp]

We're due an IT audit next month, and I suspect I'll be pushed on documenting the plan.

 

The rough plan is all in my head, and my senior tech knows most of it, but it's not on paper.

 

I did put together a disaster toolkit last year with all the installers for adobe reader, word viewer, excel viewer, IBM Storage Manager, java offline v7, putty etc etc etc and all the user manuals for the core switch, san, VM hosts, passwords, serial numbers etc.

 

(That was absolutely not a result of coming in one morning and finding both the SAN switches frozen and not knowing the IP addresses, usernames or passwords for it, and those details being held in their normal saved folders on my user account, on the SAN... Honest it wasn't.... Had to do a restore of my user area to another location to retrieve some of those, and guess the rest... the toolkit is now held both in my area AND on the remote backup server!)

[synaesthesia]

I don't know how a disaster recovery document can be so short - would that not mean that you're making the assumption that existing staff knowledge still exists?

 

No, there is an assumption that whoever is doing disaster recovery, if not us, is fully competent in the first place. I wouldn't let *anyone* near our setup if they were not experience/qualified enough to do so. You can document everything under the sun but there is absolutely no chance whatsoever it will be enough for Jo Bloggs to get up and running. The receptionist at the local A&E wouldn't be expected to get a hospital network usable again should something happen, why would it be different for us? There are 3 people in my school, myself included that I would trust with the MIS database, and none are not technical. Too much can go wrong for Jo Bloggs to go messing around with it.

 

After all, with the correct, concise information, anyone in the IT industry will be able to look at it, see everything they need to get back up and running, and do so.

Edited January 15, 2015 by synaesthesia

[kevin_lane]

Yep got one even including our off site co location server too

[bossman]

Got a BRP which is incrementally updated as the system changes, point is if there was ever a major disaster then I feel all you would require is the MIS data and the staff and students work as they would probably be hived off to other schools around the county.

 

I couldn't see another school being built within 6 Months anyway so would probably be started from scratch with brand new systems.