We looked quite hard at what would happen if we lost physical access to the server room and various parts of the campus and did the following:
Data backups that go off site every night. These are on portable drives, and include an AD backup.
A parallel SIMS server (running on a laptop) that also goes off site every night.
A spare server stored offsite, set up and ready to go. We used a server which we had decommisisoned from being a DC and set it up with a copy of the backup software, all the bits and bobs but with no AD.
In extremis, we could get a basic network running using this spare server and the laptop, using staff laptops as clients, which would preserve usernames etc.
We are talking about dire emergency here, having to operate in the car park or something like that. The bits and bobs to do this are not that expensive, and it does provide peace of mind.
We also set up a little baby DC in our office (which is about as far away from the server room as we can get in our building). We built this machine using a mini-ITX board; these fit in a 1U case and can be mounted in a standard comms cabinet if you get a short one - a good solution for putting a discrete DC somewhere in an obscure backwater well away from the server room. Its set up with DHCP and DNS and of course replicates, so would serve as an emergency DC on its own if required.
It comes down to budget and what kind of disasters you are trying to plan for.
I used to be the network operations manager at a manufacturing company in the late 90s early 00s before I started working in education. We used an open vms system that had a cluster of 3 physical servers (one of which was as far from the other two as possible) plus a seperate scsi based storage device for all the data. This system would automatically compensate for any server that had a problem so that the users would not know anything was wrong unless all three servers died at the same time. It worked flawlessly for years until we replaced it with a new unix based system running on rs6000 servers.
However on top of that we also had a contract with a disaster recovery company. If we had a major fire or a satellite fell on us we would call them up and they would arrive on site with a number of containers the same day. These containers would have the same servers and client computers as we had installed and act as offices for staff. All we needed was our most up to date backup tape. Now obviously that cost a fortune, can't remember the exact numbers but £30,000 a year would be about somewhere near the mark. Plus we had to test the system once a year by going down to their offices in Birmingham and restoring out system from tape.
If the main focus is to avoid down time due to hardware failure then clustering / virtualisation / raid / disk and tape backup / offsite replication/ generators/ ups etc... are the areas to research. Vandalism or burglary etc... may lead you down the path of spending money on better security. Some people don't even have their servers in a locked room !!! Destruction of the site is a whole other ball game, not much point in spending £100,000 on spare servers and switches etc... stored offsite (don't forget client workstations) if there is no school to work in. The only way to be 100% safe is to have a whole other school sat idle waiting for you but even that would do you no good without reliable frequent comprehensive backups.
You can get round that problem by doing an authoritative restore.
as the point of uping the power usage for doubing up of kit has come up, I was just wondering if that would come in as an issue over the road with you in KesTowers or are you better off with your power supply that we are at FaultyTowers here?
We blew the 350A fuse a little while ago in one of our phases if that counts.