Mirroring Servers or any other suggestions.

[ninjabeaver]

I was in the usual ICT weekly meeting this morning and the Deputy Heads has got me all twitchy.

 

Basically he asked how long it would take to get the PDC or any server back up and running if we had a disaster.

 

I replied approximately 2 to 3 days as long as everything went well. Reply to that was that it needs to be backup and running within a day or less.

 

I promised that I would look into it and one of my suggestions was mirroring all the servers onto a Virtual Server. I have no idea if this would work or even how to go about it.

 

Also any other suggestions please? I'm thinking that money would not be too much of an issue, but I don't want to go stupid on it as I have plenty of other plans on the go that need to be implemented to bring the school up to date.

 

Thanks in advance everyone,

[Domino]

2 to 3 days is a reasonable time span.

 

Yes, a virtual copy would help - but what if the server room burnt down?

 

Not much cop then - an offsite backup and some hastily borrowed servers would be more use

[strawberry]

virtualise as many roles as you can and have 2 separate servers with the ability to run all the servers at once, possibly with redundancy built into the servers ( so for istance having 2 dhcp servers live at any one time) but only running with half of the servers during normal operations. Place them in seperate buildings if possible. Have each server backup to the other one and to a tape.

 

some servers will be tricky to virtualise en masse, just ensure they're well backed up and if possible have some spare machinery to restore to.

[teejay]

Time for a disaster recovery plan me thinks!

One thing we've done is split the servers between two rooms in seperate buildings, so that if one building goes down we can still run from the other building.

The other idea I've been thinking about is to maybe club together with other schools in the LEA to have 1 or 2 spare servers capable of running a few virtual machines kept at the LEA data center so that if disaster struck we could grab these servers and get the core services back up and running quickly. This would be especially useful for stuff like getting SIMS back up quickly.

[ninjabeaver]

2 to 3 days is a reasonable time span.

 

Yes, a virtual copy would help - but what if the server room burnt down?

 

Not much cop then - an offsite backup and some hastily borrowed servers would be more use

 

Yeah, I was a bit taken back when he said under a day. I can understand why though, as EVERYTHING we do seems to be based on computers now from reports onwards. Grrr.

[FN-GM]

I have managed to get a PDC wish DHCP file shares printers and stuff back up fully in a few hours. The problem with that is they may expect that next time...

[IanT]

Clustering

[ninjabeaver]

Tell me about clustering then (well point me to a website if you can so I can read up on it)

 

Any starting guides to virtualiation as well?

 

Cheers.

[Sylv3r]

I'm pretty impressed the Deputy head seems to know what he is talking about.

[dhicks]

Also any other suggestions please? I'm thinking that money would not be too much of an issue

 

Fret not, this is exactly what DRBD is for. It mirrors block devices (disks, partitions, RAID arrays, encrypted partitions) between machines over a standard TCP/IP network. The documentation is comprehensive and well written, check the website. I use DRBD in conjunction with the Xen virtual machine system running on CentOS, all of which is free. Just download the CentOS CD images and install on your server - the install wizard even gives you an option to select to install Xen with no further setup needed.

 

DRBD works as a standard Linux disk driver, so should work with any Linux distribution or VM system you want. It will help performance if you have a dedicated network card in each machine for DRBD to use for mirroring. Don't worry too much, though - people seem to forget that disk reads generally far outweigh disk writes in normal operation, and it's only the disk writes DRBD needs to send across the network.

 

I'd say split your machines between two locations in your school and aim to get as fast a network connection between them as you can - it might be worth spending some of that spare cash on a couple of decent switches linked with fibre so you can mirror drives. Also worth investing in decent UPS', ones that don't trip out during power surges.

 

The Xen documentation suggests that live switch-over of a running VM is possible with around 0.8 seconds of downtime, although I am not inclined to go and unplug one of the servers to test this right now.

 

--

David Hicks

[Michael]

One question no one has asked (amazingly) is what type of setup do you currently have?

 

I can only presume you operate Active Directory, DNS, DHCP and that it also acts as a File and Print server. What other roles does this server have?

 

- How many users are in your network?

- How many hard drive(s) do you have in your server and how much disk space (roughly) are you using?

- What backup solution(s) do you currently use?

 

As a recommendation RAID is a cost effective way of adding redundancy to a server. There are different types of RAID, however I would of thought RAID1 or RAID5 would most likely be suitable for most setups. The most common point of failure in a server (in my opinion) are the hard drives themselves, so it does make a lot of sense to implement RAID1 at least on the system drive, which would host Windows, Active Directory, DNS and DHCP.

 

Installing Windows 2003 from scratch, any required drivers and then performing a System State restore can easily take a few hours itself to restore. If you implement at least RAID1 you do minimise the risk of building from scratch dramatically.

 

This then leaves just restore user data, network shares and application(s) you may be hosting. Backing up to tape or a NAS is ideally what you should be looking at. NAS boxes are more popular these days and restoring data is also much faster too. The advantage of NAS also is that you can strategically install it anywhere in the building, away from a server room, but also the cost is much lower than a tape drive per GB. It's not uncommon (even for small schools) to have a couple hundred of Gigabytes and several Terabytes in Secondary schools.

[localzuk]

First thing I'd ask is "What's the budget for equipment and software to be able to implement a less than 1 day disaster recovery plan"? Just because someone high up demands it, doesn't mean it is possible, unless appropriate funding is made available.

 

For example, the best way, in my mind to deal with this would be to have

 

a) all servers virtualised, and the disks stored on a SAN.

b) you take snapshots and store them on a second SAN, somewhere else on-site

c) you have a set of redundant spare servers sat waiting somewhere - presumably in the same place as the secondary SAN. (or as dhicks mentions mirror the block devices, but I can see this as being a bottleneck possibility).

d) you have dual core switches for the servers, and all edge switches connect to both, and possibly via different routes. A restored server is not much use if you don't have a network to connect across...

 

A thing to remember here is that you should have this formalised in a disaster recovery plan.

[DMcCoy]

The server room floods or there is a fire.

 

How long now?

 

You can't eliminate disruption in the event of a disaster, the closer you get the more it's all going to cost.

[ssiruuk2]

Neverfail can mirror windows servers and allow a passive server take over if the production one fails. No idea on pricing but might be worth a look - supports domain controllers, exchange, file servers etc.

 

Neverfail - Products

 

On the disaster recovery side obviously would be great if the backup server sits in another building

Edited February 24, 2009 by ssiruuk2

[ninjabeaver]

Thanks to everyone so far. Plenty of reading there. Thanks David for you comments and also Michael and Localzuk.

 

I'll also be looking at ssiruuk2 idea as well. For the moment I just want to get some stuff on paper and present it to the Deputy Head. I'm all for it, but I don't think any of the school staff understand it's not just a one button push for a rebuild.

 

Any more suggestions are more than welcome.

[Butuz]

There are three questions you need to ask your deputy head.

 

1. How much data can the school afford to lose? I.e all of it, some of it, none of it.

2. How long can each system be down during an emergency? 2 weeks? 5 days? 1 day?

3. Based on the above answers - will a suitable budget be provided, not just initially but yearly?

 

E.g. 2 weeks to up time will require little cost really, just a backup drive, tapes, and server warranty, or the ability to order one or more new servers should the old ones fail. So say roughly £10k.

 

1 Day uptime will require Virtualisation, Storage Area Networks, multiple seperate server rooms, etc etc. Now you can easily be talking £100k+ for an average Secondary or anything up to £500k for a very large ICT heavy school/college.

 

So you can either say to your deputy - give me your requirements and I'll give you a cost. Or you can say give me your cost and I'll give you your requirements. Both will work.

 

Butuz

[broc]

Years ago when I worked in the mainframe industry, we used to carry out a regular exercise called 'Component Failure - Impact Analysis' or CFIA for clients.

 

This exercise pulled together key skills from all areas of the centre and we went through each 'component' (hardware & software) and reviewed the type of failure they may suffer, the impact, and the recovery/repair times. You also ranked the failures in terms of how likely they were to happen.

 

This enabled us to identify those things that needed additional redundancy, those things that could be reconfigured in an emergency, and those things that were not cost effective to plan for.

 

So for example, you may decide to switch to RAID storage to provide redundancy from disk failure as it is likely and relatively cheap, but you may decide not to seriously consider the loss of the server room & the building that houses it because it is unlikely & very expensive!

 

You need to consider the wider failure scenarios too, including environmental damage due to power cuts, flood, building destruction, criminal damage,theft etc. You end up with a list of things that can be done to mitigate a disaster, those things that need spending money on, and those things not worth worrying about. These then need to be costed and presented to management who will hopefully agree to fund the high risk ones & accept you cannot (at least on a school budget) plan for every eventuality.

 

If you are paranoid, you may want to consider things like 'are you under the flightpath of a major international airport or have you a major petro-chemical plant next door' ........

[dhicks]

mirror the block devices, but I can see this as being a bottleneck possibility

 

No, DRBD doesn't act as a bottleneck, it will simply get a little behind when very busy and catch up when it gets a chance. This does mean you want a decent UPS on both machines so they have a chance to shut down properly if possible.

 

Don't know if anyone else has mentioned this too but you can get servers with dual power supplies, which after harddrives are probably the most common component to fail.

 

If you're mirroring instances of Windows in real-time you'll be needing Windows licenses on both machines, even if you're only running one at a time. You'd probably want to get Windows Enterprise or Datacentre edition.

 

I'm sure you could spend £100,000+ on doing this all very well, but this is all perfectly manageable for way less. I doubt we've spent more than £20,000 on all our servers, switches and wiring in the two years I've been here.

 

--

David Hicks

Edited February 25, 2009 by dhicks

[DAckroyd]

I was in the usual ICT weekly meeting this morning and the Deputy Heads has got me all twitchy.

 

Basically he asked how long it would take to get the PDC or any server back up and running if we had a disaster.

 

I replied approximately 2 to 3 days as long as everything went well. Reply to that was that it needs to be backup and running within a day or less.

 

I promised that I would look into it and one of my suggestions was mirroring all the servers onto a Virtual Server. I have no idea if this would work or even how to go about it.

 

Also any other suggestions please? I'm thinking that money would not be too much of an issue, but I don't want to go stupid on it as I have plenty of other plans on the go that need to be implemented to bring the school up to date.

 

Thanks in advance everyone,

 

Our servers went tits up a few weeks ago - system down for 2 and a half days (multiple problems ocuring at the same time)

 

the question of how long until it's up again is a big question - depend on the problem - I could in theory have a new set of servers built from scratch within a day but it all depends on backup and stuff.

 

if you want to see what setup we have here now with the servers now virtualised and having been through the process of recovering from a disarster resently?

[Bezwick]

I have been toying with the idea of a Large NAS drive and this imaging software.

 

Online data backup solution for Windows servers

 

Has the ability to recover an entire server to a new one in a few hours even with different hardware, also could even transfer to a virtual server. CAn do full and Incremental backups of the images.

 

I know a technician in a school down the road who swears by it. Might be worth a look into.

 

[SYNACK]

There are may things to consider with a mirrored setup and it really comes down to how much you are willing to spend as to how quick and easy the recovery is.

 

There are many options, primarily though you would be after a large shared storage array (SAN) with constant replication to a SAN in your backup location http://www.cisco.com/web/partners/pr67/pr29/solutions/downloads/HP_StorageWorksMSR_Solutio_Brief.pdf

 

On top of this you want a virtualisation environment that can provide automatic fail over. Again, how gracefully this occours is up to your budget. You can go for Windows 2008 Hyper-V with failover clustering. This will boot up a backup copy of the VM at your remote site when the primary fails but users may loose some data and you must wait while it boots.

http://www.intelalliance.com/microsoft/assets/HighAvailabilityXeon_Final2.pdf

 

The next option is a fully speced out VMware solution which can offer live spares which run the VMs synced up at each site and users can fail over without even noticing as the system is effectivly duplicated on the secondary server including memory state.

 

The next thing that you must look at is the network infrastructure that supports this and work to avoid any single point of failure by running switches in parallel with multiple links to avoid any one bit of equipments failure from bringing the system down.

 

As you can really spend as much money as you have on adding redundency it is important to have clear goals of what you want to achive before you start then pick the best solution for that level of disaster proffing.

 

Here's hp's marketing departments vision of disaster proofing:

[ame=http://www.youtube.com/watch?v=WFp-V_WRHxQ]YouTube - HP Disaster Proof Data Center[/ame]

[DAckroyd]

the best role to work to when designing a system is: Why buy 1 when you can get 2 for double the price?

 

doubling up on everything is fair enough as long as you have the infrastructure to work with that. I would love to have a few locations with servers in linked up with multi-10Gbe networking but (firstly we dont have that sort of money) the fibre is a low quality gigabit max stuff

[box_l]

acronis also do something where you put a client on the server, make an image to a remote (other building) file server and then leave the client resident.

 

it will update the image on the fly so if you server dies all you need to do is replace the box and pull the image back down.

 

they also include a tool for restoring to dissimilar hardware.

 

Online data backup solution for Windows servers

 

box

Edited February 25, 2009 by box_l
spelling

[nicholab]

You backup SAN can be virtualized lefthand and datacore have solutions. The big vendors are not best suited for schools.

[IanT]

Different subject really, my school had a power cut during the night, I came in and found part of the network down.........All servers were fine as they were on teh UPS, but my core switches went down.........need to get them on the UPS.

 

Make sure you get a nice powerful UPS installed.