Mirroring Servers or any other suggestions.

[dhicks]

There are may things to consider with a mirrored setup and it really comes down to how much you are willing to spend as to how quick and easy the recovery is.

 

I think it's simply a case of buying two lots of servers, putting them in two different locations and getting as fast a network connection between them as possible. There's no need to spend any money on software, that's all available for free, just spend your cash on hardware.

 

--

David Hicks

[SYNACK]

I think it's simply a case of buying two lots of servers, putting them in two different locations and getting as fast a network connection between them as possible. There's no need to spend any money on software, that's all available for free, just spend your cash on hardware.

 

--

David Hicks

 

Yea but high speed links and hardware especially when duplicated for redundancy do end up costing a fair amount, especially if you are looking at fast and reliable SAN storage. Also I have not seen some of the features avalible on the free systems like hot spares with less than a second failover and the automated hardware provisioning stuff. For the moment at least some of the features are only avalible with software that you have to pay for.

[KWestos]

Simple solution - just Ghost the servers (may take a while to restore) but couple this with a regular backup hey presto - cheap!

 

May take a little longer than some people to get back up and running, but there is such a thing as overkill!

 

I used to work for the government and we had nothing like what some are suggesting!

[Teth]

I don't know if offering the UK government with its world renowned IT miss spending, failed projects and data losses as an upstanding example is the best idea

 

Seriously tho any disaster can be planned for but really there is a limit to what is realistic within a school budget and the realities of school life.

 

I agree entirely that a VM system with SAN and offsite backup my personal choice has been to use DAS drives for that is the way to go to allow operations to continue after significant hardware failure.

 

If its a disc error it should be minutes to recover if users even notice a disc has failed at all.

 

Server Failure it should be a matter of hours as VMs can be shifted and run on remaining functional hardware. Redundant PSUs in servers I consider a minimum spec requirement which avoids most cases this could happen.

 

Beyond this I think good offsite backs to allow a restore a day or 2 later on alternate hardware is totally acceptable. If your server room is vandalised or burns to the ground a couple of days is acceptable. I have tested a plan in house to use the desktop machines in 1 computer room to run the server VMs which would allow 90% of our network to function even if the server room was destroyed like this. It wouldn't be fast and there would be some missing functionality like the extended storage system we have for media pupils editing video but admin and basic academic use could continue a day later.

 

If you go beyond this scenario to flood or major fire consuming multiple computer suites etc 1 day recovery is not even logical. The school will not be open the day after an event like that maybe not for weeks afterward and there will be time to order new equipment and restore from backup.

 

You should be able to protect against disc and Power failure very cheaply and I would consider this a minimum spec on new server purchases. Redundant power supplies, UPS and RAID 1 for system drive RAID 5 for user space. Upon failure of 1 server the network should function fully. A loss of performance is acceptable until new hardware is purchased. Beyond that a few days to a fortnight is unavoidable and should not cause that much disruption relative to whatever event the school has fallen victim too to cause it.

[Geoff]

There's been no mention of power redundancy so far. You might want to consider the fact that if the lights aren't on using any of the above solutions, nothings going to work.

 

To get power redundancy, failover and backup you need to do a few things.

 

- Have dual UPS' in each of your racks.

- Have servers with dual power supplies, with one power supply plugged into each UPS.

- Have your switches on UPS.

- Have a large diesel generator on site.

 

So the theory is, if one of your UPS' dies in one of your racks, you can still power the servers. Yes, the switches are on single UPS' (Larger modular switches do come with dual power supplies). Losing a switch to a shouldn't be an issue. You're network should route round the failure (see above on how to do that). All the UPS' onsite need to be tuned and load tested so they have sufficient capacity to keep things going until your diesel generator kicks in (this should be automatic on power failure).

 

Be aware this is massively expensive and is usually the lowest level of any redundancy plans. So unless your planning to run a stock exchange or start a war with some other country and need to co-ordinate your military assets I wouldn't bother.

[DAckroyd]

Simple solution - just Ghost the servers (may take a while to restore) but couple this with a regular backup hey presto - cheap!

 

If you try ghosting or snapshoting a server, if it's an AD server, then you will completely screw up the network as it will end up out of sync and you will have a bigger problem to deal with than before.

[Linfit]

We looked quite hard at what would happen if we lost physical access to the server room and various parts of the campus and did the following:

 

Data backups that go off site every night. These are on portable drives, and include an AD backup.

A parallel SIMS server (running on a laptop) that also goes off site every night.

A spare server stored offsite, set up and ready to go. We used a server which we had decommisisoned from being a DC and set it up with a copy of the backup software, all the bits and bobs but with no AD.

In extremis, we could get a basic network running using this spare server and the laptop, using staff laptops as clients, which would preserve usernames etc.

 

We are talking about dire emergency here, having to operate in the car park or something like that. The bits and bobs to do this are not that expensive, and it does provide peace of mind.

 

We also set up a little baby DC in our office (which is about as far away from the server room as we can get in our building). We built this machine using a mini-ITX board; these fit in a 1U case and can be mounted in a standard comms cabinet if you get a short one - a good solution for putting a discrete DC somewhere in an obscure backwater well away from the server room. Its set up with DHCP and DNS and of course replicates, so would serve as an emergency DC on its own if required.

[dhicks]

Yea but high speed links and hardware especially when duplicated for redundancy do end up costing a fair amount, especially if you are looking at fast and reliable SAN storage.

 

I should think that a dedicated replication link of a gigabit fibre connection, maybe multiplexing 2 or 4 fibre connections, would cover most school's needs. If you are duplicating hardware anyway then you really have no need for a SAN, simply use all your disks as local storage.

 

--

David Hicks

[richardp]

It comes down to budget and what kind of disasters you are trying to plan for.

 

I used to be the network operations manager at a manufacturing company in the late 90s early 00s before I started working in education. We used an open vms system that had a cluster of 3 physical servers (one of which was as far from the other two as possible) plus a seperate scsi based storage device for all the data. This system would automatically compensate for any server that had a problem so that the users would not know anything was wrong unless all three servers died at the same time. It worked flawlessly for years until we replaced it with a new unix based system running on rs6000 servers.

 

However on top of that we also had a contract with a disaster recovery company. If we had a major fire or a satellite fell on us we would call them up and they would arrive on site with a number of containers the same day. These containers would have the same servers and client computers as we had installed and act as offices for staff. All we needed was our most up to date backup tape. Now obviously that cost a fortune, can't remember the exact numbers but £30,000 a year would be about somewhere near the mark. Plus we had to test the system once a year by going down to their offices in Birmingham and restoring out system from tape.

 

If the main focus is to avoid down time due to hardware failure then clustering / virtualisation / raid / disk and tape backup / offsite replication/ generators/ ups etc... are the areas to research. Vandalism or burglary etc... may lead you down the path of spending money on better security. Some people don't even have their servers in a locked room !!! Destruction of the site is a whole other ball game, not much point in spending £100,000 on spare servers and switches etc... stored offsite (don't forget client workstations) if there is no school to work in. The only way to be 100% safe is to have a whole other school sat idle waiting for you but even that would do you no good without reliable frequent comprehensive backups.

 

Richard

[KWestos]

If you try ghosting or snapshoting a server, if it's an AD server, then you will completely screw up the network as it will end up out of sync and you will have a bigger problem to deal with than before.

 

Out of sync? If I were to use a NAS which took a complete backup of our PDC and BDC every weekend. I don't see what the problem would be if I were to re-image the two servers (even if only one were to fail). Please correct me if I'm wrong

[DAckroyd]

Out of sync? If I were to use a NAS which took a complete backup of our PDC and BDC every weekend. I don't see what the problem would be if I were to re-image the two servers (even if only one were to fail). Please correct me if I'm wrong

 

the problem comes when you take the image of the active directory back to a time that isn't current and you loose connection to the other DC's through the secure link through which the servers communicate - the same happens if the clocks set themselves to different times on your servers

[Geoff]

You can get round that problem by doing an authoritative restore.

[SYNACK]

I should think that a dedicated replication link of a gigabit fibre connection, maybe multiplexing 2 or 4 fibre connections, would cover most school's needs. If you are duplicating hardware anyway then you really have no need for a SAN, simply use all your disks as local storage.

 

Depends on how much data you are moving, I have 4 GB links to my SAN for two servers in addition to data on large arrays in seporate servers. You can replicate the disks into other servers as well but to get the same level of speed and reliability out of that solution you will need many more disks, interconnects and complexity which all burn power. In the end your really just spending the money in a different way. Time and power as opposed to higher end hardware, there is also heat (AC) and space to take into account. Both are completely valid solutions but comparitivly I think that they probably work out a lot closer in effective cost than you would suspect.

[DAckroyd]

as the point of uping the power usage for doubing up of kit has come up, I was just wondering if that would come in as an issue over the road with you in KesTowers or are you better off with your power supply that we are at FaultyTowers here?

[Geoff]

We blew the 350A fuse a little while ago in one of our phases if that counts.