Managing Downtime In A School Environment

[dhicks]

Most people on here say they don't have enough time as it is so the time spent building your own desktops from off-the-shelf parts

 

All of the above in this thread was regarding servers. For desktops, I think the best way to ensure uptime is to have a spare standing by ready to plug in. You don't need a four-hour reposnse-time garantee for that, though, just a standard replace-if-it-conks-out garantee is fine, an any "extended" garantee for longer than three years is probably not worth the bother.

 

whilst I would love to say that you can just swap any hardware around, we all know that you can't.

 

I think we might be looking at this with somewhat different ideas as to how to design a system. A broken server isn't something to fix, it's a bunch of generic components that you can quickly use to build a new server - shove a few new parts in and you have a new server. You don't have to worry about getting data off the harddrives or as that's already mirrored somewhere else and your spare processing capability is taking care of running the VMs that were running on that server - there's no need to fret about downtime if your system is designed with spare capacity, having hardware break doesn't stop your system running. If your RAID controller no longer works, chuck it out and buy a new one - this is still cheaper than paying Dell for a four-hour response time.

 

Every NM and Tech will have a slightly different skillset. As much as we would all love to say that we could build systems (and then document them, obviously) that *anybody* could come in and support, this is not always the case.

 

But this is the whole point of virtualisation - you don't have to worry about the hardware, that can be any old generic slot-together equipment, your servers and so on actually run on top of your virtual machine system, which is hardware-agnostic. Anyone can build a modern computer, you just chuck parts in a case.

[Duke]

I think we might be looking at this with somewhat different ideas as to how to design a system. A broken server isn't something to fix, it's a bunch of generic components that you can quickly use to build a new server - shove a few new parts in and you have a new server. You don't have to worry about getting data off the harddrives or as that's already mirrored somewhere else and your spare processing capability is taking care of running the VMs that were running on that server - there's no need to fret about downtime if your system is designed with spare capacity, having hardware break doesn't stop your system running. If your RAID controller no longer works, chuck it out and buy a new one - this is still cheaper than paying Dell for a four-hour response time.

 

I kind of agree with this in principal, but I think it assumes a properly configured, redundant, documented setup that in the real world none of us have. A broken server isn't something to fix, it's a bunch of generic components that you can quickly use to build a new server - for a VM host I can get behind this maybe, but are you saying you have no physical standalone servers, or that if you do then you will lose nothing by scrapping them and building a new one to replace them? If a component fails that takes down a server, in theory I can have it up and running next business day under the warranty. To me that seems vastly better than having to build a new physical server, install the OS and all the apps then restore all the data to it, again assuming the process to do all that is properly documented.

 

If everything is virtualised, my concern would be the shared storage. In theory having it mirrored between two home-built boxes would cover you, but personally I wouldn't be comfortable having no warranty, guarantees or support on those devices.

 

I can see where you're coming from - there's an appeal to having a Google/Facebook-style of network where no single server is of any importance, and if it dies there's no immediate impact. All I need to do is build up something that has the same functionality as the server that died and add it back into the 'pool' of servers or storage. My concern is that building an infrastructure like that properly isn't something most of us have the time or money to set up...

 

Chris

[SYNACK]

In reply to the OP, if I need to reboot something during the day to fix something major I do. If I get any complaints I simply outline the costs involved with creating the infrastructure nessisary to attain higher uptimes. Its not just about the hardware but the software to, at least two mail servers with synced DBs for access, at least two mirrored sets of the docs, clustered configuration for various other things all fo which add complexity and cost. Buying your storeage space three times over at least, all of the additional power, software liscences, software and setup complexity, increased management overhead to stage upgrades on all of the additional backup/cluster member servers (virtual or not), etc, etc, etc.

 

It is at this point where they can actually hear the sides of the school cheque book collapsing in from the imagined vacum inside that you ask them if they are serious and really would like to proceed towards a fully redundant system like that. If conveyed properly (the look of horror on their faces is the usual indicator) this usually quells any further outlandish requests like that for at least a few weeks.

[dhicks]

If everything is virtualised, my concern would be the shared storage. In theory having it mirrored between two home-built boxes would cover you, but personally I wouldn't be comfortable having no warranty, guarantees or support on those devices.

 

This is where "garantees" can be dangerous - no garantee will actually protect your data, that's up to you to sort out with backups and so forth. Paying a hardware support contract doesn't make your data safer, it just means you'll get some new, blank harddrives turn up within four hours.

 

My concern is that building an infrastructure like that properly isn't something most of us have the time or money to set up...

 

But that's my whole argument here, and the reason for my initial reply to your post - setting up exactly that kind of system is way easier and cheaper than you probably think. Setting up a virtual machine infrastructure with mirrored storage can be entirely with free software (I prefer Xen and DRBD running on Debian, but I'm sure other options exist), and the whole point of using cheap, widley-available hardware is that it's, well, cheap and you can replace a part by running down to your local PC shop.

 

Admitedly, the biggest barrier to having such a setup is probably the fiddlyness of installing Debian, Xen and DRBD - there's definatly a gap in the market there for a half-decent server management console built around the idea of mirrored storage. However, once you've built a couple of servers that way you realise it's actually dead easy - I reckon it should be possible to build a complete server, from scratch, starting from the point of going to the PC shop to get the bits and ending with a VM server added to the resources pool, in around 4 hours.

[ezzauk]

All our services should have 0% downdown time all year around. I am give the first Tuesday of every month, a 2 hour slot between 6-8am to do any server outages.

[somabc]

Transferability is a big one, my Windows sites could be easily handed off to someone else who could get to grips with most of the config even without documentation in a reasonably short timeframe. Linux is simply so customizeable that it can very quickly turn into an undecypherable maze of interlocking systems with enough config files to comfortably wallpaper the infinite chasums of hell.

 

Having gone into a school with a realitivly simple custom build of linux on it that had been built by one person then stumbled through by several others before I got to it the shear scale of the unknown elements is astonishing. That and the helpful linux community who take joy in just posting the man pages online make it a total horror show.

 

Give me a supported and somewhat standardised solution any day as at least that will not absorb so much time not only to build but to document and so much longer to transfer.

 

If you used a Configuration Management system such as Puppet with it's declarative language for expressing system configuration, a client and server for distributing it, and a library for realizing the configuration backed up by a Version Control System then you will have a full audit trail of what changes were made by who to which system and because you are writing recipes they will automatically be translated across platforms. Then whoever takes over your job should have no problem adapting!

[GrumbleDook]

Admitedly, the biggest barrier to having such a setup is probably the fiddlyness of installing Debian, Xen and DRBD - there's definatly a gap in the market there for a half-decent server management console built around the idea of mirrored storage. However, once you've built a couple of servers that way you realise it's actually dead easy - I reckon it should be possible to build a complete server, from scratch, starting from the point of going to the PC shop to get the bits and ending with a VM server added to the resources pool, in around 4 hours.

 

And so we get back to the point of skillset, and there is also the position that for some sites this will be a complete change of strategy on technology that might not fit into their existing investment ... and making any changes would have to be done over a long period of time and with careful change management ... and so we get back to downtime again.

[jayemm]

Ideally we have no scheduled downtime from 8:30am to 3:00pm however we have in the past had some, (2 occasions), where scheduled maintainance happens during this time, for different reasons. I think we all appreciate that 100% uptime is not financially viable for schools and discussing this with SLT is key.

 

We have an IBM virtualised system being brought online this summer using vmware. I have designed the system with no single point of failure. It wasnt cheap, but neither was it extortionate. I took extended warranties to ensure replacement parts are with us the following business day. An engineer can be with us, but as I have setup 2 similar systems before, I am confident of fitting the parts myself.

 

VM ware, as other virtualisation products, ensures that if there are any issues, there should be little/no disruption to service.

 

An SLA is also key as mentioned before. This will be an agreement with SLT as to which services are most important and give details of how long there can be a disruption to a service. This powerful document gives you the leverage to buy in better equipment or spares and if they dont like it, then the SLA needs changing.

[somabc]

If you want to set it up yourself these best practices were developed at Stanford University with contributions from the greater Puppet community.

 

Puppet - Puppet Best Practice - Puppet Labs

Puppet Tutorial for Linux: Powering up with Puppet | Bitfield Consulting

Pro Puppet: Amazon.co.uk: James Turnbull, Jeffrey McCune: Books

Edited April 13, 2011 by somabc

[SimpleSi]

There is no simple or general answer for this one - totally dependant on attitude of SLT/ability of tech staff to be flexible/cost of minimising disruption.

 

If SLT/tech staff working relationship is OK and both put themselves out at times to help the other then life is good

 

Si

[dhicks]

And so we get back to the point of skillset

 

I guess that's what decent, user-friendly systems management software is there for, along with consultancy services to advise people getting started.

 

and making any changes would have to be done over a long period of time and with careful change management ...

 

Not neccesarily - these are virtual machines we're talking about here, you can change around the underlying hardware without noticing any difference. You maybe need a consultancy service that could come in and switch things over in a weekend, or maybe a week of the summer holidays.

[Shuriken1]

I'm afraid I've not read all of the replies but I deal with downtime as follows:

 

Just do the work.

 

Obviously I don't plan any work during school hours and I try to do everything during the holidays, but if things break, they break. We've had two entire days offline here since Christmas due to aircon failure in the server cupboard, we had another day without any printers. If you explain to them why, there's not a great deal of argument for breakdowns. As for planned works, I definitely would not do it outside of my working hours. It's part of your job to do this work and it will improve the system. Try and do the high-profile jobs during time that people will notice, that way they will notice the improve and be able to accept the downtime was for a reason. Recabling a cabinet as you were saying is a tough one because you definitely need to do it but there's nothing in it for them, so they just see it as hassle. If there's an improvement that they will notice as a result then they will come around to your ways. As for the clubs during break and lunch, tell them to whistle. If there's an important problem that needs fixing then the club comes second! Personally I just weigh the situation, who's going to be inconvenienced, who's going to gain. Normal the latter is the whole school and that normally outweighs any single group.

 

If they're not willing to give you the time to do it and complain even when you've warned them well, a) that's your job and b) they're teachers - they love a good whinge. To most staff your job is a COMPLETE mystery, or at least that's the impression I get. So if you ever say you're causing them hassle, they can't begin to comprehend it. Sounds nasty, but I think that's true in a lot of cases.

 

Shuriken1.

[Duke]

If they're not willing to give you the time to do it and complain even when you've warned them well, a) that's your job and b) they're teachers - they love a good whinge. To most staff your job is a COMPLETE mystery, or at least that's the impression I get. So if you ever say you're causing them hassle, they can't begin to comprehend it. Sounds nasty, but I think that's true in a lot of cases.

 

Gave me a laugh, cheers!

[Shuriken1]

Gave me a laugh, cheers!

Haha, glad to oblige.