Testing / Avoiding unscheduled downtime

[User3204]

My boss / the bursar, has been getting at me (admittedly due to my errors), because we've had some unscheduled downtime (IE the network/servers broke), one due to my buggering around with switches, and one due to config being set wrong on the VM cluster (two years ago).

 

But she sees them as the same problems:

 

1. my lack of planning (okay fair due).

2. my lack of any tessting network.

3. the fact that I'm the only one who knows how some stuff works, so the other techs can't fix what I've broken.

 

So, a multipart question for you:

 

 

How do you test network changes ? As I haven't found a good way of testing for 60 odd (or normal) students logging into computers over a 2 minute period....

 

How do you go about making sure the rest of the techs / engineers, know how stuff works ?

I know documentation helps, but it's the troubleshooting aspect of it that is slowing stuff down. I've started writing a list of "If X, then try Y, else do Z". But these are supposedly intelligent people, they KNOW this stuff, it's the complicated stuff that stops them/me (nobody likes restarting the Server cluster, as it takes abotu 2 hours to come back up, but this fixes ALL problems).

 

 

 

Actually, if I'm not careful this'll turn into a bit of a rant, as one of the tech has just left and there's no plan to replace, so it's up to me to arrange some process for overcoming the understaffing problems that will ensue (and the fact we no longer have all the expertise we need). Moan, moan, moan.

[pete]

How do you test network changes ? As I haven't found a good way of testing for 60 odd (or normal) students logging into computers over a 2 minute period....

 

Using NetSupport (or insert free desktop monitoring app here) and a test student account, ideally with a follow-up script to load a 500Mb .flv from a central server in VLC. We can dump about 100 students at once from one PC running the Netsupport teacher module.

 

For more "I've changed X, what are the implications / results?". Cacti, cacti and more cacti. Before-and-after logs.

 

Copious notes, backout plans and sufficient time to test it properly. Never do anything on a Friday (Thursday is deploy day for us), never make changes when you're tired.

 

How do you go about making sure the rest of the techs / engineers, know how stuff works ?

 

A Wiki. How stuff is configured, why it's configured that way, what the implications are of changing $foo. Solutions to common problems, solutions to in-depth problems, solutions to things that should never happen again but probably will if I don't write down the solution while I remember it. Times when they should step back, break out the emergency black cockerel and do nothing else until I arrive.

 

Decent current books on the technology in use. Best practises guides. Their own VM playgrounds in which they can break things and learn what not to do.

 

Ultimately, you need to give the techs the time to play (and break stuff) on non-production kit to build their confidence.

 

Were I you, I'd probably say "well, it'd help if they had some training on $thing_only_I_can_do". If they protest, point out they've just saved several thousand by not planning to replace $guy_who_left, so it's unreasonable to expect other techs to pick up his slack and cover for you without adequate training.

[Eric]

Using NetSupport (or insert free desktop monitoring app here) and a test student account, ideally with a follow-up script to load a 500Mb .flv from a central server in VLC. We can dump about 100 students at once from one PC running the Netsupport teacher module.

 

Pete, that sounds interesting. Please would you explain this process in more detail? I'd like to add it to my armoury. Personally I've never been able to login a classroom all at once, but it sounds as if you can do it. And the vnc script is interesting too ...

[sparkeh]

How do you test network changes

In terms of software, GPO, user account changes and things like that: As much as I can I test these things in a test area. For example I have an an AD OU that is separate from the production area but basically a mirror of the production area with test accounts, test machines and copies of all the GPOs. If the change works out OK then the changes are made to the production area. If the change needs some real world testing then, if possible, I co-opt a few willing participants do things like filter an updated GPO to their machine\account or something like that.

 

Hardware can be more tricky and usually requires an agreed, or possibility of, downtime (usually in the holidays).

[BHMS]

When you can't work on hardware work out of working hours or in the holidays then you don't really have much choice and unexpected things are bound to happen. If they are uptight about it say you are fixing the problem rather than creating it, they won't know any better

[sonofsanta]

Windows updates get deployed to our office first (usually when released), and then rolled out en masse at holidays, unless absolutely critical. Likewise software updates.

 

Test OU's are essential, as are fake users in various places.

 

I nearly always wait for lunch or after school to do anything big, as well - doesn't matter so much if something falls over then as it does in the middle of a lesson.

 

Regarding knowledge sharing, as often as possible when I'm doing something that I think only I might know, I call over any one else in the office to at least show them. If it's anything more in depth, step by step guides get drawn up in Word. I like pete's idea of a wiki but I fear I'd never keep it up to date. We do leave clues for ourselves lying around the solutions section of the helpdesk, though.

 

Ultimately, of course, no plan survives contact with the enemy, and no amount of testing will ever shake out every issue, so it does seem unreasonable to hold two mistakes in two years over your head. Presumably you have learnt your lesson regarding clicking shiny buttons in the middle of the work day.

[pete]

Pete, that sounds interesting. Please would you explain this process in more detail? I'd like to add it to my armoury. Personally I've never been able to login a classroom all at once, but it sounds as if you can do it. And the vnc script is interesting too ...

 

We have a product called Netsupport School*. One of the things you can do using the teacher console is log on any account you specify to multiple computers, provided said account has the necessary rights to do so.

 

If I log onto a machine with the teacher console installed, connect Netsupport to three rooms of computers (roughly 100 machines) and tell Netsupport I want to login to all of them as "BandwidthTestKid" / WhateverThePasswordIs, it will do so. You can also lock out the local peripherals to prevent people messing with the logged on clients.

 

Anything you want to run on the clients has to be accessible from the client**. So if you want VLC to open \\server\share\bandwidthtests\video.mp4, "BandwidthTestKid" has to be able to access it.

 

I think you've misparsed "VLC" (the media player) for "VNC".

 

I have a post-login script in Start Menu > Startup on that account's Start Menu that's basically a VLC playlist of Media Studies videos from a central server.

 

 

* There is a free alternative called (iirc) ABTutor or something (have a search on here).

** You can push stuff, but it's easier for the client to grab it itself.

[Eric]

Pete, thank you for the heads up about ABTutor - I'll go and give it a try.

 

Update: AFAICS ABTutor isn't free, but Italc looks as if it could do the business.

 

I think you've misparsed "VLC" (the media player) for "VNC".

 

I did - bother.

 

I'm going to add my own suggestion for trying (some) things out in a sandbox. I've just got a shiny new Vmware setup and I'm backing up VMs with Veeam. The more expensive version of Veeam has a facility for restoring a bunch of VMs inside a sandbox built out of Vmware virtual switches and a "double-NATing router," which runs as yet another VM. The special router means that the sandboxed hosts have access to e.g. the internet and any other production hosts that they need, but they don't interfere with the rest of the production hosts.

 

Since I don't have the expensive Veeam , I'm going to try doing this by hand, then I might try automating it using Poweshell, which I understand integrates well with Veeam.

Edited September 30, 2011 by Eric
More information.

[pete]

Pete, thank you for the heads up about ABTutor - I'll go and give it a try.

 

Update: AFAICS ABTutor isn't free, but Italc looks as if it could do the business.

 

Yeah, that's the one - knew there was one but was unsure of the name.