+ Post New Thread
Results 1 to 9 of 9
How do you do....it? Thread, Testing / Avoiding unscheduled downtime in Technical; My boss / the bursar, has been getting at me (admittedly due to my errors), because we've had some unscheduled ...
  1. #1
    User3204's Avatar
    Join Date
    Aug 2006
    Location
    Wirral
    Posts
    769
    Thank Post
    55
    Thanked 66 Times in 62 Posts
    Rep Power
    34

    Testing / Avoiding unscheduled downtime

    My boss / the bursar, has been getting at me (admittedly due to my errors), because we've had some unscheduled downtime (IE the network/servers broke), one due to my buggering around with switches, and one due to config being set wrong on the VM cluster (two years ago).

    But she sees them as the same problems:

    1. my lack of planning (okay fair due).
    2. my lack of any tessting network.
    3. the fact that I'm the only one who knows how some stuff works, so the other techs can't fix what I've broken.

    So, a multipart question for you:


    How do you test network changes ? As I haven't found a good way of testing for 60 odd (or normal) students logging into computers over a 2 minute period....

    How do you go about making sure the rest of the techs / engineers, know how stuff works ?
    I know documentation helps, but it's the troubleshooting aspect of it that is slowing stuff down. I've started writing a list of "If X, then try Y, else do Z". But these are supposedly intelligent people, they KNOW this stuff, it's the complicated stuff that stops them/me (nobody likes restarting the Server cluster, as it takes abotu 2 hours to come back up, but this fixes ALL problems).



    Actually, if I'm not careful this'll turn into a bit of a rant, as one of the tech has just left and there's no plan to replace, so it's up to me to arrange some process for overcoming the understaffing problems that will ensue (and the fact we no longer have all the expertise we need). Moan, moan, moan.

  2. #2


    Join Date
    Dec 2005
    Location
    In the server room, with the lead pipe.
    Posts
    4,686
    Thank Post
    279
    Thanked 784 Times in 611 Posts
    Rep Power
    225
    Quote Originally Posted by User3204 View Post
    How do you test network changes ? As I haven't found a good way of testing for 60 odd (or normal) students logging into computers over a 2 minute period....
    Using NetSupport (or insert free desktop monitoring app here) and a test student account, ideally with a follow-up script to load a 500Mb .flv from a central server in VLC. We can dump about 100 students at once from one PC running the Netsupport teacher module.

    For more "I've changed X, what are the implications / results?". Cacti, cacti and more cacti. Before-and-after logs.

    Copious notes, backout plans and sufficient time to test it properly. Never do anything on a Friday (Thursday is deploy day for us), never make changes when you're tired.

    How do you go about making sure the rest of the techs / engineers, know how stuff works ?
    A Wiki. How stuff is configured, why it's configured that way, what the implications are of changing $foo. Solutions to common problems, solutions to in-depth problems, solutions to things that should never happen again but probably will if I don't write down the solution while I remember it. Times when they should step back, break out the emergency black cockerel and do nothing else until I arrive.

    Decent current books on the technology in use. Best practises guides. Their own VM playgrounds in which they can break things and learn what not to do.

    Ultimately, you need to give the techs the time to play (and break stuff) on non-production kit to build their confidence.

    Were I you, I'd probably say "well, it'd help if they had some training on $thing_only_I_can_do". If they protest, point out they've just saved several thousand by not planning to replace $guy_who_left, so it's unreasonable to expect other techs to pick up his slack and cover for you without adequate training.

  3. #3

    Join Date
    Apr 2009
    Location
    London
    Posts
    59
    Thank Post
    4
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Testing using Netsupport

    Quote Originally Posted by pete View Post
    Using NetSupport (or insert free desktop monitoring app here) and a test student account, ideally with a follow-up script to load a 500Mb .flv from a central server in VLC. We can dump about 100 students at once from one PC running the Netsupport teacher module.
    Pete, that sounds interesting. Please would you explain this process in more detail? I'd like to add it to my armoury. Personally I've never been able to login a classroom all at once, but it sounds as if you can do it. And the vnc script is interesting too ...

  4. #4

    sparkeh's Avatar
    Join Date
    May 2007
    Posts
    7,038
    Thank Post
    1,376
    Thanked 1,782 Times in 1,202 Posts
    Blog Entries
    22
    Rep Power
    532
    Quote Originally Posted by User3204 View Post
    How do you test network changes
    In terms of software, GPO, user account changes and things like that: As much as I can I test these things in a test area. For example I have an an AD OU that is separate from the production area but basically a mirror of the production area with test accounts, test machines and copies of all the GPOs. If the change works out OK then the changes are made to the production area. If the change needs some real world testing then, if possible, I co-opt a few willing participants do things like filter an updated GPO to their machine\account or something like that.

    Hardware can be more tricky and usually requires an agreed, or possibility of, downtime (usually in the holidays).

  5. #5

    Join Date
    Jun 2009
    Location
    Poole
    Posts
    148
    Thank Post
    4
    Thanked 40 Times in 30 Posts
    Rep Power
    20
    When you can't work on hardware work out of working hours or in the holidays then you don't really have much choice and unexpected things are bound to happen. If they are uptight about it say you are fixing the problem rather than creating it, they won't know any better

  6. #6

    sonofsanta's Avatar
    Join Date
    Dec 2009
    Location
    Lincolnshire, UK
    Posts
    5,147
    Thank Post
    919
    Thanked 1,528 Times in 1,039 Posts
    Blog Entries
    47
    Rep Power
    693
    Windows updates get deployed to our office first (usually when released), and then rolled out en masse at holidays, unless absolutely critical. Likewise software updates.

    Test OU's are essential, as are fake users in various places.

    I nearly always wait for lunch or after school to do anything big, as well - doesn't matter so much if something falls over then as it does in the middle of a lesson.

    Regarding knowledge sharing, as often as possible when I'm doing something that I think only I might know, I call over any one else in the office to at least show them. If it's anything more in depth, step by step guides get drawn up in Word. I like pete's idea of a wiki but I fear I'd never keep it up to date. We do leave clues for ourselves lying around the solutions section of the helpdesk, though.

    Ultimately, of course, no plan survives contact with the enemy, and no amount of testing will ever shake out every issue, so it does seem unreasonable to hold two mistakes in two years over your head. Presumably you have learnt your lesson regarding clicking shiny buttons in the middle of the work day.

  7. #7


    Join Date
    Dec 2005
    Location
    In the server room, with the lead pipe.
    Posts
    4,686
    Thank Post
    279
    Thanked 784 Times in 611 Posts
    Rep Power
    225
    Quote Originally Posted by Eric View Post
    Pete, that sounds interesting. Please would you explain this process in more detail? I'd like to add it to my armoury. Personally I've never been able to login a classroom all at once, but it sounds as if you can do it. And the vnc script is interesting too ...
    We have a product called Netsupport School*. One of the things you can do using the teacher console is log on any account you specify to multiple computers, provided said account has the necessary rights to do so.

    If I log onto a machine with the teacher console installed, connect Netsupport to three rooms of computers (roughly 100 machines) and tell Netsupport I want to login to all of them as "BandwidthTestKid" / WhateverThePasswordIs, it will do so. You can also lock out the local peripherals to prevent people messing with the logged on clients.

    Anything you want to run on the clients has to be accessible from the client**. So if you want VLC to open \\server\share\bandwidthtests\video.mp4, "BandwidthTestKid" has to be able to access it.

    I think you've misparsed "VLC" (the media player) for "VNC".

    I have a post-login script in Start Menu > Startup on that account's Start Menu that's basically a VLC playlist of Media Studies videos from a central server.


    * There is a free alternative called (iirc) ABTutor or something (have a search on here).
    ** You can push stuff, but it's easier for the client to grab it itself.

  8. #8

    Join Date
    Apr 2009
    Location
    London
    Posts
    59
    Thank Post
    4
    Thanked 1 Time in 1 Post
    Rep Power
    0
    Pete, thank you for the heads up about ABTutor - I'll go and give it a try.

    Update: AFAICS ABTutor isn't free, but Italc looks as if it could do the business.

    Quote Originally Posted by pete View Post
    I think you've misparsed "VLC" (the media player) for "VNC".
    I did - bother.

    I'm going to add my own suggestion for trying (some) things out in a sandbox. I've just got a shiny new Vmware setup and I'm backing up VMs with Veeam. The more expensive version of Veeam has a facility for restoring a bunch of VMs inside a sandbox built out of Vmware virtual switches and a "double-NATing router," which runs as yet another VM. The special router means that the sandboxed hosts have access to e.g. the internet and any other production hosts that they need, but they don't interfere with the rest of the production hosts.

    Since I don't have the expensive Veeam , I'm going to try doing this by hand, then I might try automating it using Poweshell, which I understand integrates well with Veeam.
    Last edited by Eric; 30th September 2011 at 03:35 PM. Reason: More information.

  9. #9


    Join Date
    Dec 2005
    Location
    In the server room, with the lead pipe.
    Posts
    4,686
    Thank Post
    279
    Thanked 784 Times in 611 Posts
    Rep Power
    225
    Quote Originally Posted by Eric View Post
    Pete, thank you for the heads up about ABTutor - I'll go and give it a try.

    Update: AFAICS ABTutor isn't free, but Italc looks as if it could do the business.
    Yeah, that's the one - knew there was one but was unsure of the name.



SHARE:
+ Post New Thread

Similar Threads

  1. Replies: 4
    Last Post: 31st March 2006, 11:11 PM
  2. Edexcel Numeracy and Literature testing software
    By tosca925 in forum Educational Software
    Replies: 2
    Last Post: 1st February 2006, 11:09 PM
  3. wiki downtime
    By russdev in forum General EduGeek News/Announcements
    Replies: 1
    Last Post: 30th December 2005, 10:17 PM
  4. Downtime coming
    By Dos_Box in forum General EduGeek News/Announcements
    Replies: 10
    Last Post: 26th July 2005, 09:16 PM
  5. Wiki Downtime
    By russdev in forum General EduGeek News/Announcements
    Replies: 1
    Last Post: 16th July 2005, 03:28 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •