Out of hours ICT Support

[rrrrr]

Hi,

Sometimes critical issues can happen out of hours, What do you guys do regarding this for your school?

 

Some of the issues I have dealt with out of hours include;

*server becomes unresponsive

*Exchange server having a wobbler

*Power Cut

*Server Hard disk failure

*Overnight backups fail

*Attack alert on UTM

*Air Con fails and servers start overheating

*Head cannot access VPN and needs it urgently

 

Do you guys act when you are first alerted in evenings/weekends/bank holidays or do you just wait until Monday?

Does your school pay you overtime or do you have any agreements in place?

As with some of these issues, they are better dealt with quickly before they escalate and become 10x worse. Also, if they are dealt with out of hours you don't have every user individually contacting you to let you know something is not working and slowing you down.

 

I find these issues come in waves, I may not have one for a while and then I have multiple in a week and find myself just fixing them from home or coming in for the peace.

 

I recently went on holiday and my technician went off the same week. while I was on holiday we had a powercut, backup failing and an attack alert on my UTM.

I ended up resolving these from abroad through VPN while on holiday, as I would not be able to relax knowing these issues are active.

 

Thanks

[Davit2005]

If a member of the SMT has a problem they speaks to the Exec director who is in charge of us, most of the time he can fix the problem. If he can't then he will speak to one of the seniors like me.

 

Before when I was technically running a network I would RDP if for instance I didn't get a backup job email from the system to check everything was OK.

 

We never get paid overtime here but the department are flexible enough for us to finish a couple of hours early. If the out of hours work is scheduled and due to take more than a few hours then we take the time off in lieu.

[elsiegee40]

You have asked a very good question here.

 

Having worked in IT since the 1980s, I am used to businesses that have agreements for on call and emergency work - procedure, expectation and pay set out in my contract of employment.

 

When I started in schools, remote access was not possible and 24 hour running in schools unheard of. IT staff in schools were doing a job when they were in school and not doing it when they weren't there.

 

Gradually, things have changed:

 

1. We got he tools to monitor servers when we weren't in school... and into the habit of "quickly remoting in" to fix, update or keep an eye on things from home. It made our lives easier... and has become a rod for own backs now that...

 

2. Staff and students are increasingly using the school's IT services round the clock.

 

We are the IT team, we are expected to deal with the issues, but schools have not, generally, updated our contracts to say what expectation and remuneration are part of the deal.

 

Every IT person who does not have out of hours work written into their contract should be negotiating with their management to sort this out. Otherwise it will result in burn out, a work to rule and angry words which could easily be avoided. 24x7 working is not an expectation, but without being formally sorted it can end up being there by default.

 

And if you are applying for a new job, make sure that you get it agreed before you sign on the dotted line.

Edited August 18, 2015 by elsiegee40

[rrrrr]

All of the issues I mentioned could only be fixed by me, There is no one on SMT who would know where to start with these. I have raised this with my business manager before and the outcome was if an issue arises I should contact the business manager to get overtime cleared before I start.

 

I find myself just dealing with the issue irrelevant of overtime as;

I cannot always get hold of business manager (busy or unsociable hours)

It could be a 5min job to fix, don't know until I get into it.

If the problem is left/ignored it will probably escalate.

Just want to get issue resolved so I can enjoy the rest of my evening/weekend/holiday and not keep thinking about all hell breaking loose when I return to work.

 

I do find that I'm increasingly not being able to switch off from work, if I am off for long periods (e.g. xmas holiday) I find myself just checking all systems are functioning.

The reason for this is that I am the responsible for the network. If it goes wrong its me who feels the heat and me who has to fix it.

[pcstru]

Do you guys act when you are first alerted in evenings/weekends/bank holidays or do you just wait until Monday?

Where possible we do try and fix things soonest. We have no contractual agreement in place for this (I have raised it regularly over the years), it is done as goodwill so sometimes it is not done [*1]. We will pay overtime in some cases but as a school, we prefer TOIL (time off in lieu). On holidays I try to ensure that there is always someone on site, so team leaders should not be authorising leave for their entire teams at the same time.

 

[*1] On at least one occasion, I thought a Sunday night without email would be a positive benefit to staff and I'd rather just chill and watch a film.

[Sagima]

The IT manager is expected to deal with out of hours things as part of his contract - the monitoring system (nagios) will email him if something stops working and he decides if it can wait or not.

If he needs my or anyone else's support the we get 1.5TOIL or 1x Hourly pay equivalent. I always take the time off but the technicans take the money.

[jamesb]

Not a school, but I'm on permanent on-call. It hasn't yet been needed as the 1st and 2nd tier can usually deal with things - I'll only get called in if we get something really weird happening.

 

If I do get ccalled in, they're flexible about any time I feel like taking off in return.

[pcstru]

The reason for this is that I am the responsible for the network. If it goes wrong its me who feels the heat and me who has to fix it.

I think that it is a natural temptation to feel that problems with the infrastructure and systems are a reflection on our ability to provide robust and reliable services, i.e. that any problems will reflect badly on us. Individually, we need to be confident that what we are providing is the best within the constraints we operate within. Many schools scrimp on funding for IT and that impacts our ability to provide redundancy or up to date, warranted equipment. People in that position are vulnerable to additional stress because people naturally have a sense of ownership over the service they provide. I think the best help for that is to be plain with management about the level of service that is possible given the funding. I don't think that will ever entirely eliminate the stress but it can help. The other thing that will help in the long term is to let things fail. I don't mean by that that anyone should help failure along but when you go home, switch off and if something does go wrong, leave it. If you come in on the Monday and people are screaming - that is a good thing, it will help get the funding that will build more resilience or pay your overtime. And if no one noticed, it will help you to relax off the next time something goes wrong out of hours.

[pcstru]

Not a school, but I'm on permanent on-call. It hasn't yet been needed as the 1st and 2nd tier can usually deal with things - I'll only get called in if we get something really weird happening.

How does that permanent on call work? What if you are free diving in the north sea or just shooting the breeze with old friends over a glass of the finest malt? I'd expect some payment just to be on call because being on call means I have to be available and ready to rock. I just can't do certain things if I am to be "on call".

[rrrrr]

We luckily have a solid, well setup and up to date network with failovers in place but this does not stop things happening. Most of the issues I mentioned in original post cannot be easily prevented

 

The only options for this I can think of are;

1, Me and the technician alternate weeks on call and compensated for this.

2, A service contract with a 3rd party company for out of hours support. Don't know how this would work as they wont know our network.

3, Ignore things and let them fail. I would personally have trouble seeing this trough without stressing myself out.

 

any other suggestions? carrying on with just me being responsible 24/7 and then just paid the odd hours overtime when things do happen don't seem fair.

 

How does that permanent on call work? What if you are free diving in the north sea or just shooting the breeze with old friends over a glass of the finest malt?

I find if I'm out and about I end up VPNing in on my phone. The curse of technology, 10years ago networks were not so mission critical and complex, and there was no way I was getting work emails & alerts as well as VPN access working on my nokia 3210!

Edited August 18, 2015 by rrrrr

[WhizInTraining]

 

Do you guys act when you are first alerted in evenings/weekends/bank holidays or do you just wait until Monday?

Does your school pay you overtime or do you have any agreements in place?

As with some of these issues, they are better dealt with quickly before they escalate and become 10x worse. Also, if they are dealt with out of hours you don't have every user individually contacting you to let you know something is not working and slowing you down.

 

I find these issues come in waves, I may not have one for a while and then I have multiple in a week and find myself just fixing them from home or coming in for the peace.

 

I recently went on holiday and my technician went off the same week. while I was on holiday we had a powercut, backup failing and an attack alert on my UTM.

I ended up resolving these from abroad through VPN while on holiday, as I would not be able to relax knowing these issues are active.

 

Thanks

 

Hi,

 

At my organisation, I don't have a VPN set up, but my colleague does.

 

We try to respond to the issues we get out of hours, when we receive them (e.g. if someone e-mails us on Saturday, then if we see it, we'll respond on Saturday).

 

If any work is required out of hours, then we are able to claim that time back as TOIL.

 

It would seem to me, a bit bizarre that your technician went away the same week as you, therefore I assume leaving no one there? At my organisation, we plan around each other, so that when my colleague is away, I'm here, and Vice versa, as we shouldn't have to worry about certain tasks when we are away on holiday.

[rrrrr]

It would seem to me, a bit bizarre that your technician went away the same week as you, therefore I assume leaving no one there? At my organisation, we plan around each other, so that when my colleague is away, I'm here, and Vice versa, as we shouldn't have to worry about certain tasks when we are away on holiday.

Sorry, I left out he went off sick, not holiday

[WhizInTraining]

Sorry, I left out he went off sick, not holiday

 

Ah,

 

Well that's a different matter then. That couldn't be avoided.

 

In my opinion, it depends on how frequent you have to perform out of hours tasks. Is it every month? Also it depends on how long it takes to resolve the task. If a task takes a mere 5 minutes to solve, then I don't even bother claiming it back as TOIL. If however, I was working for a long period of time out of hours, I would certainly claim that time back.

[jamesb]

How does that permanent on call work? What if you are free diving in the north sea or just shooting the breeze with old friends over a glass of the finest malt? I'd expect some payment just to be on call because being on call means I have to be available and ready to rock. I just can't do certain things if I am to be "on call".

 

It means that they can call me at any time. I'm free to do whatever I want, and if I'm not in a suitable state to respond I can decline (and if they can't get hold of me the same), in which case they'll call my counterparts in the States, Europe or Australasia instead. That's their second resort because if it turns out to be something which requires physical presence it can get expensive moving people around.

 

One of the global team will always be available at any time, thanks to time zones and shifts. At the moment I'm the only one in the UK - but that'll change as time goes on.

Edited August 18, 2015 by jamesb

[HarryMonkey]

Once I leave here at the end of the day, that's it. I've left. I wouldn't know about any issues until I went back into school and no-one in school has my number so I'm not contactable anyway.

[minimoo]

What happens with physical access ? (surely site manager needs to un-alarm/unlock school?)

[howartp]

What happens with physical access ? (surely site manager needs to un-alarm/unlock school?)

I was going to ask this in a new thread last week.

 

I've seen people on here who have keys to their schools, but there's others who don't.

 

What's the norm?

[rrrrr]

What happens with physical access ? (surely site manager needs to un-alarm/unlock school?)

I don't have keys, I call one of the site management team. they all live very local to the school so never been an issue.

[rrrrr]

The only options for this I can think of are;

1, Me and the technician alternate weeks on call and compensated for this.

2, A service contract with a 3rd party company for out of hours support. Don't know how this would work as they wont know our network.

3, Ignore things and let them fail. I would personally have trouble seeing this trough without stressing myself out.

 

any other suggestions? carrying on with just me being responsible 24/7 and then just paid the odd hours overtime when things do happen don't seem fair.

 

What do you guys think is a good way to move forward for a small technical team of 2? We may not have an issue for a few months but sometimes when doing major development these become a lot more common. Also, when one major issue happens it can have a knock on effect that is not always apparent. this can mean dialing in and out over a few days.

From your feedback, most of you either have;

1, a big enough team, or other branches to assist

2, Do as I am currently doing

3, wait till Monday

 

I don't think its fair or would be accepted in industry to have one guy available / on call 24/7 and only given the odd hours overtime when action is needed

thanks

Edited August 18, 2015 by rrrrr

[pcstru]

We luckily have a solid, well setup and up to date network with failovers in place but this does not stop things happening. Most of the issues I mentioned in original post cannot be easily prevented

Sure. I count myself as being in a quite fortunate position. We are well funded and able to buy enough decent kit to provision fail over and/or some hot spares. But things do still go wrong.

The only options for this I can think of are;

1, Me and the technician alternate weeks on call and compensated for this.

2, A service contract with a 3rd party company for out of hours support. Don't know how this would work as they wont know our network.

3, Ignore things and let them fail. I would personally have trouble seeing this trough without stressing myself out.

 

any other suggestions? carrying on with just me being responsible 24/7 and then just paid the odd hours overtime when things do happen don't seem fair.

I think that (you do it, someone else does it, it doesn't get done) pretty much covers it. I think the danger is you end up either stressed or resentful. My personal way of dealing with that is to make sure the issue is well understood at senior levels, i.e. that it is plain that they have made a conscious decision that a failure out of hours may not be dealt with. I don't sweat a little failure now and then because I know that actual instances of failure is helpful to those who make those decisions in assessing the value of provisioning out of hours cover. I don't think that attitude comes easily - it takes practice and needs to be cultivated. You need to be utterly confident that what you are doing is professional and entirely acceptable. I can only assure people here that it is.

[pcstru]

I don't think its fair or would be accepted in industry to have one guy available / on call 24/7 and only given the odd hours overtime when action is needed

thanks

What I suggest to management here is that we have a payment for people who are "on call". This is paid regardless of whether they are asked to respond or not. We then allow TOIL or overtime for any time they do have to spend. We have never got round to discussing what the "on call" rate should be (I think £100 a week would be a reasonable starting point) but at least it is clear that they have to decide to do something if they want a guarantee of cover.

[elsiegee40]

When I was out in the real world, we got a flat on call payment which was paid as part of our salary. This paid us to be available should the need arise.

 

If we actually got called in we we were paid, or got TOIL, for the time spent from the time of the call to getting home (if we had to go in) or call closed (if resolved from home) plus mileage if we had to drive into work. If we were working after midnight, a late start next day was also included.

 

Because I was part of a big team, first line on call was a rota of about 6 of us. It meant having a 'dry' week every 6 weeks to make sure we could drive in if necessary. However, everyone not on holiday or off sick was in reserve. It rarely happened that we ended up going in when not on first line call, but there were one or two sleepless weekends that I would love to be able erase from my memory.

[WhizInTraining]

I don't think its fair or would be accepted in industry to have one guy available / on call 24/7 and only given the odd hours overtime when action is needed

 

We work in what could be considered a shift. As I said, we're a three person "IT Support" team, but really only two of us deal with most of the issues.

 

What we have in place, is that one of us work from 8 til 4 Mon-Fri and the other person works 9-5 Mon-Fri, so that there is always someone here when our site is open.

 

Other than that, on weekends, we are very similar to you, rrrrr, we do as you currently do, but if one of us respond, then the other person doesn't need to respond.

 

Hopefully this helps.

[forkies]

I as the network manager am expected to respond to issues at a response level I feel necessary. Ie if I get a notification that all the servers are running hot, I would go into school to check as it could lead to dead servers. But if I get a notification that an access point is down its not priority and I will leave it until the morning.

 

I have keys and alarm codes to site, I also get phoned (number 3 on list behind site manager and site controller) if the burglar alarm or fire alarm is activated.

 

It is in my contract to respond to out of hours emergencies. And I get time off in lieu for dealing with these issues (which I rarely take, like my holiday). Overtime is no longer an option with less money coming in, as like most schools.

 

I am happy to do extra hours as I really like my job, my work place and my colleagues. I know some will say I'm lucky about that. I also live about 15 mins walking distance from work so not a big deal for me.

 

A example of a out of hours I had recently was a server down at 3am, I got the alert and went in to find one of our VMware servers which runs exchange had crashed due to not being updated (as we hadn't been allowed to take it down for maintenance) so while it was down I did the update as I felt it was the necessary action. I stayed in work until 4:30 in the end that day and didn't take the time in lieu but I did get a personal message and thank you from the business manager and head teacher which was really nice to be recognised for my work.

 

Obviously if I hadn't gone in then the server would of still be down the next day and I would be getting asked why? My answer would have been because it was lacking an update that prevented the error but I was unable to install it. I would probably have been quizzed why I didn't go in when it went down but would they have known it was down at 3am? Probably not.

 

I guess it is down to your contract, your commitment to work, and if they will reimburse you for your time. If like you say you have to ring the BM to get overtime approval, make that call once if there is no answer leave a message and don't go in! If they try to pin it on you, you have done what they asked. They will need to approve overtime at your discretion if their plan doesn't work.

 

Also the whole SLT and Site team (also my favourite staff) have my personal mobile for all hours emergencies.

 

Sorry it's so long but hope it helps.

[dapaulio]

All of the issues I mentioned could only be fixed by me, There is no one on SMT who would know where to start with these. I have raised this with my business manager before and the outcome was if an issue arises I should contact the business manager to get overtime cleared before I start.

 

I find myself just dealing with the issue irrelevant of overtime as;

I cannot always get hold of business manager (busy or unsociable hours)

It could be a 5min job to fix, don't know until I get into it.

If the problem is left/ignored it will probably escalate.

Just want to get issue resolved so I can enjoy the rest of my evening/weekend/holiday and not keep thinking about all hell breaking loose when I return to work.

 

I do find that I'm increasingly not being able to switch off from work, if I am off for long periods (e.g. xmas holiday) I find myself just checking all systems are functioning.

The reason for this is that I am the responsible for the network. If it goes wrong its me who feels the heat and me who has to fix it.

 

i feel your pain and i am sure many IT techs will agree, I too am responsible for a network which is accessed 24/7 and i have often thought whether i am the only one that constantly monitors the network whether that is through email/txt alerts, the occasional log on etc. i had to set up a social networking group to stop staff who i have friended on many occasions private message me on weekends, holidays etc.

 

i generally work to a rule where i assess the issue on and depending on the answers to my own questions would determine whether i fixed the issue straight away or wait till next day of work a) did the head log the issue? b) how bad is the problem c) how much heat would i get if i left it d) how much have i had to drink.

 

you learn to live with the heat, it comes as an occupational hazard working in IT. just think if you are not getting heat for not fixing an issue in your own time then you would be getting heat for something else when you return to work. its an endless cycle.