What's your process when someones asks this...

[L3OUGH]

Day 1 of the Easter hols and just had someone pop in the office with the age old question:Can you just shut down the servers real quick for 40 minutes later today?Which got me thinking, when this happens to everyone else, do you have a shutdown process? Order of which to shut your servers down? Least important first to most important last is a given of course.People just seem to assume shutting everything down is a 2 second process and IT servers are a last second thought. 25+ servers with varying services, but yeah we aren't THAT important.

[L3OUGH]

Not sure what happened to the formatting on the post. I didn't expect it to come out as one paragraph!

[DrBeaker]

Who's asking? Electricians, PAT-testers, Head, Business Manager etc?

 

What sort of servers? Physical, Hosts with VMs?

 

Which ones - non-critical, critical etc?

 

It can take anything from 10-mins to 2x hours to do a proper shutdown of servers and their services depending on what they're running.

 

IMHO needs authorisation from IT/Head depending who's in charge of IT.

[maark]

At least they asked you - have had people pull the plug in the past.

Best to have a shutdown sequence in a document somewhere in case you are not available and someone else has to do it.

Also is there anything going on that might be impacted - easter schools, staff in, remote working?

[FishCustard]

Here it'd be me authorising that, assuming that it's during the holidays and had been planned for (just in case people are expecting to work from home).

 

A sudden request on the day would almost always get a "nope" unless it was an emergency (e.g. server room flood, fire, etc.) which would obviously trump anything else.

 

If (for example) the HT told me to just shut them down, I would, but not without asking why and having the request in writing.

 

That said, I know our UPSes can keep things running for about an hour, so if it was a short outage I might risk it and call it an unscheduled test...

[bald_pig]

At my last place we had a full step by step procedure taped to the server rack.

 

Here it's just 2 servers, shut them down.

[DJ-1701]

No official procedure, but would either send an e-mail out or a message with something like NetSupport Notify, Impero, etc about emergency maintenance. Unplug the phone. Then shut down the devices as needs be.

[L3OUGH]

It's just annoying. People assume it's a 30 second no worries task. Having some aircon units installed, the electricians will need to route the mains through a mains switchboard (switch also supplies the server room). Everything will have to go off, the VM Hosts, the physicals I've got, core switch, SW appliance etc. It's all in there. No heads up either which is the worst.

[DrBeaker]

It's just annoying. People assume it's a 30 second no worries task. Having some aircon units installed, the electricians will need to route the mains through a mains switchboard (switch also supplies the server room). Everything will have to go off, the VM Hosts, the physicals I've got, core switch, SW appliance etc. It's all in there. No heads up either which is the worst.

 

Which will also knock out your Internet connection.

[synaesthesia]

In this instance, I'd just suck it up and do it. I would then make a note to the site team/SLT that the servers & core infrastructure should be on a different circuit to the rest so they'd be unaffected should further issues occur or other circuits are changed in such a way. It's not your fault.

[Fazza]

After we've stopped laughing and realise they are being serious, we would ask why they need to turn the power off and then make a decision after that. As it can take anywhere from 45 minutes to 2hrs (bloody pending Windows Updates) we'd email out and let people know that 2hrs before the power is going to be cut we will be gracefully shutting down the servers and then get on with it at the agreed time.

[Achandler]

40 minutes.. We'd let the UPS handle it.

 

Takes about 2 hours to take everything on and off again here currently.

[3s-gtech]

Cycling through each Hyper-V host and the few physical hosts we have, I can shut everything down in about 10 minutes - no order other than VM then host (obviously!) Not too bad.

[forkies]

I go through and turn off non essentials, then essentials then dcs. But for 40 mins I would leave it running on UPS. That would take the batteries down to just over half.

A full shutdown takes me about 15 mins max if I had to.

We have a dedicated DB for the server room so this we don’t really ever have to do this.

[jmak]

40 minutes.. We'd let the UPS handle it.

 

Takes about 2 hours to take everything on and off again here currently.

I reckon this. Probably not best practice, but the UPS software would gracefully shut the servers down, so risk of long term harm is minimal. On the basis it's quite a lot of hassle for me and the chances are it would be wasted effort and I'd had no warning, I'd let them take the consequences - and probably take it as an excuse for a quiet morning. I'd probably let the software take care of the re-booting too.

 

Views might be influenced by the fact that I'm on holiday [emoji41]

[mavhc]

Apart from informing users, what else is there to do? If your servers don't shut down and start up without manual help then fix that bug, if software relies on other servers starting first then make sure it has checks and waits for those services to start instead of just failing

[Ditto]

At my current school, I've no idea but I will ask the NM when I'm back to work after the holidays. By chance I checked my emails and see they've had power cuts that have impacted IT systems. The issue was sufficiently serious that the SBM notified staff expect a solution for at least 24 hours and if they were coming in to the school, to make sure their work didn't require IT access. I know the school has UPS in place, but it appears not to have handled this outage. At a previous school, which was 99% GSuite and cloud based MIS, I don't think a power outage onsite would have any impact at all - except maybe if a member of staff came onsite as they'd be no WiFi or internet, but they would have access at a second site or at home. I guess it's an example of a cloud based positive that becomes apparent only in these sort of situations.

[mavhc]

Unless the power goes out in any of the 100 places between you and the servers you're using on the internet, or a cable breaks. At least internally you can fix it.

 

I have zabbix reporting when things go wrong, really need to have an external monitor to email me when zabbix is offline

[forkies]

Unless the power goes out in any of the 100 places between you and the servers you're using on the internet, or a cable breaks. At least internally you can fix it.

 

I have zabbix reporting when things go wrong, really need to have an external monitor to email me when zabbix is offline

 

We use Pingdom to alert is when our monitoring system goes down. Plus monitoring generally of external services.