Business Continuity planning - power cuts

[psydii]

Apropos of nothing, have people considered how they might handle significant levels of powercuts?

 

I'm really aiming this question at people who run systems in places where power cuts are rare and usually short. What if they became frequent and long term?

 

I'm not sure we've fully powered down the server room since 2009.

[jmak]

If there's a power cut there won't be any pupils in school as you won't be able to run the heating and the kitchen will probably close (the fridges will go off, so it's not just hot meals that'd be off the menu). Probably best to look at cloud based stuff - you have to assume that major data centres have backup power.

 

A local school had a long term power cut due to a contractor cutting a major cable. They had a container sized diesel generator on site until the next holiday as they couldn't dig up the ground safely to repair the cable during term time.

 

I'm assuming you're referring to the possibility of generalized power cuts, not just your building. In which case you're unlikely to be able to get your hands on a generator.

[mavhc]

Offer everyone free EV charging at work, on condition that you can borrow the power back during a blackout.

[TechMonkey]

As jmak said, we've had a few of these. On was when the closest transformer on a pole popped, the other was when a contractor on a new estate managed to blow something up. No fix within 12 hours (I think) so a big arse generator was brought in. Just ensure you find out the plan to refuel it as one of the times they sent a deliver at something like 2am, thinking they would be able to just rock up, but our gates were locked. No one told us or left a message so the generator ran out of juice.

 

So have a communication plan in place. We have parental communication sorted through our cloud based MIS, with a laptop kept charged and mobile hotspot available. We have an old school phone handset (no power needed) to plug into an old fax line that main lines get diverted to. Mobiles are used to call out, with a few power blocks kept charged. Worst case admin staff can decamp to home and work. Next step is to cloudify our phone system so that we can use the app on mobiles to carry on as normal, or the people at home will carry on using the web version.

[Chris_Cook]

We had a 2 hour powercut the other week. It affected 6 schools locally, plus the ones in the MAT that we host servers for.

 

So far, what we have planned is moving to cloud-hosted mis, hosted voip phones etc and then mobiles/chromebooks/laptops with 4g dongles for key staff.

 

Obviously having UPSs with auto shutdown configured, but these are going to last 10-30 mins at most.

 

Other than that, people got on with it and used pen and paper. the power was back on for lunch, and the tills and sign in systems worked in offline mode until the servers were back up.

 

Hopefully when one of our schools gets rebuilt, then we will have more centralised network cabs that are easier to cover with UPSs to keep the infrastructure running a bit longer. with fewer servers on site, I'm hoping the UPS runtime will be longer due to less current draw.

 

Without a generator, there is only so long that a UPS will last, and owning a generator adds another maintenance task (replacing fuel, oil, testing it, testing the changeover switch etc).

 

Its also important to test, and de-brief after these incidents so that issues are resolved. We found some emergency lights not working, so that got fixed pretty quick. Our servers took an age to come back on because of windows updates. That still needs sorting.

 

If there is a plan, there is a lot that can be done to keep a school open without power.

[Andrew_C]

Bits of our site go off with sickening regularity. Often only one phase. Can't send the kids home, so boarding houses take in extras if necessary for feeding. Desk lamps (but no candles) get deployed in dark corners, books get read, etc

 

We muddle through.

[jmak]

Has anyone looked at getting something along the lines of a Tesla Powerwall instead of a UPS? Given the limited predicted lifetime (APC quote 2 years for the battery) and the price per kWh of capacity, they look pretty cheap and would give much longer run times. Might even be worth looking at for school kitchens.

[psydii]

The Fridges, the freezers, the servers and core switches, the alarm system, the boilders/pumps, the wifi.

 

It depends on the scenario. short power cuts - keep the servers, wifi and phones up. Longer powercuts the fridges and freezers become more important along with the alarm system. Multi-day powercuts - if you can also power the boilers/pumps to keep the heating on then potentially you can keep the site open. If you have power for that then you probably also have power to bring the servers up. You also probably need a generator for that.

 

As @jmak alluded to, I was however thinking more about the potential around regular extended regional power outages and IT. How do you handle the shutdown and restart of services? Last time we rebooted the servers it took the best part of a day to shake out all the long-tail problems. Basically, if endpoints boot before the core infrastructure is ready, there are problems. Given the san, the hosts and the vms take a sweet while to go from cold to running, phones, desktops etc are left in a sub-optimal state. Also switches boot at different speeds so bits of the wireless come up and assume one state, while those that power on later assume another, and they all need another reboot to bring them into line.

[mavhc]

Just fix those bugs one by one, why does the wifi do weird things? Make the desktops wait for the servers, or make them fix their problems with regular gpo refreshes, etc.

 

Maybe set the switches to only turn on PoE after a signal is sent

[dhicks]

Has anyone looked at getting something along the lines of a Tesla Powerwall instead of a UPS?

 

It's an idea - we currently have a large UPS in the basement, with (separate from the standard mains) internal cabling to power switches and a couple of workstations. The UPS needs servicing yearly, and we've replaced the batteries at least once already. Switching that for a Powerwall is something we could maybe ask our parents comittee about, it's something they might be interested in helping us with - especially if we could maybe add some solar power collection.

[psydii]

Wifi goes split-brain if parts power up while spanning tree hasn't converged. Its a feature, which is only a problem if there is no layer2/3 connectivity to the core - a very rare occurrence, but guaranteed if we have a site wide power failure of more than a couple of hours.

 

Workstations power on in seconds. The Servers take many many minutes starting with the SAN, Core Switch, ESXI Hosts, VMS (DCs first, and only after they are up start the rest of the servers)....

 

The trouble is that people don't wait - if they see a logon screen, they logon. a full cold restart takes a long time, and people will just start using things as soon as they look ready.

 

I suppose we need "once the core servers and switches are all up, disable PoE across the network" and "Once PoE is disabled on all ports across the network, re-enable PoE across the network" as two steps in the playbook for this scenario.

[Chris_Cook]

Fridges and freezers are departments/canteens problem. They should stay cold if kept closed for a good few hours. Department procedures should be in place to monitor anything critical in them.

 

Alarms should be battery backed and should be checked by the support company. These should last hours. Doors are the same. These can be tested by disconnecting the power and timing how long they last (Net2 logs should help, and email monitoring can be set up).

 

We found Dell servers started up much quicker than our old HP ones. and we moved away from a san as it often caused more problems than it ever solved. Especially once you have 10G networking between hosts + SSDs.

 

Our VMs still took most of the day to do the windows updates.

 

Once things are back up and running, we do an all staff email to say reboot your PC and it will work fully.

 

It does help having most staff + students using google docs etc as it minimizes what they need from the onsite servers - pretty much just DHCP, DNS, routing.

[mavhc]

It's an idea - we currently have a large UPS in the basement, with (separate from the standard mains) internal cabling to power switches and a couple of workstations. The UPS needs servicing yearly, and we've replaced the batteries at least once already. Switching that for a Powerwall is something we could maybe ask our parents committee about, it's something they might be interested in helping us with - especially if we could maybe add some solar power collection.

 

Lead acid vs Lithium. Lithium batteries prices have come way way down.

 

Can you fit lithium batteries into your existing ups?

[dhicks]

Can you fit lithium batteries into your existing ups?

 

Good point - quite possibly, it'd be worth checking. I think the batteries were changed a couple of years ago (or maybe a year - all my time-estimates are a year out at the moment...), they should have a few more years in them as yet.

[Andrew_C]

Can you fit lithium batteries into your existing ups?

You'd probably need to change all of the electronics too.

[mavhc]

https://download.schneider-electric.com/files?p_Doc_Ref=SPD_VAVR-A5VTU5_EN

 

Seems so, annoying