Power Surges, Cuts and server downtime... URGENT help please...
I am in desperte need of some ideas..
Over the past 2 weeks we have suffered from loads of problems, it started with a huge power surge, loads of buildings had power outages and when my network manger got into work on Sunday morning to see why our VPN was not responding, he discovered everything had faults. Servers (4 of them) were all down and when booted up 2 came up straight away, one took several attempts and one had RAID failures on 2 out of 4 and basically made it a write-off.
ALL of the network devices needed to be re-set, all had returned to factory defaults including WAPS, Modems and Routers etc.
We have a replacement server waiting HP to build it soon. Then we do the whole migration of datbases (SIMS Etc) as we had to install them all onto the domain server to get back up and running ASAP. We have also bought another UPS and daisy changed it, and tested it, to make sure we get over 15 minuntes for the main 2 servers.
Then over the last 2 weeks we have had several power outages and lightning strikes in the area. We have several occasions on evenings when there has been one server, but not another on the same UPS report a re-boot due to power failure on a morning after. Tonight we had another power surge of some sort and all servers shut down. This time they did not re-boot as the main power fuse tripped...
So, what to do? I have a principal who is livid and wants answers... well solutions really....
My initial thoughts are:
1) Bring in a specialist electrian/electrical company who can put in place some monitoring systems and help tell us if we are tripping systems internally at night time (we have issues during the day) due to flood lights and room lights being on OR if it is spikes/surges/cuts that are coming in from outside.
2) Put in some more surge protection. Currently everything is run off standard circuit breakers (trip switches / fuses). There is no specialised surge protection, perhaps there is something we need to invest in? We have individual surge protectors (5 socket extension leads with built in surge protection) on every PC and had none of these trip or cause problems. Is there something like this that we can use?
Please can anyone offer any suggestions? Thoughts?
Many thanks in advance,