So I learnt something about APC UPSes today!

[chazzy2501]

OK after 2 hours I've figured out what happened to my system on Saturday morning! It would seem that 1 of my 2 hosts power supplies failed. Being helpful it sent a spike back through the APC UPS and tripped the whole cabinet out.

 

of course the VMs on that host didn't shutdown nicely Unfortunately it seems that the UPS that powers the switches failed as well (duff battery) that was just bad luck. (it didn't tell me prior to this event), killing any chance of the first UPS telling the other servers that they were all running on battery! (So to be clear an unrelated UPS also failed when it got the spike!)

 

This was a HP G6 server PSU how on earth did it fail so badly? Why didn't the first UPS (APC 2200) stop the spike going back to the rest of the cabinet?

[DMcCoy]

I assume most of the protection is on the *input* not on the output side of the power. If you overload most UPS for example, they will immediately shutdown. With the switch one, it's worth hooking it up to a computer to configure the self testing. If you don't have a regular self test, you won't know when the battery is dead until you try to switch to it.

[chazzy2501]

The switch UPS is an APC 1000 it does a self test once a week! (configured through USB) Just rotten luck that it's battery must of crudded out between tests and this failure! One of it's batteries has 0AMPs capacity so has totally failed....batteries are on the 3rd year.

 

All UPSs are APC and are the on-line type...

Edited September 23, 2013 by chazzy2501

[glennda]

My experience of APC ups's are that they are awful.

[mdench]

Get a Borri B500 https://www.borri.co.uk/uninterruptible-power-supplies/b500-series#front

[glennda]

Get a Borri B500 https://www.borri.co.uk/uninterruptible-power-supplies/b500-series#front

 

Is that one still going strong Max?

 

I'm hopefully getting a B400r in the coming weeks (although they arn't released yet )

[mdench]

oh yeah still going strong only slight dodge moment with it was when it went into bypass mode after a powercut and having to stand in the server room holding down the power button for 5 seconds with the whole server room running which is exactly what you have to do to turn it off.... that was fun haha!

[PotNoodleTech]

My advice to stop this happening again - if your servers have dual PSUs, get 2 x smaller UPS's rather than 1 big UPS and plug one PSU from each server into each UPS so that if one UPS get's totally taken out (bad battery, supply fault, trip switch, etc), the other UPS will keep going and keep supplying power to one PSU in each server. If possible have each UPS powered from a seperate trip / phase for maximum redundancy.

 

I've done this here and have had 365+ days of uptime on the power for a very low cost (2 x APC PDU, 2 x APC UPS).