Hardware Thread, Cause of two known good UPS to hard fault. in Technical; After some ideas here.
I have had (within the space of 48hrs) two known good APC UPS of different models ...
4th March 2010, 03:58 PM #1
Cause of two known good UPS to hard fault.
After some ideas here.
I have had (within the space of 48hrs) two known good APC UPS of different models experience a hard fault (all lights flashing).
In both cases, no attempt was made to notify the server of impending shutdown - power was just cut.
None of the attached servers was under heavy load at the times (08:20 and 13:56). Nagios, Cacti and windows event logs show no wierdness around that time. The UPS logs don't show anything strange.
Performing a hard reset of both UPS brings them back fine. The one that failed yesterday has been running a heavy test load for 24hrs without issue.
I'm leaning towards it being our rubbish power supply being more rubbish than normal, but is there anything else (other than pat testing the UPS) I should be looking at?
Last edited by pete; 4th March 2010 at 04:13 PM.
IDG Tech News
4th March 2010, 04:43 PM #2
Unless the batteries within the units are faulty then I would be looking towards the power supply that feeds them.
I would invest in good insured spike bars which should eradicate any bad supply having an effect on the UPSs.
4th March 2010, 04:50 PM #3
Er, don't we have UPS for this exact problem, I do!! I would be on the phone to APC finding out exactly what happened and to make sure it doesn't again!
4th March 2010, 05:03 PM #4
we've got about 8 UPS's mainly APC 1500VA's racked and one odd one (don't know the model)
There plugged into 2 rack mount power bars and then there plugged into 8 way power surge strips at which plug into the wall.
Better be safe than sorry in this event. As each UPS is nearly £350 and the Surge protectors are only £20-40 for half decent ones.
4th March 2010, 05:06 PM #5
It could well be your power feed within school. It's usually recommended to have the server room on a separate feed from the Computer Suite and any other classrooms. Any good electrician should be able to tell you how power is setup in your establishment.
4th March 2010, 05:39 PM #6
Oh I know how it's set up - it's crap, with random stuff tagged on any old how. We're pulling 250 amps at peak load (school-wide) on a 200 amp feed (work is under way to fix this) and I'm sure I've ranted on here about it.
Originally Posted by Michael
I'm just surprised that (if it were a spike) the other UPS didn't die - sensitivity is identical on all of them and they're on the same feed.
I did check with APC who said they'd RMA them if they didn't come back with a hard reset - they did and (testing them under load) I can't see anything wrong with the UPS itself.
Surge protectors to feed the UPS is a good idea.
4th March 2010, 05:44 PM #7
It definitely does appear to be a load issue, but from what you're describing the server room still won't have a dedicated feed despite on-going work? I would probably inform your line manager in writing because next time it may kill the server and that'll take a lot of responsibility off your shoulders.
5th March 2010, 12:49 AM #8
- Rep Power
Appreciate how a typical UPS works. Power is connected directly to the computer when not in battery backup mode. So dirty power hit the UPS and computer simultaneously.
Originally Posted by pete
What is a power supply's job? To convert even dirtiest power into stable and clean DC. Your computer's power supply makes 'dirty' power completely irrelevant. A UPS is often so cheap - have cheapest power supplies - that a replacement battery costs almost as much as the entire UPS.
You have assumed the UPS is doing things that even the manufacturer does not claim. It's only function is to provide temporary power during a blackout. Read only its numeric specifications. Claims in sales brochures will be misleading and intentionally incorrect - and that is acceptable.
Any facility that has 'mission critical' servers puts the UPS at the service entrance. Only that UPS provides more than temporary power.
What is the output of a typical UPS when in battery backup mode? This 120 volt UPS outputs two 200 volt square waves with a spike of up to 270 volts. And yes, that is perfectly ideal power to any computer or electronics. Same power can be harmful to small motors and power strip protectors. Perfectly ideal power because all computer power supplies are required to be so robust - to even make that 270 volt spike irrelevant. This electrical reality is not promoted in advertising; is not known by a majority only educated in advertising.
Incandescent bulbs dimmed to 50% - and that voltage is still perfectly acceptable to any computer. Computer power supplies are required to be that robust. The UPS is installed to maintain power when voltage drops even lower - to save unsaved data. Anything else you might want to cure is best performed elsewhere.
What do you want to protect from? Then provided can be solutions.
5th March 2010, 02:00 AM #9
I would suggest that is is a line voltage problem. We had a simmilar issue when the power feed was bursting to really high voltages. We had it jumping to 280V+ at some points because of the intermittant load on the supply.
westom (welcome to the forum by the way) is correct for the lower end non line interactive units but the more sophisticated units will try to drop overvoltage by bucking (HP UPS term) some of the voltage to lower it a bit. If it becomes to high for the UPS to buck safely it will imediatly shutdown as it is out of spec and can cause damage to the circutry of the UPS.
Most APC units 3500VA etc have an adjustable overvoltage cutout so you can adjust the upper margins to a certain extent. I would suggest that the voltage or even the line frequency could be drifting outside of spec due to the excessive load. I would get in contact with the power company or an electrition to get some quality monitering gear on the feed to see just how far outside spec it is drifting so you have extra amunition for getting the feed fixed.
5th March 2010, 10:10 AM #10
Use the Powerchute software and serial / usb cable and hook a PC /Server up to the UPS. Check the log files and look at the line voltages at the time the problem occured. This may give you more insight (i.e if the line voltage is very high).
We have the opposite problem here - our line voltage is often very low. Sometimes in the morning it goes down to 204/205v and the UPSs all kick in to battery mode.
Have an electrician coming in in Easter hols to balance the phases.
5th March 2010, 10:17 AM #11
I sympathise, our school's in a rural location and the electricity comes here with lumps in it - I've had belkin surgecubes and old ups's protecting the new UPS's for a few years now
Had a new feed from a different circuit put in last summer which seems to have tidied things up a bit, too - at least the server room gets 230v now rather than 180!
5th March 2010, 11:09 AM #12
We've got a monitoring thingy on the feed atm - it's in for a week, then there's some rebalancing work (and possibly a substation/beefier feed to the school) that needs doing.
5th March 2010, 11:15 AM #13
Can't you just run a few long power leads over to the big new Tesco's they're inevitably going to build just behind you? The brightness they burn their lights at they surely won't notice a hundred-odd amps leaking out during the work week...
Originally Posted by pete
24th March 2010, 02:56 PM #14
Heh. Can't comment on relationships with Tesco without causing a diplomatic incident.
Originally Posted by sonofsanta
But the monitoring thingy? Yeah, that was crap. Either it was duff or they didn't set it up correctly, so we got 2 days of info instead of a week. Apparently there was a power cut that affected the monitor despite it a) allegedly having a 6-hour backup battery and b) Cacti showing no power blips, nor a big chunk of missing data.
So I'm annoyed atm.
Last edited by pete; 24th March 2010 at 03:00 PM.
By Dos_Box in forum General Chat
Last Post: 28th January 2013, 08:25 PM
By Dos_Box in forum Other Stuff
Last Post: 22nd December 2009, 02:13 PM
By sparkeh in forum Windows Server 2000/2003
Last Post: 21st September 2009, 03:50 PM
By PrimaryTech in forum Hardware
Last Post: 21st October 2008, 02:26 PM
By TekkyMatt in forum MIS Systems
Last Post: 27th June 2007, 09:18 AM
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)