UPS Replacement & Dual PSU Query

[CHiLL]

We've had our UPS fail a couple of times in as many months recently, so we are looking at replacing it. While the battery is well past it's EOL, the UPS put itself in battery mode, even when the power was still on. Eventually the batteries drained it everything went off. To me, this sounds more like an issue with the UPS than the battery - as it did operate (incorrectly) via the battery until it ran out.

 

We've already got quotes in and we have secured the funding, I just have some queries before I make any commitments.

 

Dual power sources

We currently only have one power source into the server room, with all devices (incl secondary PSUs) plugged into it. We will be getting a secondary power source fitted into the room, from another distribution board. My query is regarding physical connections:

A) Should we have the power terminate in DSSO sockets and have the secondary PSUs plug into those?

B) Since we are purchasing another UPS, should we utilise our old UPS with a new battery and have the power terminate in a commando socket, which is then attached to the secondary UPS?

 

I think option A) is the simplest and most common solution with B) being overkill. I was just after some thoughts on it.

 

PSUs via two inputs

Since most critical devices have dual PSUs - how does the device know which PSU to draw power from? If I choose option A) above, ideally I'd like the UPS as the primary power deliverer for the device, protecting against brown outs/spikes, etc. I'm guessing the devices aren't smart enough to choose inputs - as they won't know where it's come from, it's just power. It just got me wondering.

[3s-gtech]

With dual PSUs, the server manages the inputs with both being online and ‘hot’. Changeover is seamless for whichever fails.

 

Having a UPS on both PSUs is a good plan. Failing that, one of each should be on an independent supply (from another ring/board/phase even). You should never have both in one UPS.

[mavhc]

I plug one PSU into the mains, one into the battery

[Chris_Cook]

It depends a lot on what you are running on your servers, and how much your employer values it being on.

 

Assuming you servers have dual PSUs.

 

You can have one PSU on a ups and another on the mains. This gives you protection against a single fault - loss of power, ups failure, disconnecting ups or psu for maintenance etc. You wouldn't be able to do two of those things.

 

The ideal would be 2 UPS's plugged into separate mains supplies, and 1 PSU on each server on each ups. You could get an auto change over switch for anything with single psus like switches etc.

 

If you bring in a separate feed from another dist board, then you need to be very careful that the earths are bonded correctly. It would be worth consulting an electrician who has done this before. You don't want you or your equipment becoming the link between the 2 earths.

 

It would be worth investigating why the ups went onto battery. some newer UPS's with LCD screens can display this, or if you have the NMC it will be in the log. I'd recommend getting the NMC so you can do the safe shutdown and email someone when the power goes out. It might be you can adjust the sensitivity of it, if the voltage is within spec, but not what the UPS is expecting.

 

I think most servers traditionally just ran both PSUs in parallel and probably used diodes to prevent one back-feeding into the other. On our last lot of Dell servers there was a bios option to set a primary PSU and switch over to the backup when needed. How much power that saves I'm not sure. There are more expensive solutions that can manage all of this, but are probably out of most schools budgets. One benefit of this might be to bind a server to each ups in a 2ups/2 server setup and increase your runtime a bit.

 

If you are getting electrical work done, it might be worth seeing if the electricians can fit an SPD (Surge protection) in the board at the same time.

[CHiLL]

Thanks for the replies. The logs for my UPS (APC Smart-UPS RT 6000 RM XL) don't yield much info and the logs only go back to 21/07/2022 - which is coincidently the first day it failed. The logs it does have show some things of note:

 

05.08.2022	22:43:11	Device	UPS: The battery is now installed.	0x0130
05.08.2022	22:43:09	Device	UPS: The battery is not installed properly.	0x012F

Research shows that when it reports not installed properly, it's indicitive of a fault or complete failure of the battery.

 

The log also includes a bunch of SMTP errors (due to incorrect configuration) The times are all over the place, because desipte the NTP settings being correct and pulling from our DC, which has the correct time, it currently thinks it's 10 minutes behind (timezone is set to London and DST is disabled).

 

However, on the days that we had outages, there is nothing significant. We had our last outage on 29/07/2022 and these are the only three logs for that day and each day either side:

30.07.2022	19:22:36	System	NTP update successful.	0x004A
30.07.2022	19:22:36	System	Configuration change. 	0x0033
30.07.2022	06:11:09	System	NTP update successful.	0x004A
30.07.2022	06:11:09	System	Configuration change. 	0x0033
29.07.2022	17:02:33	System	NTP update successful.	0x004A
29.07.2022	03:55:06	System	NTP update successful.	0x004A
29.07.2022	03:55:06	System	Configuration change. 	0x0033
28.07.2022	14:42:37	System	NTP update successful.	0x004A
28.07.2022	14:42:37	System	Configuration change. 	0x0033
28.07.2022	02:18:09	System	Web user 'apc' logged out from 10.22.100.70.	0x001F
28.07.2022	02:17:19	apc	Web user 'apc' logged in from 10.22.100.70.	0x0015
28.07.2022	02:16:15	System	Web user 'apc' logged out from 10.22.100.70.	0x001F
28.07.2022	02:15:26	apc	Web user 'apc' logged in from 10.22.100.70.	0x0015
28.07.2022	01:27:04	System	NTP update successful.	0x004A

 

We are going to replace the battery as a first port of call and also get secondary power sockets installed from another ring (will also ask for surge protection to be included, thanks @Chris_Cook. It's the cheapest option at present and also the quickest, with no lead time on batteries and a multi-week lead on a replacement UPS.

 

While a dual UPS solution with two power sources is my ideal solution, it's very expensive. If the replacement batteries don't resolve the issue, then we'll replace the UPS and also have the power split between UPS and wall sockets. Maybe down the road, we can buy a secondary UPS and replace the wall sockets with a commando socket. At least the power won't be as expensive then, as it's already been run to the room, it just needs sockets converting.

 

While our UPS is 7 years old, I'm hoping that the UPS itself is designed to work for much longer, assuming battery replacements are made and not accounting for actual hardware failure.

[3s-gtech]

One of mine is 13 and still working - though it has thrown some warning signs.

 

I had another fail at 9, which is about the usual lifespan.

[PotNoodleTech]

I use 2 x UPS's powered by two different electrical phases. Each UPS powers one of the PSU's on each server. The second UPS powers the second PSU on each server.

 

That way:

If one phase goes down in the school, the system stays powered.

If one UPS goes down, the system stays powered

If the whole school power goes, both UPS share the load, doubling my on battery time (1 hour 10 mins currently).

 

I can categorically state that no matter what you read about dual server power supplies, they always balance the exact load of the server 50/50 with both PSU's being active at all times. If the server is using 200w total then each PSU draws 100w from the UPS.

 

If your UPS is already failing and the battery already done - just buy a new UPS don't try and reuse it. A failing UPS can cause catastrophic damage to the server and also cause a fire. Not worth the risk of burning down your school to save a few hundred bucks.

[Aprice]

Can't agree more with PotNoodleTech, I have had a couple of UPSs go bang on me (and I mean bang, Smoke burning smell etc) both Dell rebadged APC units probabbly 10 years old at time of destruction.

 

Older units left unplugged for a while also pose a fair risk of going pop when plugged in again for the first time.

 

I try to get our customers to replace every 5 to 6 years if possible, and those with two UPSs to replace a year or so apart so they aren't exactly the same age. Probably doesn't make much difference but might help avoid a bad batch.

[CHiLL]

We decided to replace the batteries and go from there. It's been a week now and not one error or warning has been logged by the unit, compared to the many-a-day entries we were getting for battery disconnected/connected messages.

 

We have also decided to go down the route of physical plugs on the wall from another ring as the secondary power source. We've had it quoted, but no timescale on works yet. We went for this because we can't fit another UPS in the rack without some serious jiggery-pokery of servers and storage units, involving taking them out, moving rails up, re-fitting, etc. This also gives us the option of converting the power source for the sockets into a commando socket later down the line, should we decide to fit a second UPS (if space is freed up).