Connecting up your UPS(s)

[timbo343]

I'm wondering what the best practice is for connecting up UPS(s) as we have had a failure on our APC 3000 (it went pop when we ran a self test).

 

In our server cabinet we have a single APC2200, a Dell 2700W with additional battery pack and had an APC 3000 with additional battery pack before it went pop.

 

Physical servers: 7

SANS: 2

 

Physical Server 1:

2x 480w PSU

 

Physical Server 2:

2x 750w PSU

 

Physical Server 3:

2x 480w PSU

 

Physical Server 4:

2x 500w PSU

 

Physical Server 5:

2x 500w PSU

 

SAN 1:

2x 580w PSU

 

SAN 2:

2x 573w PSU

 

Physical Server 6:

2x 460w PSU

 

Physical Server 7:

2x 460w PSU

 

PDU with a HP1800-24G (27watts) and 2x Cisco SG100 switches.

 

Should 1 of the PSUs on each device be plugged into an UPS and the other PSU be plugged into a 13 amp socket or should 1 PSUs in the server be plugged into 1 UPS and the other PSU be plugged into the other UPS?

[Mako]

I have it so 1 PSU goes into 1 UPS and the other PSU goes into another UPS.

 

Plugging directly into mains can introduce power spikes directly to the PSU if there's an issue with the mains, which could cause a PSU failure, whereas your UPSes should protect against that.

[Sephiroth]

The question is; do you have any active control from your UPSs? IE, do they shut the servers down when they get to a certain battery level or runtime? If that's the case, it'll depend on how you want shutdowns to run and how well the UPSs talk to the connected systems.

 

This is something that we're struggling with at the moment. We have 2 Eaton 5px units that we had connected PSU1 to UPS1, PSU2 to UPS2 where possible, but we've had to rethink that with the above in mind. We have ended up going with servers that have shutdown commands on them with both PSUs on one UPS.

[timbo343]

I have it so 1 PSU goes into 1 UPS and the other PSU goes into another UPS.

 

Plugging directly into mains can introduce power spikes directly to the PSU if there's an issue with the mains, which could cause a PSU failure, whereas your UPSes should protect against that.

 

Ha, you'd hope so wouldn't you but we had a PSU fail when we had a power spike when they were connected to the UPS.

 

Thought that would be the case for how it was set up.

 

I've trying to spec a new UPS and if i have all 1 of each PSU plugged in to an UPS, i'm going to need near on 6,000Watts! Surely that can't be right?

 

I mean ive got 1x 750w, 2x 460w and 2x 500w plugged in to the APC2200 and the APC is only showing around 400w / 450VA / 2A being used. Obviously the servers aren't being pushed to limit but given that, do i really need a massive UPS?

[timbo343]

The question is; do you have any active control from your UPSs? IE, do they shut the servers down when they get to a certain battery level or runtime? If that's the case, it'll depend on how you want shutdowns to run and how well the UPSs talk to the connected systems.

 

This is something that we're struggling with at the moment. We have 2 Eaton 5px units that we had connected PSU1 to UPS1, PSU2 to UPS2 where possible, but we've had to rethink that with the above in mind. We have ended up going with servers that have shutdown commands on them with both PSUs on one UPS.

 

We used to have the APC3000 control the PowerChute shutdown and we were finding that we were getting spikes so kept coming in to work and finding the entire rack was off because of these spikes.

 

The thing is, this place is completely wired up wrong anyway - there is a main circuit breaker in a switchgear room which is overloaded anyway so if we have a power failure even for just 3 mins, if the load on the breaker is too much, it breaks and there is a set routine that needs to happen to reset the breaker. Basically our Server Room and half the DT rooms and some ACs are fed off this main breaker.

[PotNoodleTech]

If you have 2 x PSU inlets on each server/switch then the best way to do it is buy 2 x UPS and on each server, put one PSU into UPS1 and the second PSU into UPS2.

 

That way - your always surge protected (unlike when you plug one PSU into the wall), and if one UPS dies catastrophically the second one should keep things powered on.

 

Also - I have each UPS on a separate actual phase on the 3 phase so that if one phase goes down for some weird reason the server room stays up. It was worth the extra wiring from the electrician.

 

Just make sure the UPS's are big enough to do the job on their own if one fails - I use 2 x 5000 Eaton for less servers than you - and I would suggest something similar sized at least for your system. Mind you the Eaton kit has been running for 6+ years solid as a rock unlike the APCs I used to run. Haven't even changed the batteries and still getting 1 hour runtime and have never had to fail over to another UPS.

 

The only things that have actually made me shut the servers down over the last 10 years are serious electrical work being done (all phases being taken down, new substation, etc).

Edited November 6, 2023 by PotNoodleTech

[timbo343]

If you have 2 x PSU inlets on each server/switch then the best way to do it is buy 2 x UPS and on each server, put one PSU into UPS1 and the second PSU into UPS2.

 

That way - your always surge protected (unlike when you plug one PSU into the wall), and if one UPS dies catastrophically the second one should keep things powered on.

 

Also - I have each UPS on a separate actual phase on the 3 phase so that if one phase goes down for some weird reason the server room stays up. It was worth the extra wiring from the electrician.

 

Just make sure the UPS's are big enough to do the job on their own if one fails - I use 2 x 5000 Eaton for less servers than you - and I would suggest something similar sized at least for your system. Mind you the Eaton kit has been running for 6+ years solid as a rock unlike the APCs I used to run. Haven't even changed the batteries and still getting 1 hour runtime and have never had to fail over to another UPS.

 

The only things that have actually made me shut the servers down over the last 10 years are serious electrical work being done (all phases being taken down, new substation, etc).

 

We are looking to get another UPS, either a 3000 or a 5000. We have the Dell 2700W and only just found out how to add the additional battery pack after all these years.. RTFM comes to mind as the setting is not in the web UI. By activating the additional battery pack we have a run time of around 3hr 20min at 21% load.

[gaz350b]

Just gotta watch the heat output. When we moved into our building the air on guys configured all the units to turn off at 5. By 9pm we had multiple dead servers.

 

The room is massive can easily fit 15 racks (it has 5) and 5m high ceiling. Only 1/2 rack of servers and a couple of core switches. Got hot pretty quick.

[timbo343]

If you have 2 x PSU inlets on each server/switch then the best way to do it is buy 2 x UPS and on each server, put one PSU into UPS1 and the second PSU into UPS2.

 

That way - your always surge protected (unlike when you plug one PSU into the wall), and if one UPS dies catastrophically the second one should keep things powered on.

 

Also - I have each UPS on a separate actual phase on the 3 phase so that if one phase goes down for some weird reason the server room stays up. It was worth the extra wiring from the electrician.

 

Just make sure the UPS's are big enough to do the job on their own if one fails - I use 2 x 5000 Eaton for less servers than you - and I would suggest something similar sized at least for your system. Mind you the Eaton kit has been running for 6+ years solid as a rock unlike the APCs I used to run. Haven't even changed the batteries and still getting 1 hour runtime and have never had to fail over to another UPS.

 

The only things that have actually made me shut the servers down over the last 10 years are serious electrical work being done (all phases being taken down, new substation, etc).

I was looking st the APC 5000 but found it has a Hardwired in lead meaning putting a 13a 3pin plug on it is a bit risky so will have to stick to the APC 3000 with additional battery packs for now.

[timbo343]

Just gotta watch the heat output. When we moved into our building the air on guys configured all the units to turn off at 5. By 9pm we had multiple dead servers.

 

The room is massive can easily fit 15 racks (it has 5) and 5m high ceiling. Only 1/2 rack of servers and a couple of core switches. Got hot pretty quick.

Our server room is tiny, 2m (at most) by about 10m.

 

The AC has failed a number of times in there (and leaked too) so i wouldn't be surprised if the batteries have taken a hit from this.

[PotNoodleTech]

I was looking st the APC 5000 but found it has a Hardwired in lead meaning putting a 13a 3pin plug on it is a bit risky so will have to stick to the APC 3000 with additional battery packs for now.

 

Yeh mine are hardwired the cost of an electrician to put 20 meters of cable in to get 2 x 32 amp outputs in the server room was negligible. That was 15 years ago mind so it'll probably cost 10 grand nowadays :/