What power provisions do you have for your server racks?

[talksr]

Hi there,

 

The school I have worked at for a long while now, has gradually moved from wall mounted cabs, to proper racks.

We have two large floor standing racks and two full 12 plug PDUs inside. So that is 24 plugs in use.

 

Unfortunately, the electrical wiring in the room has never been upgraded and all of this is split across two conventional plug sockets.

I personally feel this is quite dangerous, and really, the racks should have their own fused and independent power source.

 

To give you some examples,

I have had several outages whereby someone in a classroom trips the power for whatever reason and it shuts off everything in the rack. By the time the site manager has been found and he has tracked down the tripped fuse, all batteries in the rack have ran dry and everything is off.

 

Occasionally, the plug points the racks are connected to, will trip as they are clearly over loaded.

 

What power provisions do you guys have? I would really appreciate any pictures as I can show these to the electrician who comes in.

I am thinking of an independent and fused direct cable which terminates with one or two of these IEC309 plugs:

Along with a red power shut off button so both racks can be quickly and easily shut off in the case of a fire or smoking from within the racks.

The IEC309 leads could then connect to two new internal PDUs which will have the 12 x 2 standard 3 pin power sockets.

[Chris_Cook]

Sounds about right.

 

Whats in the racks? Is it just normal switches and ups?

 

On the back of each device, there should be something that tells you the power consumption in amps or watts. If you convert all this to the same type, add it up, then make sure for each wall socket you're not over 13 amps (or about 3000 watts if i remember rightly).

 

You probably also want to do that for the whole ring, which will probably be limited to 32 amps. Also, if there are lots of PCs or similar, earth leakage may cause it to trip.

 

If its tripping the breaker, then I would guess you are over this.

 

If its causing a problem, raise it with SLT and get it on its own ring as you said.

 

Its one of those things where if its not overloaded, its really a business decision. So you may need to sell the benefits of it to SLT - ie less downtime.

Edited November 19, 2018 by Chris_Cook

[talksr]

Sounds about right.

 

Whats in the racks? Is it just normal switches and ups?

 

On the back of each device, there should be something that tells you the power consumption in amps or watts. If you convert all this to the same type, add it up, then make sure for each wall socket you're not over 13 amps (or about 3000 watts if i remember rightly).

 

You probably also want to do that for the whole ring, which will probably be limited to 32 amps. Also, if there are lots of PCs or similar, earth leakage may cause it to trip.

 

If its tripping the breaker, then I would guess you are over this.

 

If its causing a problem, raise it with SLT and get it on its own ring as you said.

 

Its one of those things where if its not overloaded, its really a business decision. So you may need to sell the benefits of it to SLT - ie less downtime.

 

Thanks for your quick reply.

 

Getting a decision is not an issue. Thankfully, I am (in a way) part of the SLT. If it can be afforded, I make the decision on if it is or isn't done.

I will see if I can find the time in-between everything else to try and add up the wattage and see what we are looking at.

 

There are four servers and respective UPSs, CCTV DVRs, Router and firewall and then around 12 switches, 2 wireless controllers.

The room was originally a small library room, so it was never kitted out with appropriate power provisions.

We have AC, loads of data points, cabling ducts for the racks, fibre termination point, everything other than proper power!

[dhicks]

We have two large floor standing racks and two full 12 plug PDUs inside. So that is 24 plugs in use.

 

We have one rack, with 2 of the blue "commando socket" connectors providing the power for the rack. Most of our servers have dual power supplies, so we make sure we have each power supply supplied by a different circuit. One of the circuits is connected just to a UPS, which then supplies a PDU in the rack, the other connects straight to a PDU. The circuits are dedicated to the server rack, there's no other sockets connected to them - we have the electrical consumer unit in the server room so we have access to the isolator switches if we need to turn things off in a hurry. If you have cooling in your area, that should probably be on a separate circuit, too, if it isn't already.

[TechMonkey]

To give you some examples,

I have had several outages whereby someone in a classroom trips the power for whatever reason and it shuts off everything in the rack. By the time the site manager has been found and he has tracked down the tripped fuse, all batteries in the rack have ran dry and everything is off.

 

Occasionally, the plug points the racks are connected to, will trip as they are clearly over loaded.

 

Surely these are both examples of bad power routing/linking? The type of plug or how you plug the racks in won't change what trips if it is on a shared ring or if the ring is overloaded.

 

So your first example, you need to get your server room on it's own fuse, possibly even own fuse box. If that was in the server room then if it trips you will be able to reset it, if they were able to put half on one fuse and the other half on a different fuse, you would be able to spread and share the load and therefore the chance of tripping.

 

The second example would be solved by the above changes and, as you say, are a symptom of being overloaded.

[Chris_Cook]

I'd say that's overloaded, and if not, the earth leakage on that lot is going to be near the limit of one RCD.

 

I'd roughly guess an amp for each of the switches, controllers etc, and at least a couple of amps for each server PSU and each UPS. Possibly more for them.

 

Whats in the other classroom? is it standard, or an ict suite?

 

If your UPS (S?) have IEC C19/C20 sockets on them, I'd get a 16amp ceeform for each, and a separate breaker for each if possible. Then probably get each room on its own ring for all the other mains sockets. The upstream distribution board needs looking at as well to make sure it has the capacity for all this. If it goes again, find out which breaker is tripping - that will help any electrician know whats overloaded.

 

At this point, your probably best getting an electrician in to quote and recommend for this. I'd try and make sure they have some experience with providing power for ICT systems, as they will want to take into account earth leakage and have some understanding of why you want things isolated.

[Katy]

We have 3 racks, one rack has the UPS and from here C19-C20 leads feed out to the PDUs in the same and neighbouring cabinets. There's a 50A isolator on the top of the rack (as the UPS is hard wired), fed from the ceiling, so no trailing cables. UPS says we're pulling 11A total so could have done it all on a single plug top really but safer this way.

 

Before I set that up, we had trailing cables into each of the 3 UPSes from switched fused spurs on the wall - each on their own 16A breaker.

[Areku]

All floor standing cabs with 16a dedicated commando lines, hard wired into UPS's in the bottom of the cab, (the commando lines are fed into the top of the cab, to the ceiling tray, then to the high socket in each room)

our 3 cab server room also has additional commando sockets (6 in all, 3 32a and 3 16a) not that we have ever used all of them.

(all of our cabs, have their own air conditioned rooms across the site, which is good during the summer... oh no we need to repatch this cabinet ...nice and cool)

[PotNoodleTech]

Absolutely your racks should not be on the same spur as any other stuff (it suites, plugs in corridors used for vaccum cleaners, etc). They should be dedicated to your server room and it's really worth having an electrician wire it in all nice and new and tested.

 

I've got a pair of dedicated 32 amp commando sockets wired to the wall near the server rack, each socket is fed from a different distribution board so if an entire board gets taken out we still have power. Commando > 5000VA UPS > APC 32a Metered PDU > Into servers and switches.

 

Works really good the only time I've had to shut things down is when having whole site works done (like when we had a new whole school feed wired in by western power)

 

The twin 32 amp is way over rated now as I've only got 3 servers, one switch and 3 routers in there thanks to virtualisation but still, always better to have too much power than not enough!

[talksr]

Thank you everyone for your input on this. This has been really helpful. I have some electricians coming tomorrow, so this is going to be so useful.

 

Commando sockets is the name I was looking for.

I have made some bullet pointed notes from everything you have posted here.

 

To be honest, it really shouldn't be an issue for me, I am going to need to get some metered PDUs ideally, and will need to get some C19-C20 leads, as I have none. This could be done gradually, which would give far less impact on the network as I would still have the conventional 3 pin plugs.

 

I currently have three APC UPS devices.

1 x APC UPC SMT1500RMIZU

2 x APC Smart UPS x1500VA

 

To be honest, I am shocked that the current power solution has never been picked up on before. Before my time, this entire school burned down because of a fire! Really makes you wonder.

 

I will keep you all updated, but thanks so much for the help, examples of what you have and suggestions.

Edited November 22, 2018 by talksr

[talksr]

Just a quick follow up guys.

 

The electrical company have been in and quoted.

 

This quote does not include any PDUs or cables, but I can source those myself.

It seems a fair price and seems to cover what we need it to.

 

What do you think?

[MatthewL]

I don't think that is a bad price, not up on electrical bits but from past experience looks right.

[PotNoodleTech]

Seems reasonable as it's armoured cable etc.

[Andrew_C]

Please be aware that the racks should have a permanent earth connection, and not rely solely on the connector. This is due to the earth of your server room being exported to other locations. Unless adequately bonded, a fault in this other location could raise to local earth potential - and this is dangerous.

[PotNoodleTech]

This is due to the earth of your server room being exported to other locations.

 

Care to explain this a bit more?

 

The earth for all buildings should be earthed in the same way, through all of the existing power cabling right back to the main input coming from the electricity supplier. You should only have one earth in your school, back to the main supplier through the ginormous armoured cable that is plugged into your school?

[Andrew_C]

Care to explain this a bit more?

 

The earth for all buildings should be earthed in the same way, through all of the existing power cabling right back to the main input coming from the electricity supplier. You should only have one earth in your school, back to the main supplier through the ginormous armoured cable that is plugged into your school?

'all extraneous metal work liable to be energised in the event of a fault shall be bonded together' In this case the metal rack must be bonded to all the other metal in the building. Pipes, frame etc. You cannot use a plug and socket in this case.

I know lots do, but you shouldn't.

[TechMonkey]

Care to explain this a bit more?

 

The earth for all buildings should be earthed in the same way, through all of the existing power cabling right back to the main input coming from the electricity supplier. You should only have one earth in your school, back to the main supplier through the ginormous armoured cable that is plugged into your school?

 

I have a feeling this came up before and this is due to if there is an imbalance of earth in two buildings the difference could well be offset across an ethernet cable, causing issues. This is why fibre is always recommended between buildings. There is probably some very pertinent technical words and explanations that someone will be along to fill in but effectively it leads to having "a bad time"™

[Andrew_C]

Oh, and depending on your earthing arrangements, you are wrong with your one point of earth.

[PotNoodleTech]

Oh, and depending on your earthing arrangements, you are wrong with your one point of earth.

 

Please expand? Reason I ask is that I am not an electrician, but that appears to be the way things are wired here and we've only just had a brand new feed (and substation) wired in by western power. There's a huge ground bus bar grounded to our main incoming feed which then feeds a new ground cable out to all other buildings and each building has a mini version of the ground bus bar which everything is grounded to?

Edited November 27, 2018 by PotNoodleTech

[Andrew_C]

Please expand?

OK, back from the dentists' waiting room...

 

There are several possible earthing arrangements for buildings. Some, as you say have a single earth point , but a common arrangement is PME - "protective multiple earth" whereby all metal services are connected together, and to neutral somewhere near the building. At the other extreme there is TT which relies solely on the mass of the earth for the return to the generating station. In fact, the regs have a whole chapter on this stuff that I've not looked at for 25 years.

 

Now, for the wiring of earth within the building:-

Most people (I hope) understand that the earth connection is there to allow a sufficient current to flow in the event of a fault to cause the protective device to clear in a timely fashion. This includes RCDs, MCBs and old fashioned fuses. The time is up to 1/2 second for a fuse.

 

The second less well understood function is to ensure that you don't get a life threatening shock if you happen to be touching something metal with one hand, and a faulty appliance with the other. Imagine if you will pushing down the toaster lever and touching the metal case as a fault develops. Your other hand is on the tap. You die...

 

With earth bonding, the case of the toaster is connected to a wire back to earth in the system, this in turn is connected to the water pipes. Thus as the case rises up to 230V, so does the water pipe. There is then no potential across your body. You live... This is referred to as "Equipotential Earth Bonding" and is why there's lots of green & yellow wire linking pipes together under sinks and similar locations.

[Andrew_C]

if there is an imbalance of earth in two buildings the difference could well be offset across an ethernet cable

There's that too, but it ain't going to kill you!

[talksr]

Hello everyone.

It has been a while since my original post, but these things take time to pass through.

I now have three commando sockets installed and on the wall going to a dedicated fuse box which runs directly from the main power system coming into the school.

 

My next job is to order some PDUs for both of the racks.

We currently have two 12 plug PDUs, they are just standard ones with standard plugs on them.

I don't seem to have a front view of them but they have been cable tied into the rack!

 

 

I have three APC rack mounted UPS battery backup units.

I don't have rear images of all three of them, but I am assuming they will all look like this which is the number one UPS:

Was hoping they would have had the IEC320 C20 plug holes, but it seems not.

 

 

Was thinking of going for APC PDUs. Not too sure on what model would be best, but there seem to be ones which are metered?

 

Does anyone have any recommendations.

In terms of the rack capacity, not physically there today, but off of the top of my head:

7 x managed switches

2 x managed wireless controllers

3 x HP servers with dual PSUs all connected up

1 x CCTV DVR

1 x Mac Server

1 x APC Temperature Control Sensor

3 x APC UPSs

1 x LCD monitor

1 x Cisco Router

1 x Cisco firewall

1 x Cisco VoIP

 

Think that is about it.