Running your servers - the cost of electricity vs cloud hosting

[localzuk]

I am doing some planning at the moment, looking at the cost of replacement server infrastructure for next school year. One thing that I am looking at is the TCO - and electricity is making up a huge part of that.

 

Running something at 500W, 24/7/365 will cost around £1900 a year come April when prices go up, and prices will go up again in October apparently too.

 

So, for a cluster of servers, a SAN, then the backup infrastructure - a couple of NAS's and a tape drive plus air conditioning, I've figured out the potential yearly electricity cost for us will be £15000 per year. And that doesn't include buying the servers themselves, or running VMWare etc...

 

For comparison purposes, I have used the Azure pricing calculator to figure out what it would cost to lift and shift it all into Azure, with a 3 year commitment, and it has come out a little bit less than the electricity alone! This is with a VPN Gateway link to us as well, so the Azure servers will be on our network.

 

The services I'm looking at are the kinda things that we can't move to pure SaaS provisions either.

 

So, is anyone else looking at doing this?

[Duranis]

I could be wrong but I don't think anything will be running at 500w 24/7 so it may not be as expensive as you think. When I looked at this for my home pc a few years back running with a 700w PSU it varied between about 30w when idle up to a max of about 500w when under stress testing loads.

 

Obviously server hardware and usage will be different but I would suggest getting a power monitor (can be got from amazon for under £20) and checking what the actual usage is first over the course of a few days.

[ordeology]

This an interesting point and something I never took in to consideration when I redesigned our new server infrastructure for a MAT about 3 years ago. However one thing I did was reduce the footprint of our server infrastructure because over 7 years we inevitably had to buy more servers and storage to keep pace with growth so the before was 19 physical servers, 2 tape libraries 2 SANs. Then the new kit came in we reduced all of that to 6 servers and 2 SAN's, I left the Trust shortly after it was all finalised to pastures new. Now thinking about it I do hope in reducing the footprint the replacement kit at least bought in some energy savings.

[localzuk]

I could be wrong but I don't think anything will be running at 500w 24/7 so it may not be as expensive as you think. When I looked at this for my home pc a few years back running with a 700w PSU it varied between about 30w when idle up to a max of about 500w when under stress testing loads.

Obviously server hardware and usage will be different but I would suggest getting a power monitor (can be got from amazon for under £20) and checking what the actual usage is first over the course of a few days.

A server cluster running 40 VMs, and an array of 32 SSDs, plus aircon etc... is not like a home PC. The actual servers have dual 1KW PSUs. (This cluster runs a MAT, rather than a single school) Aircon alone uses at least 1KW.

 

Not to mention a £20 power monitor to look at the power usage of a rack plugged in via 16A sockets? Not ideal...

Edited March 31, 2022 by localzuk

[mavhc]

Actually measure the power usage before making drastic changes. You'd hope the school had per circuit power monitoring. What does the school pay per kWh?

 

Are you servers running at 100% cpu 24/7?

[Rob_D]

Not to mention a £20 power monitor to look at the power usage of a rack plugged in via 16A sockets? Not ideal...

 

If you have a Smart UPS wouldn't that have logs of power usage? And even if you've got a non-smart UPS you could probably still use it to get some real world numbers (unless that's where you got your original 500w from, in which case ignore my rambling).

[PotNoodleTech]

People always overestimate how much power equipment uses. Just because it's got 2 x 1000w PSU doesnt mean it uses 2000w all day long. Probably less than 200w most of the time.

 

For example, my server cabs are currently in the middle of the school day using 1.37kw accurately measured. That's 4 x Dell Servers, 1 x CCTV 1 x Mitel Phone 1 x huge 4u cisco router, 2 x aruba core switch and 1 x small cisco router and 1 x BT fibre interface.

 

So that's 31.2kwh per day or £3.40 per day, £1252 per year at current prices for the entire lot.

 

Don't get me wrong these energy price increases are going to make it more expensive to run your servers 24/7 and it's a good idea to virtualise and reduce the requirement for physical equipment but is it worth ripping out your entire system because of £1252 worth of electrciity costs? We're really talking minuscule figures for server rooms compared to a secondary school's entire site wide electricity usage.

[Davit2005]

I could be wrong but I don't think anything will be running at 500w 24/7 so it may not be as expensive as you think. When I looked at this for my home pc a few years back running with a 700w PSU it varied between about 30w when idle up to a max of about 500w when under stress testing loads.

 

Obviously server hardware and usage will be different but I would suggest getting a power monitor (can be got from amazon for under £20) and checking what the actual usage is first over the course of a few days.

 

My home network runs at 400w coonstantly at the moment. That includes 2 low powered hypervisors, a few layer 3 switches, modems, routers, 2 Synology desktop NAS units (one of those is full of SSDs), Smart Home hubs and APs.

 

But cannot really compare a home network to a work network. Running my few VMs still works out cheaper than hosting them in any cloud I've looked into for personal use whilst I have got an Azure subscription with one lab VM as pay as you go. Also do not have to think about cooling at home for 9-10 months of the year, where I live anyway :-) Also systems used for productivity need to be available in the most part 24/7 and protected from downtime (and damage) caused by heat and humidity.

 

Cloud/third party hosting for an org could prob work out a cheaper option with a solution with SaaS if possible, PaaS or if need be IaaS for custom or where there is not a SaaS/PaaS solution.

Edited March 31, 2022 by Davit2005

[jthompson]

I was going to say that a UPS with a management card could give you a pretty accurate cost calculation. Our APC unit includes weekly kWh usage logs and a cost calculator. The device itself can also display the instantaneous output Wattage. We monitor it via SNMP but sadly that particular metric isn't one of the things that's monitorable.

 

I guess it's difficult to judge what the cost would be for equipment that you don't own yet

[southhamster]

When we specced our servers we looked at the L versions CPU which are lower power. The Dell idrac gives an accurate measure of power consumption and From memory graphs as well. Also SSD's should use less power.

[stevewill6]

I just checked the ILO on our HP servers and it gives a graph of energy usage. Servers draw on average 150w each.

[localzuk]

@PotNoodleTech - you may want to re-look at your unit pricing. When existing contracts end, the unit price will be going up a lot. We're going from 13p per unit here at the moment, and if it is anything like consumer pricing, we're looking at 44p a unit instead. So, your £3.40 a day ends up nearly £14 a day.

[Andrew_C]

Has commercial pricing ever been capped?

[localzuk]

Has commercial pricing ever been capped?

Don't think so, no. But most schools operated fixed pricing contracts and a lot of those seem to run April -> March. At least they do in Somerset from what I've seen.

[dhicks]

You'd hope the school had per circuit power monitoring.

 

Is anyone using a per-circuit power monitoring system of some sort? It sounds like a good idea - is this something you can install on current electricity infrastructure without having to re-cable anything, or is this something you need to get an electrician in to sort out? I'd hope we could simply buy wireless-enabled circuit breakers to swap in for the current modules in consumer units, is it that simple?

[mavhc]

You could get ones that use Current transformers, like you might put in your home to measure solar/mains/battery usage, battery powered, connect to usb dongle using wireless.

 

If you're editing circuit breakers you'd need an electrician anyway, in which case, sdm120m, uses rs485 serial to send data, wired in directly, about £15, plus about £10 for a usb to rs485 dongle.

[Jamman960]

Cloud hosting prices will no doubt rise at some point to account for the rises, I imagine their margins aren't big enough to absorb the increase even if they wanted to.

[dhicks]

If you're editing circuit breakers you'd need an electrician anyway, in which case, sdm120m, uses rs485 serial to send data, wired in directly, about £15, plus about £10 for a usb to rs485 dongle.

 

Does that mean we would get an electrician to swap out the mini circuit breakers (the MCBs) in a consumer unit for Eastron SDM120M MCBs, then there is some way we can connect those MCBs to something via wireless to start logging power usage data? Would we just need the one wireless connection per consumer unit, or one per MCB? Looking at Eastron's website:

 

https://www.eastroneurope.com/products/view/sdm120modbus

 

they only seem to do a 45A version - I think most of our mains circuits are 32A, with some lower-rated lighting circuits.

[Andrew_C]

Is anyone using a per-circuit power monitoring system of some sort? It sounds like a good idea - is this something you can install on current electricity infrastructure without having to re-cable anything, or is this something you need to get an electrician in to sort out? I'd hope we could simply buy wireless-enabled circuit breakers to swap in for the current modules in consumer units, is it that simple?

 

Or, alternatively, someone is talking out of their hat. (again)

[Koldov]

It is possible that in a year or so energy rates might become more stable and so the unit prices might reduce back to reasonable levels* and so the expected calculation on supposed 'extra' future savings will be null and void.

 

However, at least you would have factored in a 3 year static cost for your budget as opposed to variable energy prices (hopefully the next contract period wouldn't be at an extortionate rate as once you'd made the move and got rid of your kit you're pretty much stuck in the cloud).

 

 

*Of course on the other hand... they may not!

[southhamster]

Does that mean we would get an electrician to swap out the mini circuit breakers (the MCBs) in a consumer unit for Eastron SDM120M MCBs, then there is some way we can connect those MCBs to something via wireless to start logging power usage data? Would we just need the one wireless connection per consumer unit, or one per MCB? Looking at Eastron's website:

 

https://www.eastroneurope.com/products/view/sdm120modbus

 

they only seem to do a 45A version - I think most of our mains circuits are 32A, with some lower-rated lighting circuits.

This will give you the idea. I'm sure there are commercial versions too. One problem with basic kit including smartmeters is they measure all current flow in and out. If you have solar PV it makes a nonsense of the data.

https://openenergymonitor.org/

[jthompson]

This will give you the idea. I'm sure there are commercial versions too. One problem with basic kit including smartmeters is they measure all current flow in and out. If you have solar PV it makes a nonsense of the data.

https://openenergymonitor.org/

 

I think current transofrmer sensors like those intended for domestic use and undone if there is a three-phase supply involved, which I gather is more likely for a school. I don't know enough about electrical stuff to know whether that would still be an issue once inside the building and downstream from the main supply into the site, though.

[dhicks]

However, at least you would have factored in a 3 year static cost for your budget as opposed to variable energy prices (hopefully the next contract period wouldn't be at an extortionate rate as once you'd made the move and got rid of your kit you're pretty much stuck in the cloud).

 

Good point, you might want to consider how you go about switching cloud providers if you wanted, whether that be for server hosting (renting VMs to install your own software on from Amazon, OVH, etc) or for as-a-service providers (how easy is it to switch between Google and Microsoft if you wanted?). I get the impresion this is where your modern DevOps-style management techniques come in - servers and the services they run should be able to be built on any suitible hardware provider, ideally you shouldn't be repliant on a specific bit of hardware to keep running. Of course, on-site savings for moving your infrastructure to the cloud might be about more than just power consumption - you might be able to fire your network manager if they're no longer needed to maintain on-site hardware :-)

 

For power consumption monitoring, it's something I'd like to be able to do, and not neccesarily just for the immediate (potential) monetary savings. It's the sort of thing that you could build into the curriculum quite nicely (how much power do we use in our classroom?), and it gives children a nice, practical, hands-on idea of what power is, how it works and what sort of scale we are using it at. I'd like people to be able to see the difference in scale in usage for a lightbulb compard to a heater / oven, and then see how even small usage can be large if run all the time (lightbulb left on all day).

[psydii]

Before we get too excited, increases in energy costs will affect cloud datacentres too once their current contracts with suppliers come up for renewal.

[MartinT]

Don't forget to factor in the cost of the aircon to keep the server room cool, especially during summer. That's a real power guzzler.