Replacing servers in the age of virtualisation

[localzuk]

So, our servers here are 3 1/2 years old, and I'll be looking to replace them next year, possibly.

 

One of the things that struck me was that realistically, it doesn't matter that they're older any more - now that virtualisation has provided the resilience to cope with failures, do we actually need to replace them simply because of age?

 

In terms of capability, our cluster is still very capable. Sure, we can't ask it to do much more without sticking a bit more RAM in, for example, but everything else works well.

 

My question is - are people still sticking to their old rolling replacement program for their servers now that it isn't really necessary?

 

Or, are you doing what I'm thinking of doing - compromising, and adding another node next year, and another when one of the old ones dies? That way we have the resilience for one to die without fear of everything being on one, old, server?

 

Note also, I'm not talking about storage - that's an entirely different matter.

[rich_tech]

I don't see the point in replacing servers if their still under warranty and working well, I don't think Virtualisation puts major strain on things these days, when configured correctly other than storage and the network with writes and events occuring. MS have improved it too to be fair on the OS.

 

I dont know what cover you have on your servers, but our virtualisation nodes came with 5 years warranty from HP and they have been completely solid in that time, (seperate issue I know, but the SAN is a little more questionable from my own experiences, though we did opt for an entry level P2000 here, which has had quite a few issues in its time and parts swapped out for me to say I am happy with it overall).

 

I would probably add on as your going and retire when failing or issues arise there myself.

[localzuk]

Warranty, we have a 4 year warranty, but can extend to 5 (although, that extra year wouldn't be cheap!).

[Gardinho]

I wouldn't replace for the sake of it. What hypervisor do you use?

 

How many VMs do you have per host? If one failed could the estate handle the entire VM allocation prior to replacing the server?

 

I still have an old esxi server running some less important VMs. If that server fails the vms will automatically migrate to the other hosts, then I will worry about replacing (or introducing) a new server to the estate.

[Sagima]

All of our production servers are replaced when the warranty expires after 5 years.

[localzuk]

We're running Hyper-V. Our setup consists of 2 VM servers, with 15 VMs on each at the moment, but our situation is more complex (soon to be a MAT, and we're gonna centralise a bunch of stuff). Can run everything off 1 server just fine, but I dislike the idea of all our eggs being in 1, old, basket. Especially if the servers are servicing more than 1 school.

 

I'm more interested in finding out what you guys do with regards to replacing servers now, rather than my specific situation (as its also about long term planning). Would you run VM hosts without an active warranty, for example?

[zag]

No need for rolling replacement in my opinion since the advent of the SSD.

 

Replacing equipment should be about need and capability rather than the timescale or warranties these days.

 

I haven't used a server warranty since about 2010.

[pcstru]

I'm more interested in finding out what you guys do with regards to replacing servers now, rather than my specific situation (as its also about long term planning). Would you run VM hosts without an active warranty, for example?

Some. We have redundancy built in both to storage and our hypervisors so we can lose some capacity before we need to make hard choices about which services we would run. We still maintain some servers on warranty (our storage nodes for example) simply because the cost of a quick replacement is potentially quite high and disruptive.

[CyberNerd]

Have you considered running some of the services in GCE,AWS or Azure ?

From what I've seen of pricing they seem to be cost effective for on demand processing, but less so for things that run 24/7

[localzuk]

Not much of what we do is on demand.

[pcstru]

No need for rolling replacement in my opinion since the advent of the SSD.

Replacing equipment should be about need and capability rather than the timescale or warranties these days.

Need and capability of what? I run rolling replacement for servers on a 5 year plan, not because the servers themselves are unfit at that point but because financial constraints mean that spreading the cost that way is nicely controlled. If I wait and replace only when they absolutely must be replaced, I potentially face a huge bill over a short time which we will then struggle to do. Senior management will also then become used to a lower spend per year and when I then unexpectedly pitch up asking for tens of thousands of pounds to replace kit, they may at that point have other priorities. Strategic financial planning necessarily involves planning replacement of kit - not quite irrespective of capability (it is an informed prediction on the equipment lifetime but like all long term predictions, it has quite high error/tolerance ), or in other words - the "need and capability" of the hardware may not be the ultimate deciding factor.

[localzuk]

Need and capability of what? I run rolling replacement for servers on a 5 year plan, not because the servers themselves are unfit at that point but because financial constraints mean that spreading the cost that way is nicely controlled. If I wait and replace only when they absolutely must be replaced, I potentially face a huge bill over a short time which we will then struggle to do. Senior management will also then become used to a lower spend per year and when I then unexpectedly pitch up asking for tens of thousands of pounds to replace kit, they may at that point have other priorities. Strategic financial planning necessarily involves planning replacement of kit - not quite irrespective of capability (it is an informed prediction on the equipment lifetime but like all long term predictions, it has quite high error/tolerance ), or in other words - the "need and capability" of the hardware may not be the ultimate deciding factor.

 

Indeed. In which case, are you giving servers a longer life expectancy now than in the past?

[CyberNerd]

Not much of what we do is on demand.

 

SIMS SQL would be a good candidate. I'm not sure how to do a clustered MSSQL but I think running it on a low powered machine and booting up new servers in the cloud on demand would be a good trick to both keep costs down and provide a nice backup strategy.

[localzuk]

SIMS SQL would be a good candidate. I'm not sure how to do a clustered MSSQL but I think running it on a low powered machine and booting up new servers in the cloud on demand would be a good trick to both keep costs down and provide a nice backup strategy.

 

That would very much depend on such a setup being supported by Capita.

[CyberNerd]

That would very much depend on such a setup being supported by Capita.

 

I think that's how they do their hosted SIMS, which runs in Azure. Presumably very large academy chains must cluster too?

[Dave_O]

Would you run hosts without an active warranty?

 

Yes.

 

If you are using servers with no moving parts (hypervisor from USB) then from my experience they run fine for 7-8 years or more. I'd suggest buying one new server with shed loads of ram (enough to run the majority of your VMs on) with 5 years warranty, add that into the mix. Then replace one existing in 2 -3 years (or on failure of existing) decommissioning the old ones as and when necessary. Tier your VMs with mission critical on the new kit and lesser VMs on the older. Replacing every 3 years is a waste of good money in a school environment. My old first generation IBM X3650s could still run all of our server estate now and I bought them back in 2006. Moving forwards on site server requirements are going to reduce in the next few years anyway so no point in investing in a lot of kit.

 

On a different yet similar note, laptops and desktops, there really is no point buying long warranties on these. Just buy an extra machine and use that when one dies. We presently have less than 40% of our end user devices covered by warranty. The OptiPlex 780s with 4Gb of memory are still fine after 6 or so years. Looking to use them for at least another 2 years.

[localzuk]

On a different yet similar note, laptops and desktops, there really is no point buying long warranties on these. Just buy an extra machine and use that when one dies. We presently have less than 40% of our end user devices covered by warranty. The OptiPlex 780s with 4Gb of memory are still fine after 6 or so years. Looking to use them for at least another 2 years.

 

Depends how much the warranty costs. A 5 year warranty for us isn't too expensive on the client machines we buy.

[pcstru]

Indeed. In which case, are you giving servers a longer life expectancy now than in the past?

Yes. This year I was asked to half the IT budget and extending the replacement schedule is important to being able to do that. I haven't quite finished grappling with the consequences of the cuts yet, but right now I'm assuming our VM hosts will now be looking at 7 years in the rack - assuming we can increase spending again in two years. I think we have been quite fortunate over the last few years so we are in good shape to weather a bit of austerity on the infrastructure, somewhat less so when it comes to curriculum or staff machines!

[Dave_O]

Need and capability of what? I run rolling replacement for servers on a 5 year plan, not because the servers themselves are unfit at that point but because financial constraints mean that spreading the cost that way is nicely controlled. If I wait and replace only when they absolutely must be replaced, I potentially face a huge bill over a short time which we will then struggle to do. Senior management will also then become used to a lower spend per year and when I then unexpectedly pitch up asking for tens of thousands of pounds to replace kit, they may at that point have other priorities. Strategic financial planning necessarily involves planning replacement of kit - not quite irrespective of capability (it is an informed prediction on the equipment lifetime but like all long term predictions, it has quite high error/tolerance ), or in other words - the "need and capability" of the hardware may not be the ultimate deciding factor.

 

With good financial and IT systems planning it would not be unexpected. We use contingency funds, risk management and regular meeting to predict short to medium term (up to 3 years) expenditure. Longer term IT planning is more of a crystal ball gazing exercise however that's only based on my 30 years of doing this kind of stuff.

 

With the prevailing market trends, the government's lack of faith in IT in education, I see IT becoming more of a back seat passenger in the short term. In the longer term, I would hazard a guess that IT in schools will be driven and guided, quite rightly in my opinion, more by e-learning educational specialist rather than those with a profusion of technical skills, making these kinds of questions of little relevance.

Edited April 18, 2016 by Dave_O

[Dave_O]

Depends how much the warranty costs. A 5 year warranty for us isn't too expensive on the client machines we buy.

 

Let's say it adds an extra £10 to the cost of each machine to move from 3 to 5 years. If you buy 30 machines that's £300 for which you can buy an extra machine. Looking back at our records we have had 0 machines with problems in years 3 - 5. So every 30 machines you we would have wasted £300. Most of our failures (usually power supply or HD) occur in years 5 - 7. Last years stats on the 5 - 7 year old machines was 2 power supply failures and 2 HDs. There was no cost for these parts as we took them from the spares we had.

[localzuk]

Let's say it adds an extra £10 to the cost of each machine to move from 3 to 5 years. If you buy 30 machines that's £300 for which you can buy an extra machine. Looking back at our records we have had 0 machines with problems in years 3 - 5. So every 30 machines you we would have wasted £300. Most of our failures (usually power supply or HD) occur in years 5 - 7. Last years stats on the 5 - 7 year old machines was 2 power supply failures and 2 HDs. There was no cost for these parts as we took them from the spares we had.

 

You've been lucky with zero failures. We have about 2 out of 30 have some form of issue every year, during warranty years. That's across makes and models too, so not even just one bad brand.

[2ilent8cho]

We just decommissioned this year 4 servers we bought in 2006 and had been on 24/7. They were still working but we had things like a few power packs in the redundant PSUs go and quite a few disks start to fail in the RAID so we decided it was time to start replacing.

[Dave_O]

Impressive! 10 years

[pcstru]

With good financial and IT systems planning it would not be unexpected. We use contingency funds, risk management and regular meeting to predict short to medium term (up to 3 years) expenditure. Longer term IT planning is more of a crystal ball gazing exercise however that's only based on my 30 years of doing this kind of stuff.

"unexpected" was perhaps a poor choice of word. I'd agree you can use risk management to some degree to plan for the "expected unexpected", but there are limits in a school environment building in proper contingency since we can't carry forward funds and a contingency pot would need to be large enough to cover replacement of (say) our SAN but might well not be used. I can't see senior management being terribly happy with an approach which maximises the money tied up in that way. Warranties would probably be more desirable in that sense since the expenditure is planned, effectively we offload the risk to the equipment suppliers and minimise the year to year expenditure.

With the prevailing market trends, the government's lack of faith in IT in education, I see IT becoming more of a back seat passenger in the short term. In the longer term, I would hazard a guess that IT in schools will be driven and guided, quite rightly in my opinion, more by e-learning educational specialist rather than those with a profusion of technical skills, making these kinds of questions of little relevance.

I'd like to think that here we operate in harness with the pedagogy experts setting out the requirements and the IT specialists then need to provide infrastructure to facilitate those. We did have a head of ICT who had a whole school role driving forward IT in the curriculum and who thought he should be in overall charge of IT. I think for it to work well, that person would have to have great breadth and depth of knowledge. I suspect that the bar to the future you predict is that there are just not many around with those skills and knowledge.

 

We also deal with requirements which come from administration (and in our specific case by two businesses) and teaching staff are generally a little light on understanding the requirements there - so while I think that in a school the student outcomes are the most important consideration, there are other factors which will make a good working partnership between IT professionals and teachers the most common way to get the best value from IT in larger schools.

Edited April 18, 2016 by pcstru

[Dave_O]

We also deal with requirements which come from administration (and in our specific case by two businesses) and teaching staff are generally a little light on understanding the requirements there - so while I think that in a school the student outcomes are the most important consideration, there are other factors which will make a good working partnership between IT professionals and teachers the most common way to get the best value from IT in larger schools.

 

The time of the IT Manager are, I believe, drawing to an end. The number of times infrastructure and strategic decisions have to be made are very limited. How often do you change wireless or switches? As for the on-site stuff, as stated previously, on the decline. You just have to look at the pittance that is being offered in salaries for an IT Manager to show how much schools value IT technical skills. As for a symbiotic (my word) partnership between IT professionals and teachers, please! Call me a cynic, but after 31 years with time on both sides this is an illusion. Teachers and I do have to cite the new ones in particular here, stick with what they know best and suits them. So getting them, for example, to move their "stuff" to O365 will only work when you make their home folders read only, tell them they have 6 weeks and remove the drive after that. As for getting them to stop using removable media... Symbiosis or partnership suggests there are benefactors on both side. There are not. I see it more as a oligarchical relationship.

 

When teaching staff understand this relationship, only then can we move forward.