Make a case for new Servers

[BKGarry]

Good afternoon all,

 

I need to tap your collective wisdom and advice here,

 

Currently we have 10 physical 2003 servers, which are 5 Years old and were put in as part of BSF. We are currently in the process of coming out of our service contract and are taking our IT services back in house. Over 2 years ago the LEA contract finished and the data centre was dismantled with use having a contract direct with the old contractors and have a completely onsite solution (which is a plus)

 

I am currently fighting the battle that we need to replace our servers, especially with the age of them and 2003 server coming to the end of its life. The business manager is on my side but the headteacher is saying that we don't need to do this and that we need to improve innovation on the front end. I am trying to put forward the argument that a lot of innovation on the front end needs a decent and modern back end to work but I am failing to win her over.

 

I have costs of how much it will be to do it, and managed to get it down to 21k mark for hardware investment but still cannot get it her to see my side of things. Now I need to come up with something that would really make her happy and allow us to move past Win 7 laptop clients (we already have some iPads but trying to use the shared is a nightmare) and we have no physical computer rooms, so whatever I need to impress has to cover what we can do at present and show something awesome.

 

Also I need a backup plan in order to upgrade all of our current server OS's to 2012 r2 with minimal downtime and maximum impact, with the domain upgraded and being able to support a better range of devices.

 

Kind of getting lost in it all now, and don't know how to try and get through this, to thought I would tap up the wisdom and ask if anyone has any suggestions on how to keep things working to their best and try and improve everything.

 

Look forward to the advice and any suggestions that come through this,

[machin05]

Surely the end of Microsoft support and security updates/patches for the server OS should be enough of an argument. Are they out of warranty now too?

[Ephelyon]

Could you get the costs down further by going virtual at the same time?

[BKGarry]

That is going virtual @ephyelyon we are using 3.5tb of space at the moment and yes @machin05 they are out of warranty

 

I don't want tonnes of physical servers anymore :-(

[AlexB]

How are the warranties looking on the existing servers? If they have expired, are spare parts for the servers readily available and how long do you expect them to be available?

 

If you get past that with bad answers, the next question is. What impact would server X failing have, if there being no parts to repair it in a timely fashion?

 

You can apply pretty much the same logic on the software side once Server 2003 goes EOL.

[Ephelyon]

Have a word with @ICTDirect_Dave to see if they have any good deals on refurb server hardware that comes with a decent warranty (look at a separate maintenance agreement too). We recently saved a bomb with them on last summer's server refresh; we got the project in under £10K, and that's with hosts with 24 cores and 128GB RAM each!

[BKGarry]

Warranties finished in 2013 and the build date on them is January 2010 model is Dell PowerEdge 1950

 

I am already having trouble getting HDDs for the disk array ad having to buy SAS2 over the SAS1 drives already in there. I am truly scared if something else fails in them

 

- - - Updated - - -

 

I'll see ICT direct at BETT tomorrow :-)

[Michael]

I agree in principle that you should have at least 2 possibly 3 physical DCs and everything else should be virtual. There's definitely more of a benefit at Secondary or Higher Education, but not so much at Primary when most of them only have 2 physical servers, both acting as DCs for redundancy.

 

10 servers even in a big server room is going to take a lot of space. Rack based servers would probably be the way to go also.

[teejay]

Approach it from a different angle, look at creating a Disaster Planning and Recovery policy and look at the impact to Teaching and Learning, Safeguarding, Data Protection etc, also required timescales for recovery. This ideally needs the input of SLT and Governors, not just yourself. We did this a few years ago and went from servers held together with string like you to a fully virtualised server infrastructure with failover to another building due to the Governors demanding a proper solution, but this only happened by documenting the impact of server failure rather than saying we needed shiny new servers. We have also leased the servers rather than purchasing as this now appears as a budgetary fixed cost and also means the servers are replaced every 3 years, so we don't end up with ancient equipment on its last legs.

[E_G_R2]

The business manager is on my side but the headteacher is saying that we don't need to do this and that we need to improve innovation on the front end.

 

Reminds me of a place where I once worked. Just before I joined the LEA had told the ehad that he needed to invest in a whole new infrastructure. He gave them the finger and spent all his money on bright and shiney. Net result it was all unworkable becasue the infrastructure couldn't handle the bright and shiney. Up end (and he was luckey) they managed to get an interest free loan from CISCO that covered the infrastructure upgrade that enabled bright and shiney to work.

[strawberry]

Don't compromise. make sure you write down exactly what the dangers of failure are and about the increased likelihood of failure. (risk being likelyhood of failure x impact of failure) translate how a failure would impact on learning.

[pcstru]

What exactly do you think the consequences of not replacing the hardware will be?

[BKGarry]

At the moment I quite regularly get disk failures on the SCSI disk array (locally attached to one server) which has 3 RAID 5 virtual disks and 14 physical disks in total. When one drive fails raid decides it doesn't want to continue working, windows looses all the disks, with this no one can work as all staff and student profiles, home drives, shared drives and network installed applications are on this one disk array in one server. I have to replace the disk and reboot the server to get it recognise there are disks there still. This can take between 30mins and an hour depending on how temperamental it feels like being. this has happened once in the summer holidays this year and twice since then.

 

Then what happens if we get a 2003 exploit and all our DCs and everything is on there and there is no support to fix it.

 

On the plus side, the head doesn't want new shiney things, she just want innovative use of current shiney things. but people don't like the shiney things as they don't work like the regular things.

[pcstru]

At the moment I quite regularly get disk failures on the SCSI disk array (locally attached to one server) which has 3 RAID 5 virtual disks and 14 physical disks in total. When one drive fails raid decides it doesn't want to continue working, windows looses all the disks, with this no one can work as all staff and student profiles, home drives, shared drives and network installed applications are on this one disk array in one server. I have to replace the disk and reboot the server to get it recognise there are disks there still. This can take between 30mins and an hour depending on how temperamental it feels like being. this has happened once in the summer holidays this year and twice since then.

What did the head think of those outages?

 

Without knowing the exact detail of what your servers are running, I'd suggest going back with a proposal that tries to spread out the upgrades over 2-3 years. Prioritise the critical services and try to maximise the resilience you get early. The sell on this is that the Head will not be faced with a huge bill all at once, but a more sustainable, even expenditure that allows the infrastructure to keep pace with the increasing demands that will be placed on it.

 

At the end of the day, the Head does have a Veto, so all you can do is try and be clear on the possible consequences of their decisions.

Then what happens if we get a 2003 exploit and all our DCs and everything is on there and there is no support to fix it.

At that point, you have the same conversations again, but then you are negotiating from a very strong position!

[BKGarry]

The head didn't seem to fazed by the outages :-(

 

To me it is unacceptable that the disk array is not responding as expected with the RAID not allowing a single drive failure.

 

I am looking at splitting it, that would be about 10.5k each year, as it really is a 2 and 2 jobby, what is planned.

 

My most important thing at the moment is backups, we have a nice shiney tapeloader, but we have not got software to backup from all the individual servers to it some April (as the commvault server is offsite and controlled and licensed to the managed service) For Symantec Backup exec to use these tapes I am looking at 6k for Microsoft DPM with a nice 20tb server I would be spending 5k, but how can I justify buying new hardware purely for backup when we have a tape loader :-/

[pcstru]

The head didn't seem to fazed by the outages :-(

What services are you running on those 10 boxes? How many endpoints are you supporting? How many users?

To me it is unacceptable that the disk array is not responding as expected with the RAID not allowing a single drive failure.

Technology is just pants. I've had £30K dual node SAN's taken out by a single disc _not_ failing. And then there are the horrors of RAID 5. Building reliable infrastructure tends to be expensive - perhaps a few hours outage isn't too much of an issue, even if (say) HMI were visiting, but what about data loss?

I am looking at splitting it, that would be about 10.5k each year, as it really is a 2 and 2 jobby, what is planned.

At the end of the day, a school needs IT but heads are dictators and if they are happy with the risk/cost balance that an outage or even data loss involves, you might just have to accept that. If your multi year split plan fails, perhaps approach it from the other side - ask her how much she wants to spend a year and you will attempt to construct a journey to an infrastructure that can be reliable and maintained on that budget.

[BKGarry]

We have 1300 client devices which 1200 are wireless. With about 1150 users. I will list the other details when I get home a bit later.

[BKGarry]

Ok. All servers are the same power edge model with a 2.20ghz dual core processor 4gb ram and 500gb in case storage.

1-primary dc, DNS and dhcp

2-secondary dc and DNS

3-sims and Bromcom TWF/MCAS

4-remote desktop for Mac users and spiceworks

5-safecom printing and KMS

6-cashless catering and access control

7-root certificate authority (using IAS)

8-iis (windows 2008 r2) for moodle and HAP+

9-backup Radius cert and Symantec endpoint security

10-Main file server, with all profiles, shares, home drives 8tb of storage on SCSI aray an 14 LTO4 tape loader.

Edited January 22, 2015 by BKGarry

[Oddbod]

I had this problem with a faculty head some years ago, when they bought the bare minimum during a 90k refurb project… they took the sales reps recommendation over mine… put the money on the shiny front end, total nonchalance on the backend hardware: single raid1 mirror, single tape, bare minimum memory in the server, etc…

Nice how the money just ‘appeared’ three months later when over Christmas the overloaded server failed with a downtime of two weeks. Fortunately, I had put some of my hard earned into having a backup plan (not recommended, but that's another story) Recovered all the data but reminded him just what this would do to him if it failed during exam time - how many pissed off parents would be banging on his door?

[tmcd35]

I think the best way forward is to work with the Business Manager and develop a rolling program for IT replacements. Get as flat as possible annual budget for total IT spend and present that to the head. They'll likely find that much easier to plan against and budget for than a sudden request for 10 new servers every 5 or so years. We budget £2500 per year for server replacements.

 

As other have said look at buying 1 or 2 servers now and then see how you can repurpose your existing kit to tide you over till next year. Can you use the ram/cpu/hdd from one server to boost another?

[BKGarry]

I wish I could juice up one server from another, but the only slots available are RAM and that's one of the things that isn't giving me hassle :-)