Do schools spend more money on back-end ICT than is necessary?

[CyBeRkId2002]

thanks for all the feedback! Lots to think about and many differing views in an area I have been left behind in for a while (Server Virtualisation and - in fact - server and SAN hardware in general) and will definately give us something to work on over the summer break!

 

My initial thoughts are:

 

• 60k is way too much (already knew this)!
  
• I should be able to get a decent virtualised system and SAN for under the 30k mark! I am hoping to find a solution which offers a five year warranty for this kind of price as it will be much easier to sell to SLT as a ~6k yearly cost
  

.

 

A couple of things I would like clearing up if possible:

 

• I mention in my initial post about NAS vs SAN but nobody has really commented on it rather than to say go for SAN. Why is this? Surely we could get a fully duplicated NAS solution far FAR cheaper... is that not something people have looked at?
  
• How do people select and work out processor requirements when looking at virtualisation beyond looking at the number of cores?
  

 

Final question (for now). I have calculated the amount of RAM needed on our server infrastructure (in total) as this:

 

File Servers 4gb (x3 12gb)

Exchange 16gb

SIMs 16gb

Webserver 8gb

Print Server 4gb

WDS 4gb

LCS 4gb

Cashless Catering 4gb

 

Total (approx 68gb)

 

Does this seem about right? Would I be looking at getting around this amount (75-100gb) on one server or both for complete failover??? Also, would we be best putting things like Print Server/Webserver/WDS etc. on one server or their own virtualised servers??? Seems to be a huge RAM overhead for something which one VM could handle. Some goes for file-servers really.

 

Cheers for all the advice so far and any more to come...

[CyberNerd]

 

File Servers 4gb (x3 12gb)

Webserver 8gb

Print Server 4gb

 

We serve files for 2000 users on 1GB Ram without excessive swapping.

You could run all these machines on Linux and cut a huge percentage from your system requirement and licensing bill.

I'd also outsource the exchange to the web too.

[gshaw]

£30k will give you all you need with money to spare, even more so if you don't mind Hyper-V. RAM is fairly cheap these days so you can add room for expansion without breaking the bank.

 

The price shoots up when you add a 2nd replicated SAN, in an ideal world cloud hosted replication \ backups would cover this role but it's an area that really hasn't sorted itself out cost-wise as yet. For us Veeam provides an acceptable compromise.

 

One thing you will need to do is play the game with manufacturers and suppliers, make the most of educational bid pricing, special offers and the competitive market to drive your quotes down... amazing how much margin there is in this stuff once you get going!

Edited July 5, 2012 by gshaw

[psydii]

Two thoughts:

Could you not give the individual companies that supplied the parts (SAN, Hosts, management tools etc) a call and see if you can negotiate extended warranties and support directly? If the kit is still no where near its capacity then it is quite acceptable to sweat it some more, while you soften up management to the price tag for its replacement (look - spec well initially and get 6-10 yrs out of it!)

 

 

If that isn't a go-er or the solution running on that hardware is something you feel is holding your school back, then try giving Viglen a call. They solved this exact problem for me at the price point you expect with the configuration you are looking for.

 

They did a lot of work on this to get onto the BECTA (now DfE) ICT Services Framework. You just want one of their out of the box solutions, probably with a partially managed services element on the core services e.g. SAN config, VM infrastructure and Fileserver, add others up to the point you can afford, but do most of the finer windows config yourself. Make sure that you include maintenance contracts for the various component parts they provide - often hotfixes from the vendors are the solution to irritations. ("that's fixed in version 6.0.2.23, 4.0.109.23 is really old now - can you upgrade? It will definately fix the problem you are having.")

 

On the RAM overhead: each box needs enough ram to support the full set of servers if one of your two proposed hosts fails. I believe that you can over commit by making use of the Balloon driver - but I've not tried (why mess with something that is working?) My physical hosts have 48GB, so my total VM requirements is tuned to 24GB per physical host. Same size site a you.

 

One of the design criteria we had was that I didn't want me or my team to be having to constantly be messing around on the servers - they just needed to go in, get configured and then work, so that we could spend our time focusing on directly supporting the users. The VM/SAN enviroment has certainly enabled this. Perhaps we could have delivered the same initial services for 10-15k less, but as the services evolved more and more time would have to be spend adding and maintaining physical hosts, piling on additional services onto initially well behaved servers and generally pushing them to past their limits and without an pre-existing strong change control ethic in the team that would have lead to down time and reduced front line support time. Now if we need a new service/VM we can either did through our handover notes and give it a go, or call down on pre-paid engineer time from Viglen to spin up a new serer. Easily worth the 10-15K in a single year alone!

 

Though I must say that I am quite envious of glennda's solution, I think ours might be able to do that too with a major firmware update on the san and the latest vSphere - but I'm in no hurry to mess with that.

Edited July 5, 2012 by psydii

[MrWu]

SAN vs NAS : I think NAS works on SMB level and Hyper-V r2 requires storage to be locally mounted (someone correct me if Im wrong though) Hyper-V r3 might be a game changer with this. But if you are using NAS as a replicating fileserver, then I think that's fine.

 

Price wise HP is still doing 50 percent cash back on SANs and cash back on 2nd processor if you buy double the RAM (up to 5 server on one invoice) shop around, its a pain but could save you money.

 

I managed to quote for

 

2 lots of DL380 g7 with dual Xeon 5620, 64gb+ Ram, 2x146gb 15k SAS, dual SAS 6gb HBAs, 10gb NIC

MSA P2000 G3 6gb SAS SAN, 12 x 600gb 10k SAS

SFP cables and SAS cables

 

Just over £13500 for the lot after cashback.

 

But boy make sure you budget for SFP and SAS cables, I could buy another DL380 with the prices of all the interconnects lol!

 

I do agree the OP about the backoffice end. User perception wise they do not much care for how many replicated SANS and how good your failover are while they couldn't even get an interactive whiteboard or a new bulb for their projectors...(god school budgets are as complicated and sectioned as in corporate!). We luckily had some money left in this year and will be replacing a lot of the PCs with SSDs and Windows 7, which should, with the new upgrades in the server room, give the kids and teachers a little boost when they return September time.

Edited July 5, 2012 by MrWu

[RTFM]

You have to spend money on the things you are talking about to have a reliable, resilient network. Switches and server room are the things people don't see but if you skimp on them and problems arise they certainly notice due to the fact things stop working.

 

We went from 12 old physical servers to 3 HP DL360 G7 hosts and a netapp 2040 San to virtualise everything. The resiliency and backup / restore solution we now have is brilliant. We also purchased a second 2040 for backup, fall over etc.

 

It cost us plenty of money and obviously needed some justification but at the end of the day, if there are problems we are now in a much better position to solve them quickly and with minimal impact.

[SYNACK]

You have to spend money on the things you are talking about to have a reliable, resilient network. Switches and server room are the things people don't see but if you skimp on them and problems arise they certainly notice due to the fact things stop working.

 

We went from 12 old physical servers to 3 HP DL360 G7 hosts and a netapp 2040 San to virtualise everything. The resiliency and backup / restore solution we now have is brilliant. We also purchased a second 2040 for backup, fall over etc.

 

It cost us plenty of money and obviously needed some justification but at the end of the day, if there are problems we are now in a much better position to solve them quickly and with minimal impact.

 

Yeap, it is always a balance with the budget avalible but I use the building metaphore a lot regarding building a house on a solid foundation rather than a waterlogged sandpit. The core infrastructure is key to what your client gear can actually do comfortably. If you have a good network underpinning everything then things are a lot more reliable and easier to fix.

[RTFM]

Yeap, it is always a balance with the budget avalible but I use the building metaphore a lot regarding building a house on a solid foundation rather than a waterlogged sandpit. The core infrastructure is key to what your client gear can actually do comfortably. If you have a good network underpinning everything then things are a lot more reliable and easier to fix.

 

Absolutely. A perfect analogy

Edited July 5, 2012 by RTFM

[glennda]

Two thoughts:

Could you not give the individual companies that supplied the parts (SAN, Hosts, management tools etc) a call and see if you can negotiate extended warranties and support directly? If the kit is still no where near its capacity then it is quite acceptable to sweat it some more, while you soften up management to the price tag for its replacement (look - spec well initially and get 6-10 yrs out of it!)

 

 

If that isn't a go-er or the solution running on that hardware is something you feel is holding your school back, then try giving Viglen a call. They solved this exact problem for me at the price point you expect with the configuration you are looking for.

 

They did a lot of work on this to get onto the BECTA (now DfE) ICT Services Framework. You just want one of their out of the box solutions, probably with a partially managed services element on the core services e.g. SAN config, VM infrastructure and Fileserver, add others up to the point you can afford, but do most of the finer windows config yourself. Make sure that you include maintenance contracts for the various component parts they provide - often hotfixes from the vendors are the solution to irritations. ("that's fixed in version 6.0.2.23, 4.0.109.23 is really old now - can you upgrade? It will definately fix the problem you are having.")

 

On the RAM overhead: each box needs enough ram to support the full set of servers if one of your two proposed hosts fails. I believe that you can over commit by making use of the Balloon driver - but I've not tried (why mess with something that is working?) My physical hosts have 48GB, so my total VM requirements is tuned to 24GB per physical host. Same size site a you.

 

One of the design criteria we had was that I didn't want me or my team to be having to constantly be messing around on the servers - they just needed to go in, get configured and then work, so that we could spend our time focusing on directly supporting the users. The VM/SAN enviroment has certainly enabled this. Perhaps we could have delivered the same initial services for 10-15k less, but as the services evolved more and more time would have to be spend adding and maintaining physical hosts, piling on additional services onto initially well behaved servers and generally pushing them to past their limits and without an pre-existing strong change control ethic in the team that would have lead to down time and reduced front line support time. Now if we need a new service/VM we can either did through our handover notes and give it a go, or call down on pre-paid engineer time from Viglen to spin up a new serer. Easily worth the 10-15K in a single year alone!

 

Though I must say that I am quite envious of glennda's solution, I think ours might be able to do that too with a major firmware update on the san and the latest vSphere - but I'm in no hurry to mess with that.

 

Which one do you like? My first post was my last workplace and the second one (Equallogic) is a clients.

 

Shudders - please don't mention viglen on the forum again!!

[tmcd35]

Over the past three and half years I've spent around £20k putting in a v. High end bespoke NAS and 3 good virtual host servers. Part of me would like to have the OPs problem, and I'd sure love to have a redundant SAN but honestly I don't think I could justify it. Is there any thing we do that truely requires that level of redundancy? Or isn't really just an over priced luxury? I can't help thinking there are better uses a school could put £30k to that would have more of an impact on teaching and learning - our core business after all. Hell we'd have to sack a teacher to get that kind of money to spend (maybe not such a bad idea).

 

But still, different schools, different budgets, different priorities.

 

My other concern is, ok you spend £30k now for this shiny new kit but, what about when it needs replacing. Can you be sure another £30k will be available in 5years time?

[dhicks]

You don't need a SAN for a school network, local storage on the virtual machine host will be fine. A Dell PowerEdge 510 with 64GB of RAM, dual processors and 8 SAS harddrives giving you at least 1TB of RAIDed storage for VM images is around £5,000, and that should cover the basic functions for most school networks (DC, email, printer server, MIS, etc). It's probably overkill in most cases - local SAS drives on a hardware RAID card are going to be the highest throughput, lowest latency disk solution short of a very expensive SSD array or SAN.

 

You'll need (at least) another server for user file storage, and of course you'll need another (larger) server to back that file server up (with room for previous version snapshots), but £5,000 for the two should get you a couple of servers that can take a decent number of 2TB / 3TB SATA drives. Ideally, that backup server will be in a separate location to your primary server, and you'd also have a second VM host server for failover. Importantly, your backup VM host doesn't need the same kind of specification as your primary server - single processor, SATA drives, enough RAM to run a minimal system while you repair the primary server. Hopefully a sudden, unexpected, catastrophic failure of your primary server happens very rarely, and a half hour or so's delay while you boot up VM images on the secondary server is generally quite acceptable in most school situations.

Edited July 5, 2012 by dhicks

[RTFM]

You don't need a SAN for a school network.

 

Why not?

 

If the money is available to buy one, everything else which would have an impact on the network etc has been done (new pc's, monitors, switches, wireless etc) why not get a SAN? Or in our case 2.

 

A school has, in a lot of cases, a bigger user base than a medium and large business and has critical data which needs to be securely maintained, backed up etc. If anything the work being done on a school network is more taxing on the infrastructure than in businesses when you think about all the video and photo editing, data being pushed around etc.

 

Surely it makes sense to have the best you can afford (but still get value for money)?

[teejay]

We've had 3 instances where the has hit the fan with our server infrastructure in the last year, first we had an arson attack on the classroom next to the server room, luckily the fire brigade turned up in the nick of time and we just had a little bit of smoke damage. Next we found our 'highly available' virtualisation solution (3 headers, 2 switches for redundancy and 2 SANS) went completely pear shaped due to both SANS having a problem at the same time which took 3 days to get a partial network up and a week to get the full network up. This was followed by a hardware failure on out NAS storage which took 3 days to get fixed by the manufacturer. Stressed is an understatement recently!

After discussion with SLT and governors we're now specifying a solution which has redundancy across into another building and also an offsite DR into another school. This is costing an eye watering amount of money but it is required to meet the SLT/Governors requirements for availability.

My advise now would be to do a DR plan covering the main possible disaster such as loss of server room, major hardware failure, loss of school site. It then needs a decision from SLT as to the timescales they require for each service to be up and running and then a solution specified and costed around that. It's then up to them if they want to spend the money to fulfill those requirements, if they don't want to spend that, then they have to accept compromises.

[tmcd35]

My advise now would be to do a DR plan covering the main possible disaster such as loss of server room, major hardware failure, loss of school site. It then needs a decision from SLT as to the timescales they require for each service to be up and running and then a solution specified and costed around that. It's then up to them if they want to spend the money to fulfill those requirements, if they don't want to spend that, then they have to accept compromises.

 

This is the nub of it. Redundant SANs are nice if you can afford them, but far more important is a good disaster recovery plan.

[nicholab]

I think a lot of Schools don't spend enough on the infrastructural as heads only seem to care about the bling they can see!! If you look at the total staff salary pot to amount spent on I bet is a lot lower then in the corporate world. Also schools seem bad and rolling plans for renewal. Edited July 6, 2012 by nicholab

[dhicks]

If the money is available to buy one, everything else which would have an impact on the network etc has been done (new pc's, monitors, switches, wireless etc) why not get a SAN?

 

A SAN is for situations where you need more procesing power in proportion to your storage space. The whole idea of a SAN is that it sits at the end of a network connection of some kind, dissasociated from your processing machine. This can limit throughput and add latency - obviously the higher-end SAN you buy the better throughput you get and less latency, and even iSCSI-over-ethernet really isn't bad, but you'll still get better performance by having a RAID array directly attached to your processors. In a smaller-scale setup (probably half-a-dozen processing machines or less), a SAN just makes for another box to buy, power and maintain.

 

A school has, in a lot of cases, a bigger user base than a medium and large business and has critical data which needs to be securely maintained, backed up etc. If anything the work being done on a school network is more taxing on the infrastructure than in businesses when you think about all the video and photo editing, data being pushed around etc.

 

It rather depends on what kind of scale you're operating, and what kind of budget is available over what period of time. If you're relying on a third party to provide the hardware then you'll probably get a SAN as that tends to be what people know how to specify and support, and £50,000 over, say, three years really isn't that much for IT infrastructure for what, as you point out, is generally quite a sizeable organisation. Schools still tend to be under-provided for actual storage volume available - the amount of images and video the average school can produce in a year is growing and can quite overwealm some school's storage facilities.

 

--

David Hicks

[Gibson335]

When 'selling' to SLT, it's always good to mention savings. So, for instance, we were running 12 servers, and by the time our current transition is completed we'll have a SAN and 2 physical servers. If you were on a 5 year cycle of replacement, then you have an immediate purchase and re-purchase saving. But the big 'sell' at the moment is energey...and we'll reduce 24 PSUs down to 6, which is an energy saving on hardware, not to mention potential savings on cooling. No matter what you plump for, always introduce savings and never forget energy costs.

[SYNACK]

A SAN is for situations where you need more procesing power in proportion to your storage space. The whole idea of a SAN is that it sits at the end of a network connection of some kind, dissasociated from your processing machine. This can limit throughput and add latency - obviously the higher-end SAN you buy the better throughput you get and less latency, and even iSCSI-over-ethernet really isn't bad, but you'll still get better performance by having a RAID array directly attached to your processors. In a smaller-scale setup (probably half-a-dozen processing machines or less), a SAN just makes for another box to buy, power and maintain.

If you go with a SAS SAN most of those issues of scale go away, you get faster performance than all but the fastest raid cards with stacks of drives and you get the added benefits of a decentralized storage like snapshots, multiple controllers, multipath etc. It basically means you can buy one epic storage system and tie into it from all your other machines more efficient utilization of what you have. It can also make some stuff easier, you don't need hds in your individual servers as you can just boot off a sas HBA. With some if you later need iSCSI you can just swap out the controllers in the SAN. You can easily add disk shelves for stacks more drives and this can easily be divided up or reallocated between physical hosts. In all it just makes the resources much more adaptable as you can allocate the resource pool to where it is needed. It winds me up going back to isolated storage on servers now as it is slow and limiting as its locked in and unmovable if things change without lots of messing.

[gshaw]

SAN vs NAS : I think NAS works on SMB level and Hyper-V r2 requires storage to be locally mounted (someone correct me if Im wrong though) Hyper-V r3 might be a game changer with this. But if you are using NAS as a replicating fileserver, then I think that's fine.

 

Price wise HP is still doing 50 percent cash back on SANs and cash back on 2nd processor if you buy double the RAM (up to 5 server on one invoice) shop around, its a pain but could save you money.

 

You have to wonder why HP are going so cheap on their SAN kit atm... is it the competition from the likes of the EMC VNXe eating into their market or are they trying to shift old stock before bringing out something newer in the near future? Guess that's the "joy" of IT... when do you jump

[psydii]

Which one do you like? My first post was my last workplace and the second one (Equallogic) is a clients.

 

Shudders - please don't mention viglen on the forum again!!

 

The Equallogic one.

 

Meh, if I can put up with seeing RM advertising everywhere on here, then you'll have to put up with me mentioning Viglen from time to time. Viglen Storage Group are pretty awesome and the solution works very well. We recently had a third party in to create a new VM for a bespoke app and they commented that our set up was textbook, making their job easy. You get what you ask for (just don't ask for too much - play your suppliers to their strengths and everybody is happy).

 

On those who quite rightly say that you can run a large secondary on DAS (I have, it was good). Having a well specified, designed and supported SAN based VM infrastructure reduces my teams' time spent on infrastructure management, increasing time available to support front line staff. The real magic is in the backups though. VEEAM takes (nearly) all the hassle out of it - it also provides a ready made infrastructure for testing changes before making them to the live servers. It makes 'best practice' change control management not only possible - but easy, again delivering benefit directly to the end users (quicker changes, less downtime, improved skill and discipline in the back end team).

 

Sometimes spending cash is not and expense but an investment.

[glennda]

The Equallogic one.

 

Meh, if I can put up with seeing RM advertising everywhere on here, then you'll have to put up with me mentioning Viglen from time to time. Viglen Storage Group are pretty awesome and the solution works very well. We recently had a third party in to create a new VM for a bespoke app and they commented that our set up was textbook, making their job easy. You get what you ask for (just don't ask for too much - play your suppliers to their strengths and everybody is happy).

 

On those who quite rightly say that you can run a large secondary on DAS (I have, it was good). Having a well specified, designed and supported SAN based VM infrastructure reduces my teams' time spent on infrastructure management, increasing time available to support front line staff. The real magic is in the backups though. VEEAM takes (nearly) all the hassle out of it - it also provides a ready made infrastructure for testing changes before making them to the live servers. It makes 'best practice' change control management not only possible - but easy, again delivering benefit directly to the end users (quicker changes, less downtime, improved skill and discipline in the back end team).

 

Sometimes spending cash is not and expense but an investment.

 

You wouldn't if you saw the mess me and @oxide54 had to clean up from (I essentially watched him 4 years ago but learnt a lot). You wouldn't go near them. Luckily I havn't had to deal with RM directly (apart from get software from them which took 4 weeks to give me a cd) so don't have a direct gripe.

 

The Equallogic one is a client I am currently setting up in London. Its nice SAN to install as it does most of the work for you with its tie in's to VMware (essentially you install the module to Vcenter tell if you want a new datastore it creates a lun and maps it to all hosts for you). It also has a dedicated MPIO config which load balances the nics better. Its connected to 3 x HP DL380 G7's booting from SSD with 100GB ram both Dual 6 core Xeons. Its soon to be running 50 VDI desktops along with around 20 Servers. Sadly they could only afford the 1GB ISCSI model not the 10GBE which would have been nice!! Its replacing an ISCSI MSA2000 but they also have a Dot Hill for backups.

 

I'm quite lucky thats the San of choice my boss recommends to clients. If they cannot afford that we tend to look towards OPEN-E on HP Hardware for ISCSI SANs.

[zag]

Haven't read the whole thread but £30-60k for virtualization in a 1000 student school............ You have got to be kidding me !!!!??

 

It cost us 6k in total to virtualize 16 servers. Our school has 900 students.

 

You don't need outside companies to spec it up or massive SANS to store everything. I learned how to do it on this site in a few hours. virtualboy blog - Site Home - TechNet Blogs

 

4 simple rack servers with 32gb of RAM and the hyper-v role installed will get you up and running in about a day.

 

So in answer to the question, yes your spending too much.

[nicholab]

That is still only £33 pounds a student so you should have some money left.

[RTFM]

Haven't read the whole thread but £30-60k for virtualization in a 1000 student school............ You have got to be kidding me !!!!??

 

Crikey, you won't like how much we spent then if you think £30-60k is too much :|

[psydii]

Sounds like the mess was there before you. I've found that any vendor working to a non technical spec with no/poor technical oversight will give a post-event technical reviewer the hebejeebies. If it involves their own bespoke software then it gets worse. Avoid these two elements and you get a good service.

 

On costs: I know that if during the lifertime of the system my entire team walked, the school would not need to spend a single (unbudgeted for) penny keeping the servers, switches and wireless running until they found replacements, and then the replacements would have access to the techincal architects to bring them up to speed on how it was hung together. This with the benefits listed earlier for less than 1 years worth of the salary of a London NM. It all depends on the funding and strategic planning/vision of the school.