Do schools spend more money on back-end ICT than is necessary?

[glennda]

Sounds like the mess was there before you. I've found that any vendor working to a non technical spec with no/poor technical oversight will give a post-event technical reviewer the hebejeebies. If it involves their own bespoke software then it gets worse. Avoid these two elements and you get a good service.

 

On costs: I know that if during the lifertime of the system my entire team walked, the school would not need to spend a single (unbudgeted for) penny keeping the servers, switches and wireless running until they found replacements, and then the replacements would have access to the techincal architects to bring them up to speed on how it was hung together. This with the benefits listed earlier for less than 1 years worth of the salary of a London NM. It all depends on the funding and strategic planning/vision of the school.

 

Problem was before we got there it was Viglen completely in Charge - it was a fully managed service so you can only really blame ermm Viglen - there shoddy hardware desktop wise along with some very badly configured servers and Classlink....... enough said after that word!

[sidewinder]

Problem was before we got there it was Viglen completely in Charge - it was a fully managed service so you can only really blame ermm Viglen - there shoddy hardware desktop wise along with some very badly configured servers and Classlink....... enough said after that word!

 

Reading the words Viglen and Classlink I feel your pain

[MrWu]

You have to wonder why HP are going so cheap on their SAN kit atm... is it the competition from the likes of the EMC VNXe eating into their market or are they trying to shift old stock before bringing out something newer in the near future? Guess that's the "joy" of IT... when do you jump

 

I have a feeling they will bring out a MSA P2000 G4 to compliment the Gen 8 servers...I've got DL Gen 7s and don't mind grabbing some bargins and they all match in the Cab! :-)

[glennda]

I have a feeling they will bring out a MSA P2000 G4 to compliment the Gen 8 servers...I've got DL Gen 7s and don't mind grabbing some bargins and they all match in the Cab! :-)

 

TBH the P2000 is a Dot Hill San rebadged. They are not compliant with VMware ESXI 5.0 hence why we no longer use them. As you are going Hyper-V you should be ok!

[SYNACK]

TBH the P2000 is a Dot Hill San rebadged. They are not compliant with VMware ESXI 5.0 hence why we no longer use them. As you are going Hyper-V you should be ok!

 

We use an older MSA2012i with Hyper-V no problems, it does have its advantages when it comes to driver support.

[glennda]

We use an older MSA2012i with Hyper-V no problems, it does have its advantages when it comes to driver support.

 

Yup - i used to use the MSA2012FC with Linux but then Linux doesn't tend to mind what you throw at it! As said above the only reason i'm getting rid of the P2000 is to move to VMware 5.0

[oxide54]

You wouldn't if you saw the mess me and @oxide54 had to clean up from (I essentially watched him 4 years ago but learnt a lot). You wouldn't go near them. Luckily I havn't had to deal with RM directly (apart from get software from them which took 4 weeks to give me a cd) so don't have a direct gripe.

 

The Equallogic one is a client I am currently setting up in London. Its nice SAN to install as it does most of the work for you with its tie in's to VMware (essentially you install the module to Vcenter tell if you want a new datastore it creates a lun and maps it to all hosts for you). It also has a dedicated MPIO config which load balances the nics better. Its connected to 3 x HP DL380 G7's booting from SSD with 100GB ram both Dual 6 core Xeons. Its soon to be running 50 VDI desktops along with around 20 Servers. Sadly they could only afford the 1GB ISCSI model not the 10GBE which would have been nice!! Its replacing an ISCSI MSA2000 but they also have a Dot Hill for backups.

 

I'm quite lucky thats the San of choice my boss recommends to clients. If they cannot afford that we tend to look towards OPEN-E on HP Hardware for ISCSI SANs.

 

viglen phoned me up trying to flog me machines the other day, after rolling around on the floor and splitting my sides in laughter for around 10 minutes, I took great delight in telling them that the only way i would buy machines from them is if they changed their company name, all their staff, their location, and their business practices.

 

+MSA 2000 is dot hill rebadged.

Edited July 8, 2012 by oxide54

[alttab]

The issue with SMT and ICT spend from a few years ago, is that back then it would have been the case that a commercial provider would have gone in

and the budget would have been there to go to town on the switched infrastructure, expensive wifi, blade servers with good SAN backend, vmware with

all the bells and whistles.

 

these days money is tight, but where there are the budgets there are demands elsewhere (think whole school ICT demands) so that

could be a desire to go bells and whistles on a VLE that SMT can see in action, or 1:1 device programs that SMT can see and get their heads around. Key issue is who is driving the ICT and what their specific points of interest are.

 

Going back to SMT a couple years down the line from a major infrastructure spend is a hard sell. And here it does help to manage the realities of what your dealing with before getting carried away with technical specs that can easily escalate. These days commercial providers themselves are having to reign in what they spec, because they know the schools aren't going to allocate the budgets in that direction. And providers obviously have to make their margins.

 

the reality is this, if you can't spend the money on redundant SAN with clustered servers, then make a best efforts approach to speccing something that gives you adequate performance and a DR strategy that you can recover from. Sub-second recovery can be costly, but is that really what you need ? based on a realistic assumption of what your budget is, how important Recovery time is over having a reliable backup plan. That sort of thing.

[Mr.Ben]

I'm looking at Virtualising the 28 servers here (they were here before me!) and consolidating several of them. The total cost for a Clustered HV Solution using SAS will be under 20k for a two node cluster with the abilty to add a third node.

 

The maintenance budget for machines and software is roughly 60k oer annum, 30k of which is spent on desktops/laptops etc. Any other development is bid for and then replacements planned for (depending on the system, cost and maintenance)

 

I see this as a 5 year spend cycle, with 20k needed each 5 years. All of the equipment is under extended 5 year warranty and so the maintenance cost is minimal.

[m25man]

The total cost for a Clustered HV Solution using SAS will be under 20k for a two node cluster with the abilty to add a third node.

 

Firstly, a two node cluster unfortunately whilst a good starting point rapidly looses credibility unless both nodes can assume the full load of its partner eg if each node is running 4 servers can Node 2 assume the running of the Node 1 VMs easily?

This tends to mean that you over spec the first two nodes, when in reality its probably better to go for the 3 nodes in the first place.

 

Secondly, can you elaborate on your chosen hardware and levels of redundancy to be achieved with a £20k budget?

 

Low budget solutions always have a trade off of some kind, sometimes the features can be added later at additional cost, would your £20k include such items as Backup, De-Dupe, Replication and DR?

 

Low cost and budget is good, but as already stated in this thread the demands and expectations of others can make £20k look incredibly small fry!

 

I think @alltab 's post makes incredibly good sense, especially as we have already heard from someone who had the full redundant option and still managed to be down for a week!

[jamesfed]

Firstly, a two node cluster unfortunately whilst a good starting point rapidly looses credibility unless both nodes can assume the full load of its partner eg if each node is running 4 servers can Node 2 assume the running of the Node 1 VMs easily?

 

Just thought I would add my 2p about 2 node clusters....

 

Although we don't have a SAN (and so no clusters) we do only have 2 virtual host servers using local storage. We've opted to go with a solution I would call 'acceptable losses' where each host has VMs that run critical services using the clustering features built right into the software (nothing at the Hypervisor layer).

That way we could lose a virtual host and keep the ‘core systems’ up and running while useful but non-essential services are kept under backup ready to be restored onto new hardware/spare resources on the running server.

Right now I consider essential to be-

• AD/DHCP/DNS (Uses built in clustering)

• File Access (DFS-R)

• Firewall/VPN/Internet Access (Built in clustering in TMG 2010)

• Lync 2010 PBX (Built in clustering)

Email and VLE is hosted ‘in da Cloud’ and I would be ok with loosing Printing, Intranet, WDS, WSUS and a few other minor things for the sake of a £££s spend.

[dhicks]

a two node cluster unfortunately whilst a good starting point rapidly looses credibility unless both nodes can assume the full load of its partner

 

Also, if you go for real-time replication between nodes you can assume that any disk writes will have to be sent to the secondary node and confirmed as written at the same time as the primary node, adding disk-write latency. You can get round this, of course, with a large disk cache and reliable UPS' (or a battery-backed RAID card) powering each node.

[broc]

On the subject of spending too much, what about backups? How much do you spend on backups & DR compared to what you spend on 'live' systems?

 

We have a SAN with 6tb data, soon to be upgraded to 12tb capacity. This will eventually support 3x Hyper-V host servers. At the moment our backups go offsite as part of a managed solution; next year we need to bring these back in-house. The plan is to use DPM for backups, the big questions for me right now are backup What and Where? The easy answer is backup Everything to a Remote location but justifying the cost for what is effectively insurance is always difficult unless you have experienced data loss on a large scale.

 

There is plenty of sound advice which says buy enterprise class disks, SAS not SATA, LTO tape libraries & don't DIY build your own storage servers but when you are looking at backing up 12tb data the costs start to mount up.

 

Do I buy another SAN & replicate? Do I buy a server chassis with lots of disk bays & fill it.. with SATA or SAS HDD? Or do I buy a couple of NAS boxes & set them up to replicate?

 

I have a building on our campus that could provide a home to equipment & give us some 'offsite' capability, but how far do you go in considering a disaster? A fire or flood in the server room (Sprinklers in a server room not a good idea) is one thing; getting Senior Management to consider a disaster of epic proportions is more challenging..... What about an accident at the top of the school drive which led to a flammable liquid spill flowing down the driveway, entering site surface water drains & catching fire... taking out the whole site?

[zag]

Cost me about 3k for our backup solution. Most of that was backupexec licenses.

 

The other 500 quid was for a basic workstation with a couple of 2tb disks in it.

[jamesfed]

We recently got lucky with our disk based backup server (the case, PSU and HDDs were donated to us by a local company) - have upgraded the innards with SAS/SCSI cards that we had spare and spent about £280 on largely desktop PC parts (mobo/CPU/SSD to boot off of/RAM) - the rather interesting hybrid of Enterprise SATA, LSI RAID5 and desktop mobo has worked a treat for us.

 

Did the maths and figured out that had we of bought it all new this 6.6TB (after RAID5) backup system would've cost us £1,400.

 

Quite interestingly enough a similar setup from RM was in the £3,300 region (not that we would ever buy off them I do enjoy pointing and laughing at their prices).

[psydii]

What is the bandwidth available for getting data off/on your SAN? The full 12TB over 1GbE will take 27hrs absolute minimum.

 

D2D2T is really the only way with that sort of file/folder based data set. Backup to the SATA disks, and from there periodic backups to tape. You can't feed tape fast enough to go direct from SAN hosts to tape. (IMO - YMMV comments welcome!)

 

Also if your SAN fabric throws its toys out of the pram, you don't want your backup data to be under its control. So maybe a second 'SAN' that's really a glorified NAS with iSCSI capabilities should host the disk element of the solution.

 

 

You really want your whole Backup system (end to end) to be backed by 24/7 support contracts. When you've got to restore that amount of data, it is going to involve nights and quite likely at least one weekend. If things come unstuck at 5pm on Friday it's a lot less stressful knowing you can pick up the phone to experts who do this sort of thing every single day who can help you out.

 

If I was building my own, I'd be tempted by using a DL380 and DAS MSA or two with a 4hr 24/7 Carepaq + Software support, and an HP LTO4 as the target. First having had the solution run through an HP configurator by one of their partners to make sure it was a supported config.

[psydii]

On the topic of BYO.

 

I've seen too many weird little (and large) problems solved by firmware updates to key components to trust anything critical that doesn't come with the option of a 5 year support contract from the OEM.

[pcstru]

On the subject of spending too much, what about backups? How much do you spend on backups & DR compared to what you spend on 'live' systems?

A not insignificant fraction!

Do I buy another SAN & replicate? Do I buy a server chassis with lots of disk bays & fill it.. with SATA or SAS HDD? Or do I buy a couple of NAS boxes & set them up to replicate?

Replication can be part of a good backup strategy, but it is not (usually) itself a good backup strategy. You may need to protect against 'failure' modes which no one notices for weeks. You don't want to be looking at a good replication of corrupt data - of maybe, judging by the number who seem to favour a few discs for backup, that's all the fashion these days.

 

I have a building on our campus that could provide a home to equipment & give us some 'offsite' capability, but how far do you go in considering a disaster? A fire or flood in the server room (Sprinklers in a server room not a good idea) is one thing; getting Senior Management to consider a disaster of epic proportions is more challenging..... What about an accident at the top of the school drive which led to a flammable liquid spill flowing down the driveway, entering site surface water drains & catching fire... taking out the whole site?

 

IMO what we should be doing is 'selling failure' to the decision makers. We know the systems will fail at some point, somehow. The key is to know how systems will cope with failure (graceful or with a bang), how you can recover from that, how long that will take and how much that will cost. That can then be balanced against the cost to the business of such a failure.

 

Now, having said that, IMO in any modestly sized organisation, a backup strategy that does not involve tape is suspect. That's a simple indicator based on cost per GB, despite the falling cost of disc, you still need too much of it in too many places for it to be sufficiently robust and cost effective.