Best way forward - Virtulisation

[twin--turbo]

When was the last time you had a host fail?

 

Last year, one of our Blades blew, and due to a bug in Citrix HA was off ( we had not been told by the installer ) so nothing migrated including one of the DC's. Our DHCP scope on the secondary was too small and only the thin clients got IP's the VDI's had no spare IP so Nowt worked.

 

Half a day later the VM's were manaual moved to another hypervisor and the system back up and running whilst we waited for HP to pull their finger out and replace the blown blade.

 

Proves you can have lots of Wizzy stuff and still be subject to failure.

 

 

The live migration is good though if you need to do any maintenance on a Hypervisor machine, we could reboot all the physical hypervisors without any disruption to service by moving VM's to otehr hosts.

 

Rob

[mrbios]

I've gone round robin, back to local VM storage now. Cost and complexity of multiple SANS in seperate buildings etc is just not worth it in your average secondary as sans are just as likely to fail as local storage (ask me how I know).

 

When using something like Veeam you can take daily full VM image backups and farm them out to several remote servers which negates a lot of the the "amazeballs" benefit of live migration etc so as well as saving many (tens of?) thousands of £ on SANS, you also avoid spending many thousands of £ on vmware licences too.

 

My whole outlook has changed. I think so many schools are wasting vast sums of money on infrastructure "what if" scenarios, "what if" scenarios that very very rarely ever come to frutition and thus very rarely prove anywhere near cost effective. When was the last time you had a host fail? 2007 for me, and that was because the server should have been replaced 3 years prior. I knew it was gonna fail eventually.

 

I digress... my 2p anyway.

 

It's arguable based on the requirements of the school, planning for what if scenarios is only one aspect of many that having multiple SANs covers. Personally I've found having the ability to live migrate VMs between hosts and storage appliances on the fly has given me huge gains and been well worth the money spent on those particular features.

 

We're actually running two of almost everything now, two exchange servers, two file servers, multiple DCs (one per building) etc, and all files on DFSR network shares, so if one building burns to a crisp everything can still carry on running from the other building without anyone even realising...BUT that is done based on the uptime requirements of the school, they realised their heavy reliance on the IT infrastructure and outlined what they expected of us and the system so we met those needs, not because we like to splash thousands of pounds on a fancy setup

 

EDIT: oh and as for host failures, twice in the past 5 years, once a RAID card and the second a motherboard died. SAN failures though? 1 and that was down to a stick of ECC memory failing rather than the SAN itself..i think i know what i prefer........also RAID6 all the way.

Edited June 21, 2012 by mrbios

[fiza]

I've gone round robin, back to local VM storage now. Cost and complexity of multiple SANS in seperate buildings etc is just not worth it in your average secondary as sans are just as likely to fail as local storage (ask me how I know).

 

When using something like Veeam you can take daily full VM image backups and farm them out to several remote servers which negates a lot of the the "amazeballs" benefit of live migration etc so as well as saving many (tens of?) thousands of £ on SANS, you also avoid spending many thousands of £ on vmware licences too.

 

My whole outlook has changed. I think so many schools are wasting vast sums of money on infrastructure "what if" scenarios, "what if" scenarios that very very rarely ever come to frutition and thus very rarely prove anywhere near cost effective. When was the last time you had a host fail? 2007 for me, and that was because the server should have been replaced 3 years prior. I knew it was gonna fail eventually.

 

I digress... my 2p anyway.

 

Thats a spanner in the works!! To SAN or not to SAN. That is the question!!!

If I can bring the VMs back online after a hardware failure as quickly as possible then SAN or not I dont mind.

[fiza]

We dont have that much money so I need to do the best thing with a limited budget.

[twin--turbo]

Does not have to be a SAN, a NAS will do for a small implementation so long as it runs a supported protocol for the Hypervisor storage subsystem ( NFS/ISCSI )

 

The best way forward, is to start.

 

Do you have an old server with say 8GB of ram adn Dual 64Bit Xeons with Hardware Hypervisor support.. If so put on ESXi and have a play.

 

I have 2 old Dell PE2800's that do it nicely for fun. they can be got for absolute peanuts on ebay ( Like they don't often sell at all!! )

 

Build a windows box on it and a linux box on it and some thing else.

 

Our old PE2900III's with 24G of ram and two 4 core Xeons were each happily runing 7-8 Linux/Netware/windws servers each.

 

Rob

[mrbios]

Thats a spanner in the works!! To SAN or not to SAN. That is the question!!!

If I can bring the VMs back online after a hardware failure as quickly as possible then SAN or not I dont mind.

 

I don't know how hyper-v works in this respect but i know with vmware at least if a host dies, you could literally use anything that'll run the hypervisor to bring those machines back online or bring them back online on another host instantly (of which your physical boxes would be powerful enough to run it all on one without issue) whereas if you're hosting them locally you're relying on backups.....now obviously if your SAN dies you again need to restore from backup, but if you have two SANs and multiple hosts you can restore those backups to the other san or host and have them back up and running instantly again......thus the benefit of multiple SANs.

[twin--turbo]

If you want Free HA, then the XEN hypervisor on another platform ( Citrix Xen is just a rehash of the opensource XEN hypervisor) such as SUSE can be made to do HA. Not looked at it for a long time though.

 

You can even build a VM to be your "test" storage server .

 

Rob

[twin--turbo]

I don't know how hyper-v works in this respect but i know with vmware at least if a host dies, you could literally use anything that'll run the hypervisor to bring those machines back online or bring them back online on another host instantly (of which your physical boxes would be powerful enough to run it all on one without issue) whereas if you're hosting them locally you're relying on backups.....now obviously if your SAN dies you again need to restore from backup, but if you have two SANs and multiple hosts you can restore those backups to the other san or host and have them back up and running instantly again......thus the benefit of multiple SANs.

 

Openfiler ( opensource san ) can do HA, not tied it but if somone wants to demonstrate to senior MGMT then it would be cool to show them the full resiliance a bit of ££££ will get them..

 

 

We have 2 HP Lefthand Sans to "Hopefully" rely on.

 

TT

[fiza]

Does not have to be a SAN, a NAS will do for a small implementation so long as it runs a supported protocol for the Hypervisor storage subsystem ( NFS/ISCSI )

 

The best way forward, is to start.

 

Do you have an old server with say 8GB of ram adn Dual 64Bit Xeons with Hardware Hypervisor support.. If so put on ESXi and have a play.

 

I have 2 old Dell PE2800's that do it nicely for fun. they can be got for absolute peanuts on ebay ( Like they don't often sell at all!! )

 

Build a windows box on it and a linux box on it and some thing else.

 

Our old PE2900III's with 24G of ram and two 4 core Xeons were each happily runing 7-8 Linux/Netware/windws servers each.

 

Rob

 

We do have old servers but these are the ones running the Network at the moment and that I want to virtualise.

[PotNoodleTech]

Hey I am not meaning to put spanners in the works there is a lot of good advice around, and I am not saying "DO NOT EVER USE A SAN", just letting you know that SANS(s) are not the be all and end all for either uptime or performance, I've been there and done that and when it came to replacing our SANS, I just simply decided not to replace them at all.

 

I had two SANS, one of them failed catastophically within 3 years. Deep down, I couldn't then trust the other one either even though it did stay up until I decomissioned it. I've had tens of physical hosts over the years and i've only ever lost one outright. Thats not a good ratio for the SANS and I hear about problems with SANS and NAS's in other schools in my area all the time.

 

If you can justify spending th £££'s going for the full on 99.9999% uptime then by all means go for it. I couldnt justify it based on the fact that my budgets are being cut year on year and some years I have not even been able to have a technician. When we couldn't afford a £15k technician I could not justify spending £15k on SANS.

[pantscat]

@AButters - (Sorry to thread hijack) What manufacturer was your SAN? (Tier 1? Sun, LeftHand, NetAPP, EMC?)

 

I'm intrigued by those Infortrend boxes... they're mega-cheap!

[PotNoodleTech]

Tier 1. That's all your gonna get out of me sorry

[pantscat]

Fair enough!

[mrbios]

Hey I am not meaning to put spanners in the works there is a lot of good advice around, and I am not saying "DO NOT EVER USE A SAN", just letting you know that SANS(s) are not the be all and end all for either uptime or performance, I've been there and done that and when it came to replacing our SANS, I just simply decided not to replace them at all.

 

I had two SANS, one of them failed catastophically within 3 years. Deep down, I couldn't then trust the other one either even though it did stay up until I decomissioned it. I've had tens of physical hosts over the years and i've only ever lost one outright. Thats not a good ratio for the SANS and I hear about problems with SANS and NAS's in other schools in my area all the time.

 

If you can justify spending th £££'s going for the full on 99.9999% uptime then by all means go for it. I couldnt justify it based on the fact that my budgets are being cut year on year and some years I have not even been able to have a technician. When we couldn't afford a £15k technician I could not justify spending £15k on SANS.

 

Ahhh i can relate to the "catastrophe = deep down can never trust again" position (never touching a motherboard made by MSI again put it that way!... nearly set fire to an IT room!) Though i think one day you'll reconsider as the benefits are there, you've just been a bit unlucky

 

I'm intrigued by those Infortrend boxes... they're mega-cheap!

 

Anything you want to know? We've had 4 + 1 JBOD in the time i've been here (ALL still running, 1 about to be decommissioned for being 7 years old)

Edited June 21, 2012 by mrbios

[twin--turbo]

We do have old servers but these are the ones running the Network at the moment and that I want to virtualise.

 

Go on ebay and spend £100 on a Poweredge 2800/2850 with 8-10GB of ram and a couple of 143GB disks. You may even get 2 servers and more power for your £100.

 

Build a test environment and don't waste thousands on stuff untill your sure your happy with what's going on.

 

Theres other costs that will mount up, L3 switch, UPS, Backup device.

 

Rob

[twin--turbo]

Dell PowerEdge 1850 1U server, ideal terminal/web server | eBay

 

Bit of extra ram for £10 and you haev a smart little virtual host for 2-3 servers or more if your frugal on the specs

 

TT

[mrbios]

Build a test environment and don't waste thousands on stuff untill your sure your happy with what's going on.

 

This +1!

 

Really i think going by what you want or rather what you're not 100% sure you want you need to go back to step 1....as you've kind of started a few steps ahead of yourself

 

Sit down analyse what exactly you want to achieve from all this, analsyse what the various ways you can achieve that by comparing different products and their features, then move on to what hardware you want and how you're going to implement your hardware, proper planning is key to implementing a good virtualised infrastructure

 

And don't forget, optimal configuration is 90% of the task, fast/big fancy hardware will only get you so far for so long.

Edited June 21, 2012 by mrbios

[plexer]

Depends on the hypervisor you want to use as hyper-v won't work on the PE2800 or PE1850 plus a load others.

 

Ben

[twin--turbo]

A recent 64Bit Desktop ( anything in the last 4 years or so ) will also most likly support Various Hypervisors, I have run Vmware and Citrix Xen on a Dell Optiplex 740 , Lenovo Athalon X2 64 ( on an ITX board USFF pC ).

 

Rob

[MrWu]

Interesting discussions here, and some of you guys have an IT budget that would put a banking corporation to shame.

I'm in kinda same position really. I do appreciate the benefits of virtualisation but I'm of the attitude that it needs to be done properly if you are going to bung all your servers into a SAN, that is DOUBLE up on everything for redundancy. I've recently priced up a very nice 3 host and SAN with fibre switch, and even utilising my existing equipment the whole project was approaching £30k+ which very hard to justify to Bursar when to him it looks like 4 boxes and a shed load of cables lol. So back to what Abutters mentioned using local hosts and good backups...this might be better solution for some. It's also setting up realistic expectations with SLAs with the SMTs, although in truth all this will go out of the window when they can't host a class due to 1 hour downtime.

 

So I'm on a crossroad as well .... Does one choose a well designed VM+ SAN ? Local storage with VMs ? Go back to old school, single servers with good backups? Maybe looking at important servers in the school(fileservers,exchange,sql) and clustering those?

Edited June 21, 2012 by MrWu

[mrbios]

Interesting discussions here, and some of you guys have an IT budget that would put a banking corporation to shame.

I'm in kinda same position really. I do appreciate the benefits of virtualisation but I'm of the attitude that it needs to be done properly if you are going to bung all your servers into a SAN, that is DOUBLE up on everything for redundancy. I've recently priced up a very nice 3 host and SAN with fibre switch, and even utilising my existing equipment the whole project was approaching £30k+ which very hard to justify to Bursar when to him it looks like 4 boxes and a shed load of cables lol. So back to what Abutters mentioned using local hosts and good backups...this might be better solution for some. It's also setting up realistic expectations with SLAs with the SMTs, although in truth all this will go out of the window when they can't host a class due to 1 hour downtime.

 

So I'm on a crossroad as well .... Does one choose a well designed VM+ SAN ? Local storage with VMs ? Go back to old school, single servers with good backups? Maybe looking at important servers in the school(fileservers,exchange,sql) and clustering those?

 

You're looking at it from the "doing it all in one go" sense though, i can't speak for others here but personally i've rolled out and implemented my virtualisation plan in stages over the course of 3-4 years. No massive £30k spending on one go, instead we've spent varied amounts each year for the past 3-4 years, (never more than £10k in one go apart from one year when we reallllly needed a new core switch which came out of capital expenditure) implementing bits and bobs as we go and slowly gearing it towards being a very fine tuned system, this way also allows you to keep up to date.....if we had done everything in one go 4 years ago then i expect we'd be seeing a need to spend lots in one go again in a few years time which to a bursar wouldn't have been viable.

[MrWu]

That would have been the sensible way to do it.

 

Unfortunately I wish the SMT had a bit more common sense, they talked to me about this massive project when I started before Christmas where decision has already been made to go wireless, whereby the recommendation I made was to look at the infrastructure, server and Electrics first. Being new boy or honeymoon period they did find the money but the date and deadline to get whatever done never shifted from September 2012, which in a way I shot myself in the foot for suggesting a strategy lol!

 

On the good side the school has been upgraded to OM3, UPS phase 1 done, new digital displays, DPM sorting out all backups, Forefront replacing Sophos, Exchange 2010 migrated, Windows 2008 domian installed, test wireless with Unifi installed...they can't complain for lack of progress.

 

Summer will be insane, school wide wireless, replace all switches / core + VLAN, Replace filservers, virtualise, windows7 roll out, new terminal server farm etc. new email domain, new website.

 

PS hardest part I find working at a school(my background was corporate) ICT does not have a huge voting power on decisions. Still, nice busy place to be!

 

Sorry I digress from the thread, but mrbios I would love to be where you are right now, you seem to have your setup nicely organised.

[fiza]

@mrbios - I think I will be taking a few steps back and slowing things down

 

I think I will start with the server I have and start testing/playing on that with Hyper-V and some test VMs.

[twin--turbo]

With the old single box paln you stay on the treadmil of refreshing all the hardware and going through the pain of migrating from all the old servers to new servers, and that cost soon mounts up.

 

With VM, if the performance is a bit lack luster. Chuck in some more virtual ram or an extra virtual proc.. If the Hypervisor is being stretched then It's time for an additional one and split the load.

 

We could never do what we want with traditional servers, partly because of power, heat and space restrictions emposed during new build.

 

Rob

[twin--turbo]

This is or main site server room, we have another cabling closet on this site. The other site has a 4 Cab server room and 4 small cabling closets.

 

Some of the equipment ( Dell Stuff ) was from our old buildings but we have put them to good use as VMware and Xen standalone hosts using lcoal storage. Still more work to be done as we have only just released the legacy EMC san from it's former duties.

 

 

Rob