Best way forward - Virtulisation

[zag]

I've had my servers virtualized for a few years now.

 

I wouldn't bother with SANS or redundancy personally. Thats more use for datacenters and banks who need it 24/7 100% uptime thing and are willing to pay through the nose for it.

 

In a school a few rack servers, hyper-v and a good bit of backup software is all you need. I think it cost us less than 6k to do the lot.

 

If I had a local hard disk fail, I would simply restore the full Hyper-v server backup I make every 2 weeks, then the latest nightly database/file backup. Everything could be up and running in 45mins.

 

Virtualization makes disaster recovery so much easier.

 

And yeh don't do it all in one go. We have really benefited from the advances in technology by doing it over a number of years. Just as an example the original servers I bought for about 1k came with 16gb of ram. These days I can get 32GB RAM and an SSD drive to boot for that.

Edited June 22, 2012 by zag

[pantscat]

Anything you want to know? We've had 4 + 1 JBOD in the time i've been here (ALL still running, 1 about to be decommissioned for being 7 years old)

 

Do you use them as an iSCSI target for VMs?

What's the performance like? (I'm running a Netapp FAS-2020 here... and the performance is adequate but not amazing, so I have considered using a secondary SAN to speed up the more disk intensive VMs...)

[mrbios]

Do you use them as an iSCSI target for VMs?

What's the performance like? (I'm running a Netapp FAS-2020 here... and the performance is adequate but not amazing, so I have considered using a secondary SAN to speed up the more disk intensive VMs...)

 

Yep I'm using two Eonstor DS S12E-G2140's as my VM iSCSI targets, each one has 3 RAID6 arrays in a tiered storage setup, one of those Eonstor DS's has a JBOD off it with 2 more arrays for file storage, then there's an older A16E with two more arrays synchronised via DFSR with the JBOD file storage.

 

All of them are reliable, did have a stick of ECC cache memory die in the A16E a while back, but that's an incredibly rare occourance for a stick of ECC memory to completely fail though, so i think we were just unlucky there. As for performance, they're pretty good, for a short period of time we had the entire VM storage on ONE 4 disk RAID6 array on the A16E, only time they visibly struggled was during backups.

 

What sort of VMs are you looking to improve performance on, SQL perchance? I find the SANs themselves cope with everything i throw at them, the only limiting factor is disk IOPs which could be the issue in your case?

[dhicks]

How would people recommend I proceed?

 

SANs are, in general, more suited to larger setups (probably half a dozen or more processing nodes), just because it takes some economics of scale to make them worthwhile - decent speed / latency connections between VMs and their virtual harddrives (i.e. preferablly fibre channel) and failover (i.e. preferably mirrored SAN hardware). For the average school (or, indeed, small business) setup I'd aim to use local storage on each processing node. You can have instant failover by real-time mirroring volumes between physical machines - set the machines up in pairs, preferably in different physical locations for extra security, and have them mirror each other's virtual machine volumes. In the past I've used Xen and DRBD for this, and this is my current home server setup, and it can allow for good instant-recovery abilities.

 

However, in my experience, most schools and smaller businesses don't actually need proper, millisecond-response failover. Sudden, unexpected, catastophic hardware failure should, hopefully, be very rare, and a few minutes of downtime while you boot up VMs on another machine are generally perfectly acceptable. This gets rid of the need for real-time mirroring of virtual machine images in exchange for taking regular backup snapshots instead and having those snapshots exported to a different storage volume. This is also rather easier to set up as a proper backup solution, with previous versions of virtual machine images available in case of disaster or configuration problems.

 

I'd use that Dell R510 as your main VM host, hosting your Domain Controller, print server, MIS server and general applications server(s) on local storage. Give the R310 an equal amount of local storage and have it regularly take backup snapshots from the R510 (have the R310 host another domain controller, though, don't take images of the first domain controller, DCs don't restore well from image backups as they use a lot of time-sensitive data). You can host as many extra VMs as you can fit on, although you might want to bear in mind that if one machine conks out one day you'll want to be able to run a minimally functional system on one machine.

 

Use one of your other servers as a dedicated file server for user files - we've found FreeNAS to work well, with a nice, GUI-based interface and support for ZFS. Bear in mind that ZFS' best feature, block-level deduplication, takes a lot of RAM - your file server might need 8GB or more of RAM and a decent processor to make best use of ZFS. Have another dedicated server, with at least 1.5 times the file server's storage size, as a backup for the file server, with regular snapshots or backups available so users can get to previous versions of files.

[pantscat]

What sort of VMs are you looking to improve performance on, SQL perchance? I find the SANs themselves cope with everything i throw at them, the only limiting factor is disk IOPs which could be the issue in your case?

 

Yep... disk IOPS is the issue with SQL... oddly enough everything else (including Exchange 2010, which surprised me) is perfectly happy.

Edited June 22, 2012 by pantscat
Poor grammar!

[mrbios]

Yep... disk IOPS is the issue with SQL... oddly enough everything else (including Exchange 2010, which surprised me) is perfectly happy.

 

Does your NetApp box support SSDs? even if it was just the two SSDs in RAID1 would do the job, you can leave the OS on regular storage and just move all the mdf and ldf files on to newly allocated SSD drive. Did the exact same thing only yesterday (granted i'm sort of trialling consumer based SSDs in my SAN, though a guy local to us has had a lot of success with the same attempt)

[pantscat]

@mrbios - that's a very good question. I have a feeling there is no official support for SSDs in the Netapp chassis yet. But I'll do some research!

[mrbios]

@mrbios - that's a very good question. I have a feeling there is no official support for SSDs in the Netapp chassis yet. But I'll do some research!

 

I should add though, that in an ideal world i'd say RAID6 minimum for an SSD array when using consumer rather than enterprise ones, as due to the way the drives wear when it finally comes to them failing they're all likely to fail around the same time (not mega close...but certainly closer than hdds) so having that bit of extra redundancy makes you feel that bit safer

[dhicks]

Yep... disk IOPS is the issue with SQL... oddly enough everything else (including Exchange 2010, which surprised me) is perfectly happy.

 

Exchange will cheerfully use pretty much all the RAM you can throw at it, and the more data it can cache in RAM the less disk access it has to do. Print servers are something that benifit from having a local disk available for spooling large print jobs - even if you have centralised storage you could always make a small local disk available for the print spooler.

[pantscat]

Oddly I find our print server doesn't really have much of an IOPS issue. Just SIMS...

[zag]

SANs are, in general, more suited to larger setups (probably half a dozen or more processing nodes), just because it takes some economics of scale to make them worthwhile - decent speed / latency connections between VMs and their virtual harddrives (i.e. preferablly fibre channel) and failover (i.e. preferably mirrored SAN hardware). For the average school (or, indeed, small business) setup I'd aim to use local storage on each processing node. You can have instant failover by real-time mirroring volumes between physical machines - set the machines up in pairs, preferably in different physical locations for extra security, and have them mirror each other's virtual machine volumes. In the past I've used Xen and DRBD for this, and this is my current home server setup, and it can allow for good instant-recovery abilities.

 

However, in my experience, most schools and smaller businesses don't actually need proper, millisecond-response failover. Sudden, unexpected, catastophic hardware failure should, hopefully, be very rare, and a few minutes of downtime while you boot up VMs on another machine are generally perfectly acceptable. This gets rid of the need for real-time mirroring of virtual machine images in exchange for taking regular backup snapshots instead and having those snapshots exported to a different storage volume. This is also rather easier to set up as a proper backup solution, with previous versions of virtual machine images available in case of disaster or configuration problems.

 

I'd use that Dell R510 as your main VM host, hosting your Domain Controller, print server, MIS server and general applications server(s) on local storage. Give the R310 an equal amount of local storage and have it regularly take backup snapshots from the R510 (have the R310 host another domain controller, though, don't take images of the first domain controller, DCs don't restore well from image backups as they use a lot of time-sensitive data). You can host as many extra VMs as you can fit on, although you might want to bear in mind that if one machine conks out one day you'll want to be able to run a minimally functional system on one machine.

 

Use one of your other servers as a dedicated file server for user files - we've found FreeNAS to work well, with a nice, GUI-based interface and support for ZFS. Bear in mind that ZFS' best feature, block-level deduplication, takes a lot of RAM - your file server might need 8GB or more of RAM and a decent processor to make best use of ZFS. Have another dedicated server, with at least 1.5 times the file server's storage size, as a backup for the file server, with regular snapshots or backups available so users can get to previous versions of files.

 

Anyone considering a new virtualization setup should read that and be pretty much ready to go.

 

Excellent advice.

[PotNoodleTech]

Oddly I find our print server doesn't really have much of an IOPS issue. Just SIMS...

 

Fit more RAM. SQL can cache the entire SIMS DB in RAM and just write the changes to disk. Speeds up SIMS no end. My SIMS VM barely reads and writes.

 

Butuz

[Willott]

I'll just point back to the LSI Switched SAS idea - it kinda makes a SAN out of your MSA60s and SAS HBAs in your virtual hosts.

 

6Gb/s SAS Switch

http://www.lsi.com/downloads/Public/SAS%20Switch/SAS%20Switch%20Common%20Files/LSI-Purdue_CS.pdf

 

Should work out cheaper than a full on SAN, and give you failover capability. I've yet to test this out, but have had various reports of it working with virtual environments.

 

Cheers

 

Will

[pantscat]

Fit more RAM. SQL can cache the entire SIMS DB in RAM and just write the changes to disk. Speeds up SIMS no end. My SIMS VM barely reads and writes.

 

Butuz

 

Awesome... I'll be ditching the 2003R2 VM for a 2008R2 box... RAM-a-plenty!