What can/should I Virtualise?

[DMcCoy]

A SAN becomes much more useful when you can share the virtual disks between hosts without having to move them, but for this you need a file system that can have multiple hosts accessing the LUN. While a SAN is still useful outside of virtualisation for its other features, it's this sharing that is important when virtualising.

 

I'm currently using ESX 3 with vmfs 3 file system on all my LUNs, all hosts share the same LUNs so I can run any VM on any host within 60 seconds. Care is still needed when formating the lun and virtual disks to make sure they are properly aligned, you can lose up to around 10% perfomance if you don't.

[dhicks]

and then if disaster happens just bring the Virtual OS online on the second piece of hardware and point it the mirrored files.

 

Exactly.

 

So really my initial plans aren't viable as I also need a server mirroring the files incase of disaster.

 

No! You still only need two! Split your VMs evenly between the two and have each one mirror the file systems of the other. If one machine conks out the other one can take the whole load for a while, just a bit slower.

 

Note that this kind of implies you have the better sort of server here - decent hardware RAID, decent number of fast harddrives, probably best to have a dedicated network link between the two servers. Don't worry, though, you're saving way more money by not buying a SAN!

 

I can see why people go for the SAN option with this. 2 SANS different locations mirroring each other. Virtual servers pointing to files on the SAN.. easy to load up another instance on a different server pointing at the mirrored SAN if neccesary.. it all makes sense.

 

Don't do it! Don't be seduced by the dark side! SANs are good if you're Amazon, Google, a render farm, the human genome project, etc - people for whom the amount of CPU (or RAM I/O, anyway) power they need outweighs the disk I/O speed they need, and who are doing this kind of thing on a large enough scale that adding something like fibre channel links doesn't make much difference, percentage wise, to the final amount they spend. A school doesn't need that kind of CPU power (and, if you do decide you need a batch-job render farm or whatever, remember you've got several labs of PCs sitting idle from 4pm until 9am...) - the main problem is disk I/O speed and network speed.

 

We're having the exact same debate that comes up repeatedly in computer science - offload processing to a dedicated, more loosely-coupled device, or keep things tightly-coupled and reduce the cost/communications overhead. The same thing has happened with mathematical operations (CPUs, then external floating point units, then integrated chips, now physics cards), graphics (graphics? terminals! ah - vectors! ooh - bitmaps! CGA/EGA/VGA - external cards! Onboard! External! Onboard for laptops! External cards again! Onboard again! TWO External cards!), sound (bleeper! DAC! Onboard! Soundblaster! Onboard! MIDI! Onboard MIDI! USB! Onboard! Surround sound! Onboard again!), etc, etc...

 

Note that this isn't a all-SANs-are-bad rant, it's just that I don't think they're suitible for any but maybe the very largest of school installations. Admittedly, this might change over a few years if things become more web-based (I figure web servers need more processing power than straight file servers), but I doubt it as I figure processors will just keep getting cheaper anyway.

 

But what if you are my position with my current setup (and not enough funds for the SAN option). What would those of you who are already virtualised recommend I do with my current setup in order to get less wasted power from the current servers, hopefully cut down on the number we are running and get good redundancy via the use of Virtualisation.

 

Mirrored servers, as above. I'd recommended spending more on them than we did (£99 each...), though.

 

--

David Hicks

[DMcCoy]

The problem with that is you may well increase the amount of hardware you have for something thats primarily for consolidation.

 

SAN is not about speed! It's about connectivity, features and consolidation.

[tmcd35]

Although I'm planning on having all our drives run within vmdk files on our SAN, I can see how using RAW disks might work. Unless I'm missing something, you can...

 

Have a C: drive as a vmdk that you can restore/rerun anywhere. That VM can have a RAW disk assigned to it, drive D: for user data. If that RAW disk is on a SAN then the VM can be restarted on any host that as access to that SAN and it will be able to see the D: drive.

 

So for disaster recovery. If the server hosting the VM dies, you can restore/run the VM on another server quickly and it will have access to the data. If the user data was lost, a new D: drive can be assigned/attached to the VM and the data should be restored from your standard weekly/daily data backups.

 

Just because a server is within a virtual environment is no excuse for not keeping standard tape/ext. hdd backups of data.

 

In this scenario, mirroring SANs, file servers, etc is just plain over kill in a school environment IMHO. But SANs themselves, they provide a very good and useful method of separating data storage from physical hardware. And as I have said above, they don't need to cost the ten's of thousands you'd perhaps expect.

 

SAN is not about speed! It's about connectivity, features and consolidation.

Exactly!

[tmcd35]

Okay, lacking a bit of Netiquette, two post on a row, but just spotted this...

 

But what if you are my position with my current setup (and not enough funds for the SAN option).

 

What would those of you who are already virtualised recommend I do with my current setup in order to get less wasted power from the current servers, hopefully cut down on the number we are running and get good redundancy via the use of Virtualisation.

 

PS. I know I must be sounding a bit dumb when it comes to this, but I am just exploring all my options .... thanks for all the input so far.

 

Just thought I'd say one, probably very important thing, along the lines of the planning/research advise I gave earlier.

 

If you decide to go down the virtualisation route - spend money now, don't do a half job, you'd regret it - we are!

virtualise

If you are going to proper I'd say choose between the free Xen and the expensive VMWare ESX and do the job right first time.

 

We bought 1 ESX server and a RAID array hooked up as DAS. We then bought a 2nd ESX server and found we couldn't connect it to the DAS, we had used both available connections. We then bought a SAN server to control the DAS over an iSCSI SAN so all required servers can use the storage space. The headaches this piece-mill approach has caused just isn't funny.

 

I'm currently moving VM stored on the RAID from LUN's presented as DAS on the original ESX server to LUNS on the same RAID box that are presented to the same ESX server but this time via the iSCSI SAN.

 

We should have bought both servers and the SAN day 1!

[torledo]

@dhicks - in terms of the SAN cost issue, that really is a no-brainer these days with iSCSI for schools. Gigabit switches, iscsi gigabit integrated network cards on servers and the cheaper cost of iscsi arrays compared to Fc or multiprotocol mean that it's very cost effective. I would also say the TCO is much lower than some sort of distributed system where you've got lots of bits of hardware that can balls things up.

 

As dhicks has said, SANs aren't just about speed - centralized storage management and the flexivility of a a centralized storage solution are the main advantages aswell as fault tolerance and scalability. And i think they're very much a technology appropriate for a school environment, with Microsofts SAN plug'n'play architecure sourcing and setting up an iscsi SAN has never been easier.

 

Also, if we're talking about virtualization NAS is also a viable alternalitive for storage and access of virtual machine files. Again it's the idea of the centralized storage pool model which tolerates hardware failure and eases the management burden.

[srochford]

My question is, is it a good idea to have all of those files (100GB +) held within the Virtual Server file?

 

 

Yes. For MS Virtual Server I don't think you really have a choice - there is 1 actual file per virtual hard drive (so if you virtualise a server which had 2 physical drives then you end up with a DriveC.vhd file and a DriveD.vhd file - names of your choice, obviously!)

[m25man]

Virtualisation of your Datacentre is probably the worse thing you can do in a school.

 

Once you sell the concept to the SMT they will immediately begin to assume;

 

It's Virtual so it must be easy to implement, takes absolutely no room up so the datacentre can be turned into a staff meeting room, is incredibly fast as there are no moving parts, and costs "Virtually" nothing.

Of course, ongoing licensing can be paid for with the "Virtual Money" from the DFeS!

[dhicks]

I would also say the TCO is much lower than some sort of distributed system where you've got lots of bits of hardware that can balls things up.

 

I'm thinking of the kind of system that would have less bits of hardware in it - you could have just the two servers with some wires running directly between them. No need for a SAN device, no need for dedicated switches. You just get a bunch of harddrives in a decent sized RAID array and shove them in a case with a couple of decent processors and a motherboard.

 

Note: the above system is slightly theoretical until I can actually get Xen and DRDB to install/compile/run/hell, anything at the same time...

 

What's the practical limit for RAID size? A given system is only going to be capable of so much data throughput, whether it's hauling data off harddrives and on to the network, or running VMs locally on them. What's the optimal number/performance ratio? Is, say, a 10-disk RAID 10 array going to be around the best performance you'll get? Or a 6-disk RAID 10? Or a 9-disk RAID 50 array (3 RAID 5 arrays striped)?

 

--

David Hicks

[tmcd35]

I'm thinking of the kind of system that would have less bits of hardware in it - you could have just the two servers with some wires running directly between them. No need for a SAN device, no need for dedicated switches. You just get a bunch of harddrives in a decent sized RAID array and shove them in a case with a couple of decent processors and a motherboard.

 

We started of with this set up. And our original plan was to stick with this set-up. We bought an HP DL380 server to run ESX and a Promise vTRAK MP310 RAID array. The array as 12x 320Mb SATA-II drives. 2x5 RAID-5 arrays stripped (RAID-50) and 2xhot spares. The vTRAK is plugged directly connected to th ESX server via 320mbps SCSI. We'd always planned to buy a second ESX server to be connected to the vTRAKs second SCSI controller this April.

 

Our main file server duly ran out of space and is still not ready to be virtualised. So as an emergency measure we put a new SCSI card in the file server and connected it to the VTRAKs second SCSI controller. Giving it a LUN and solving the space issue.

 

Now, we bought the second ESX server and found we have no way of connecting it to the vTRAK while our file server is using the second controller port. Also we realised that when the file server is virtualised we will have a high spec'd physical server (dual 3Ghz HT Xeon, 8Gb Ram) and nothing to use it for.

 

We decided the file server will make a good back if one of the two ESX servers went down. We are going to put the free VMWare server on this server and give it access to the vTRAKs RAID Array. We are also looking for a new MIS and that may be given a VM or it may have a physical server and allocated space on the vTRAK.

 

So we potentially have 4 servers all needing access to the RAID Array and only enough physical connections for 2 of them to actually be connected. The solution?

 

We got a beefed up desktop that was used as a server until that machine was virtualised. Shoved in a SCSI controller and some NIC's. Plugged it into the vTRAK in place of the file server. Used a bit of software called SANMelody, and Bob just might be your uncle (he's not mine?), All servers have access to the 2.5Tb RAID-50 4-drive redundancy Array through the SAN.

 

As I said above, we learnt from our mistakes. We should have bought both ESX servers and the SAN day one. Instead we tried to save money, use DAS instead of SAN, buy the server piece mill - and we have paid the price.

 

What ever you do, if you decide that a DAS RAID Array is right for you, or decide a SAN is right for you, if you are going to virtualise you need to research, research, research - plan, plan, plan - then research and plan some more. Then buy what you need day one - don't cut corners or try to put off buying anything that is really needed day one. And after all that expect there to be problems and downtime for issue you never expected or for things you though you'd planed for.

Edited April 16, 2008 by tmcd35

[TheFopp]

Thanks guys.

 

I've got plenty to think about from this thread about how I approach virtualisation (If at all!!!! ). Please keep any thoughts coming though as I'm not going to be attempting this until the Summer hols at the earliest!