What can/should I Virtualise?

[TheFopp]

Over the summer I'll be getting a couple of new servers to replace our two oldest, and I'm interested in a): the idea of using virtual servers on these to help cut down the number of servers I have, and b): the idea of being able to restore a server very quickly in the event of a server dying. So I'd like to hear from those of you who have already done this with your thoughts on our setup.

 

Currently we have 6 servers:

 

1. DC, DNS, DHCP, WSUS, IIS, McAfee Updates

2. DC, DNS, Fileserver (110GB data)

3. SQL (sims + a couple of smaller databases we've built) (20GB data)

4. Moodle (10GB data)

5. Exchange, Ranger Outpost, DC (I know it shouldn't be, but the guys who did the install did it this way and it hasn't caused any problems - touch wood)

6. just one IIS instance

 

The 2 oldest servers I will be replacing are 1 & 2. So here are my thoughts on virtualising

 

New servers

1. VM1 - DC, DNS, DHCP

VM2 - WSUS, McAfee Updates, IIS (from both IIS machines as WSUS needs it anyway)

 

2. VM1 - DC, DNS

VM2 - Fileserver

VM3 - Ranger Outpost

 

Old servers

3. VM1 - SQL

4. Moodle

5. Exchange

 

Does that look reasonable?

Does it make sense to virtualise the fileserver when it holds so much data - and will increase in datasize? (taking into account that we can't afford a SAN)

Is SQL happy on a VM?

What about Moodle, would that be happy as a VM on the same box as SQL, thereby reducing the number of servers required to 4?

 

VMWare or Virtual Server (free editions - can't afford ESX)

 

I also know you need plenty of memory on the host servers to run a number of VMs. I'm looking at HP DL380s for the servers. If I go with more than 4GB of RAM fro each server, do I need to run a different version of Server 2003 (currently running Server 2003 Standard) to access all the available memory?

 

Any advice or thoughts would be most welcome.

 

Cheers

Adrian

[Ric_]

VMWare or MS... I think that the general consensus is down to personal choice (although if you use VMWare you can upgrade to a higher product without having to convert images).

 

RAM... If you are sticking with Windows as your base OS, x64 edition of Server 2003 standard will do the job for up to 8GB RAM - after this you need Enterprise or if you stick with 32bit. Of course, you could run Linux as your base OS. There are also licensing considerations for virtualisation and I'm not the best to describe these.

 

Your spread of services doesn't look too bad, although I would separate out IIS from your update stuff since WSUS can be a bit funny.

[TheFopp]

Your spread of services doesn't look too bad, although I would separate out IIS from your update stuff since WSUS can be a bit funny.

 

I was under the impression that WSUS needs IIS to run - web-based admin centre.

[Dos_Box]

I have my Sophos and WSUS servers virtualised. Oh, and the KS3 testing platform too

[Geoff]

Also remember MS Virtual Server is really poor at running Linux VMs. So if you want to put anything in running under Linux, you should probably go for VMWare.

[ZeroHour]

I am curious about hypervisor vs. vmware free as options rather then virtual server as once hypervisor comes out I think that will probably be EOL for virtual server.

[tmcd35]

We are currently in the process of moving our entire server farm onto two ESX servers. I am currently building a third Windows based VMWare server as an extra backup server from one of the old, but still very good, servers that has now been virtualised.

 

We are also in the process of moving everything across to an iSCSI SAN environment - very existing times.

 

The best advise - you can never do enough research and planning! And even with all the research an planning and testing, unexpected things will go wrong. But I think in the long run it has been a worth while project here.

 

A very long way down the road we want to split our virtual servers up more, so that the MIS has it's own virtual server, the maths Kaelidos program as it own VM, we'll have individual dedicate VM's for DHCP, DNS, etc - or at least thats the plan.

 

Server 20003 SE will access 8Gb using PAE, basically memory paging, but it's not ideal. For more than 4Gb you really need a 64bit OS. For a VMWare box with >4Gb ram I'd go linux. In deed the only reason the machine I'm building now is running Windows is because I want it as a back up SAN server as well as a backup VMWare server and the SAN software runs on Windows only.

 

For licencing, remember each VM is in effect a seperate physical machine and needs all the licences any seperate physical machine would need - OS, CAL's, whatever. If you have a licence for Windows 2003 Server Enterprise R2 (must be R2), than you can run up to 4 VM's from the one licence.

[srochford]

When you dcpromo the box running Exchange down to a member server, Exchange will break - I've now forgotten the details (I think it's OWA that falls over) but it's messy (I've done it :-()

 

Either leave it as it is or put in a second Exchange server, move all the mailboxes to it, remove Exchange from the DC, dcpromo down, put Exchange back, move the mail back.

 

You definitely want a 64 bit OS to let you get at lots of RAM. The standard version will work but the Enterprise version of the server will allow you up to 4 VMs at no extra cost. If you're not going till the summer I'd be tempted to see if 2008 Hyper-V has been released by then and go with it.

[dhicks]

the idea of being able to restore a server very quickly in the event of a server dying.

 

I'm using Xen (also free, built-in to Ubuntu Server, although I'm having a few fiddly installation issues). I'm setting up DRDB to clone virtual file systems at the block level, so as data is written to one filesystem it is cloned across the network to another one. If one machine goes down another should be able to start right up with something like 800ms of downtime (seemingly... I'll believe it when I see it!).

 

> VM1 - DC, DNS, DHCP

 

I'm using Linux VMs, so there's no licensing involved, but I was rather thinking I'd have a separate VM for each of those. Doesn't one version of Windows Vista/Server 2008 let you run four virtual machine copies or something?

 

> Does it make sense to virtualise the fileserver when it holds so much

> data - and will increase in datasize? (taking into account that we can't

> afford a SAN)

 

The file system and the VM itself will be separate. You can even assign a VM to run off of a number of physical harddrives. Personally, I'm still of the opinion that a SAN is a waste of money for what any school is likely to do with it. SANs let you detach storage from processing power, so you can add more "costly" processing power as you like, at the cost of needing a whole bunch more equipment to communicate between the SAN and the processing kit. Most school's probably actually use very little processing power, mostly what they need is disk I/O performance, so you might as well have the processing power tightly-coupled to the storage (i.e. "a server with a nice big RAID array") and ditch the communications overhead. Processors are cheap these days, anyhow. Failover can be handled very nicely (when I get it to compile...) by block-level filesystem cloning (see DRDB, above).

 

> Is SQL happy on a VM?

> What about Moodle, would that be happy as a VM on the same box as

> SQL, thereby reducing the number of servers required to 4?

 

You can even get Moodle on a ready-made VM, so yes, it runs on a VM just fine (note that it's one of the things that'll need a bit more processing/RAM power - PHP works by smooshing together pages of text). Not sure about SQL - see if there are any ready-made SQL appliances around, that'll at least tell you if it's suitible to run on a VM or not.

 

> VMWare or Virtual Server (free editions - can't afford ESX)

 

Xen!

 

Stuff your servers with as many ethernet ports as you can - you can get four-on-one PCI Express cards. This means you can use an aggregated connection to the switch, so maybe a 4Gbps link, and maybe a direct link to another server if you want to use something like DRDB to clone filesystems.

 

--

David Hicks

[torledo]

We are currently in the process of moving our entire server farm onto two ESX servers. I am currently building a third Windows based VMWare server as an extra backup server from one of the old, but still very good, servers that has now been virtualised.

 

We are also in the process of moving everything across to an iSCSI SAN environment - very existing times.

 

The best advise - you can never do enough research and planning! And even with all the research an planning and testing, unexpected things will go wrong. But I think in the long run it has been a worth while project here.

 

A very long way down the road we want to split our virtual servers up more, so that the MIS has it's own virtual server, the maths Kaelidos program as it own VM, we'll have individual dedicate VM's for DHCP, DNS, etc - or at least thats the plan.

 

Server 20003 SE will access 8Gb using PAE, basically memory paging, but it's not ideal. For more than 4Gb you really need a 64bit OS. For a VMWare box with >4Gb ram I'd go linux. In deed the only reason the machine I'm building now is running Windows is because I want it as a back up SAN server as well as a backup VMWare server and the SAN software runs on Windows only.

 

For licencing, remember each VM is in effect a seperate physical machine and needs all the licences any seperate physical machine would need - OS, CAL's, whatever. If you have a licence for Windows 2003 Server Enterprise R2 (must be R2), than you can run up to 4 VM's from the one licence.

 

I think it's more practicaly to have application isolation environments for things such as dhcp, dns and you're maths thingy than dedicated seperate VM's with seperate OS instances. Using something like Parallels/Virtuozzo containers for instance.

 

As you've said it's one for down the road....so you've got plenty of change to change you're mind

[DMcCoy]

I'm using Linux VMs, so there's no licensing involved, but I was rather thinking I'd have a separate VM for each of those. Doesn't one version of Windows Vista/Server 2008 let you run four virtual machine copies or something?

 

From the windows perspective you need a server/exchange etc license for each host that it's used on unless it is only moved once every 180 days. This affects VMotion and the Xen equivalent. I bought datacenter licenses as it has unlimited virtualisation rights. The enterprise one has up to 4 additional copies of server 2003 but *only* on the host which holds the base license.

 

So if you plan to migrate as single instance of exchange between 10 blades more frequently than once every 180 days then you could potentially need 10 licenses for that single instance! Some may say this is due to the fact that MS don't have an equivilent application yet

[tmcd35]

I

> Does it make sense to virtualise the fileserver when it holds so much

> data - and will increase in datasize? (taking into account that we can't

> afford a SAN)

 

The file system and the VM itself will be separate. You can even assign a VM to run off of a number of physical harddrives. Personally, I'm still of the opinion that a SAN is a waste of money for what any school is likely to do with it. SANs let you detach storage from processing power, so you can add more "costly" processing power as you like, at the cost of needing a whole bunch more equipment to communicate between the SAN and the processing kit. Most school's probably actually use very little processing power, mostly what they need is disk I/O performance, so you might as well have the processing power tightly-coupled to the storage (i.e. "a server with a nice big RAID array") and ditch the communications overhead. Processors are cheap these days, anyhow. Failover can be handled very nicely (when I get it to compile...) by block-level filesystem cloning (see DRDB, above).

 

I have two ESX boxes attached to our SAN and a third VMWare (free vers.) going online soon. Depending on what we do with the MIS a fourth server may be added to the SAN. Also our File Server, thats not yet virtual, has been given some temporary SAN space to tide it over during the migration process.

 

I have, in the short time it's been in place, found that the SAN has given us greater scope/freedom with storage provisioning amongst servers. Extending drive space to individual servers and adding more storage to the SAN are now task that are quite simple to do. Where has adding more storage to an existing physical server can be quite a troublesome task.

 

SANs don't have to cost the earth - A Promise VTrack, SATA-II drives and SANMelody software!

 

I think it's more practicaly to have application isolation environments for things such as dhcp, dns and you're maths thingy than dedicated seperate VM's with seperate OS instances. Using something like Parallels/Virtuozzo containers for instance.

 

I've been looking into Application Vitualisation for general application deployment across the LAN. I can't find a system that's cheap/good enough for what I want

 

I still prefer separate VM's for separate tasks. Currently ony VM/Physical box may provide 6/7 different services - DHCP, DNS, AD, Printing, Apps, Files, etc.

 

If one of the hosted service forces you into restarting the entire server - all the service are lost.

 

Consider a Printer not responding and the Print Spooler freezing when you try to restart the service. Now imagine the MIS being on that machine. Yo restart the machine to get the printers back and loose the MIS in the process!

[dhicks]

Extending drive space to individual servers and adding more storage to the SAN are now task that are quite simple to do. Where has adding more storage to an existing physical server can be quite a troublesome task.

 

Admittedly I'm coming at this from a Linux point of view, where you run virtual machines on top the logical volume manager (which itself runs on top of mdadm software RAID - yes this would be more efficient with a decent hardware controller, but you only get so much for a £99 server). It's like having a "virtual SAN", but minus the expensive bits.

 

Dedicated VMs for DNS/DHCP/etc: there must be some ultra-small Linux distribution out there that take something like 64MB of RAM and can just run DHCP or whatever? No need for a whole mega-distribution with GUI and everything. DamnSmall could probably do it (and that has a GUI!).

 

--

David Hicks

[tmcd35]

Dedicated VMs for DNS/DHCP/etc: there must be some ultra-small Linux distribution out there that take something like 64MB of RAM and can just run DHCP or whatever? No need for a whole mega-distribution with GUI and everything. DamnSmall could probably do it (and that has a GUI!).

 

--

David Hicks

 

I like this mans thinking. Yep, Ideally, I'd replace most service currently being bundled together on the over stretched servers we have at the moment into multiple Linux mini-OS's dedicated to the task. No need to run Windows for DHCP, DNS, possibly printing and file sharing, and likely a few other services as well.

[TheFopp]

> Does it make sense to virtualise the fileserver when it holds so much

> data - and will increase in datasize? (taking into account that we can't

> afford a SAN)

 

The file system and the VM itself will be separate. You can even assign a VM to run off of a number of physical harddrives. Personally, I'm still of the opinion that a SAN is a waste of money for what any school is likely to do with it. SANs let you detach storage from processing power, so you can add more "costly" processing power as you like, at the cost of needing a whole bunch more equipment to communicate between the SAN and the processing kit. Most school's probably actually use very little processing power, mostly what they need is disk I/O performance, so you might as well have the processing power tightly-coupled to the storage (i.e. "a server with a nice big RAID array") and ditch the communications overhead. Processors are cheap these days, anyhow. Failover can be handled very nicely (when I get it to compile...) by block-level filesystem cloning (see DRDB, above).

--

David Hicks

 

Not sure I follow you there. If I create a Virtual Server instance and port my current fileserver (C: Windows D: Files) surely the virtual server would have to hold all the files for the fileserver so the image for the VS would be 100GB in size.

 

My question is, is it a good idea to have all of those files (100GB +) held within the Virtual Server file?

[dhicks]

Not sure I follow you there. If I create a Virtual Server instance and port my current fileserver (C: Windows D: Files) surely the virtual server would have to hold all the files for the fileserver so the image for the VS would be 100GB in size.

 

My question is, is it a good idea to have all of those files (100GB +) held within the Virtual Server file?

 

Again, I'm maybe used to a Linux/Unix point of view here, but I thought Windows -based virtual machine systems could also give VMs direct access to individual harddrives (or RAID volumes). You could assign a VM a file for storage of its operating system and a separate file (or a whole harddrive, or several harddrives) to store its data (the files it's going to be serving). That way the virtual machine itself can be detached from its data if wished. This is what I've done with all my Xen VMs (I've given them a third drive fro swap space, too). I run VirtualBox on my Windows PC at home, I'll have a look through that and see what backing stores it can use as "harddrives" for VMs.

 

--

David Hicks

[TheFopp]

Again, I'm maybe used to a Linux/Unix point of view here, but I thought Windows -based virtual machine systems could also give VMs direct access to individual harddrives (or RAID volumes). You could assign a VM a file for storage of its operating system and a separate file (or a whole harddrive, or several harddrives) to store its data (the files it's going to be serving). That way the virtual machine itself can be detached from its data if wished. This is what I've done with all my Xen VMs (I've given them a third drive fro swap space, too). I run VirtualBox on my Windows PC at home, I'll have a look through that and see what backing stores it can use as "harddrives" for VMs.

 

--

David Hicks

 

Yes, I get the idea of a seperate file for each Partition in effect. I would have one file for the OS Partition and another file for the Files partition but what I'm asking is there any practical limit as to how big you should let this get eg. is practical to have a Virtual file that would hold my 100GB+ of files - or is there a size limit I shouldn't go past. This file would obviously be stored on whatever Disk hardware i had in the host server, but is it a good idea to have a single file this size or potentially much bigger from a performance point of view.

[localzuk]

Yes, I get the idea of a seperate file for each Partition in effect. I would have one file for the OS Partition and another file for the Files partition but what I'm asking is there any practical limit as to how big you should let this get eg. is practical to have a Virtual file that would hold my 100GB+ of files - or is there a size limit I shouldn't go past. This file would obviously be stored on whatever Disk hardware i had in the host server, but is it a good idea to have a single file this size or potentially much bigger from a performance point of view.

 

I think what dhicks was getting at is that you don't need to use a file at all. You should be able to assign physical partitions directly to your virtual machines. For that sort of number of files I would never consider using a file, the performance will suck.

[TheFopp]

I think what dhicks was getting at is that you don't need to use a file at all. You should be able to assign physical partitions directly to your virtual machines. For that sort of number of files I would never consider using a file, the performance will suck.

 

Ah... so the actual 100GB of user-accessible files I want to share from the file server aren't actually virtualised.... as performance would suck, they are left on the hard disks as normal. So really the idea of being able to bring a Virtual server back online quickly onto another host server should the original host hardware die is not applicable in this case as I'd still have to do a full restore of all the userfiles using Backup Exec or whatever as normal, (presuming the Files and virtual server were all running on the same server) it would just be the virtual OS partition in effect that I could get back running quickly.

 

Am I starting to get it?!

 

If this is the case what about things like SQL then, What would be the best way to do Virtual Server of this?

[Ric_]

@TheFopp: What you would do is mirror the files on the second piece of hardware (or somewhere else).

[TheFopp]

@TheFopp: What you would do is mirror the files on the second piece of hardware (or somewhere else).

...

 

and then if disaster happens just bring the Virtual OS online on the second piece of hardware and point it the mirrored files.

 

Ok.. getting there.

 

So really my initial plans aren't viable as I also need a server mirroring the files incase of disaster. So its not really going to cut down on servers by the time I'm done. I can see why people go for the SAN option with this. 2 SANS different locations mirroring each other. Virtual servers pointing to files on the SAN.. easy to load up another instance on a different server pointing at the mirrored SAN if neccesary.. it all makes sense.

 

But what if you are my position with my current setup (and not enough funds for the SAN option).

 

What would those of you who are already virtualised recommend I do with my current setup in order to get less wasted power from the current servers, hopefully cut down on the number we are running and get good redundancy via the use of Virtualisation.

 

PS. I know I must be sounding a bit dumb when it comes to this, but I am just exploring all my options .... thanks for all the input so far.

[DMcCoy]

If you are going to start tying Virtual OS to physical drives then you are going to make life very difficult if you want to move things. You really want to use the virtual disk options, performance will be a bit lower, but if it's that critical whatever you are virtualising shouldn't be. A san is really only to become useful when it has a cluster file system too.

[localzuk]

If you are going to start tying Virtual OS to physical drives then you are going to make life very difficult if you want to move things. You really want to use the virtual disk options, performance will be a bit lower, but if it's that critical whatever you are virtualising shouldn't be. A san is really only to become useful when it has a cluster file system too.

 

If you are going to virtualise 100GB+ of files then performance will be more than a 'bit lower'. It will be downright awful.

[DMcCoy]

If you are going to virtualise 100GB+ of files then performance will be more than a 'bit lower'. It will be downright awful.

 

It depends on what you are using, I've got around 1TB of virtual disk files, no performance issues.

 

Assigning a raw disk for a VM, unless it's a large database really is shooting yourself in the foot with respect to one of the largest advantages of virtualisation. Plus you need one disk for very VM. If you don't have one per vm then you are going to have locking issues on the drive anyway, so multiple disk files make little difference.

[torledo]

If you are going to start tying Virtual OS to physical drives then you are going to make life very difficult if you want to move things. You really want to use the virtual disk options, performance will be a bit lower, but if it's that critical whatever you are virtualising shouldn't be. A san is really only to become useful when it has a cluster file system too.

 

Have you had a play with any of the available clustered file systems on you're san ? If so, what would you recommend.

 

There are quite a lot large SAN implementation without any sort of clustered FS implementation, when you say a SAN only becomes useful with a clustered FS are you specifically referring to clustered hardware setups such as an ESX farm connected to a SAN fabric ?