Virtualization Plan Sanity Check

[Irazmus]

We are currently planning our first real foray into server virtualization and I'd appreciate a sanity check for our plans from someone who has more virtualization experience than me.

 

This is going to be a long one, so please bear with me.

 

History:

Over the last few years our server numbers have grown to the extent that we are over provisioned and most servers are under utilized (but have insufficient RAM for running multiple VMs). In addition, our SIMS and file servers (both Dell PowerEdge 2800s) are about to come out of warranty with no option to extend.

 

Due to local and national politics we are very short of money this year and cannot afford virtualization servers and to replace the file server.

 

What we can afford is to buy a server to start virtualizing and to move half the drives from our file server into our SIMS server and run the two machines as mirrored openfilers until we can afford a replacement next year.

 

For the VM host we're looking at buying an HP Proliant DL385

1 x AMD Opteron 6176SE (2.3GHz, 12 core) (Second socket for later expansion)
16 GB (8 x 2 GB RDIMM) Performance DDR3 RDIMM (256GB max)
4 x 600GB 6GB/s 10Krpm SAS DP (Expandable to 8)
OR
4 x 500GB 3GB/s 7.2Krpm SATA (Expandable to 8)

with a second to be bought in year 2 or 3 for redundancy. Until we buy the second, our existing servers will be used as backup VM hosts, with each being added to the cluster when it's current role is virtualized.

 

VMs will be stored on the VM host and mirrored to their backup machine with DRBD [ DRBD:What is DRBD ] to allow for failover without a SAN.

 

Virtualization will be provided by KVM on Debian (or possibly Gentoo), with Ganeti [ ganeti - Cluster-based virtualization management software - Google Project Hosting ] for VM management.

 

The VM's I intend to have running from this host are:

  HDD	Use			OS	RAM
-------------------------------------------------
20GB	Host			Debian	1GB

50GB	SIMS			W2K3	2GB
20GB	DC			W2K3	1GB
20GB	IIS			W2K3	1GB
40GB	SuccessMaker		W2K3	1GB
100GB	WDS			W2K3	1GB
15GB	Print			Debian	1GB
100GB	Webserver		Debian	2GB
20GB	Proxy			*nix	2GB
15GB	Intranet		Debian	512MB
15GB	Syslog			Debian	512MB
240GB	Zimbra (e-Mail)		Debian	2GB

655GB	Total				15GB

with the possibility for separate MSSQL and MySQL hosts, and a puppet server further down the line.

 

----------------------------------------------------------------------------------------

 

Can anyone see any issues with this plan or am I actually managing to have a lucid moment?

Edited April 28, 2011 by Irazmus
Added VM OS details

[jamesb]

Looks reasonable enough to me. Cut a bit fine maybe, but reasonable.

 

Can your backup hypervisors support everything if the main hypervisor goes down? If not, have you given thought to which server roles you could afford to lose?

 

One other thought - do you have any other DCs or were you planning to cut down to just the one? Whether or not it's virtual, I'd recommend having a second one somewhere - it doesn't need to be a high spec, just needs to exist.

[sted]

can i suggest (assuming using hyperv on 08r2 as a host os) you give it 2gb ram of its own at least

does wds really need its own server seems a bit overkill for what is essentially a tftp server as it really has no impact on the server itself apart from the disk space it requires. Why not merge it wth the print server?

[FN-GM]

Can you merge SuccessMaker, Intranrt & syslog. This will free up some ram

[jamesb]

can i suggest (assuming using hyperv on 08r2 as a host os)

Main Page - KVM - KVM as host.

 

you give it 2gb ram of its own at least

 

Wouldn't hurt, but if you're having to be tight on resources probably not necessary. With Hyper-V yep.

 

does wds really need its own server seems a bit overkill for what is essentially a tftp server as it really has no impact on the server itself apart from the disk space it requires. Why not merge it wth the print server?

 

No need to combine servers - if you split the roles onto individual servers you can monitor their performance, and how much they actually use in terms of resources, and then lower what you're providing them with (or raise it) as time goes on in order to fine-tune the setup.

[glennda]

We currently use KVM on Ubuntu and its pretty good - i currently run 3 x DL380 G5 Dual Quad Xeon with 64GB ram in the main two and 12 in the backup - all connected via Fiber Channel to an MSA 2000. We use an Oracle Filesystem cluster for the San and these hosts run pretty much the entire network - the only problem i have is with file servers and thats because my San is not big enough to house the data (will be after summer when get extra enclosure)

 

I currently run about 20+ servers of varying roles on these hosts (mainly the two with 64gb ram in) cpu doesnt max about 40%

[sted]

No need to combine servers - if you split the roles onto individual servers you can monitor their performance, and how much they actually use in terms of resources, and then lower what you're providing them with (or raise it) as time goes on in order to fine-tune the setup.

 

granted but its still at last 20gb iirc used for a server08 install thats hard drive space for nothing and its still takes cpu cycles even if not many that could probably be better used elsewhere

[jamesb]

granted but its still at last 20gb iirc used for a server08 install thats hard drive space for nothing and its still takes cpu cycles even if not many that could probably be better used elsewhere

 

I may be too used to vSphere, but why would you have separate, full server images for each of your servers rather than having a single 'core' image with the base OS, then several separate linked clones with only the extra software installed on each?

[glennda]

can i suggest (assuming using hyperv on 08r2 as a host os) you give it 2gb ram of its own at least

does wds really need its own server seems a bit overkill for what is essentially a tftp server as it really has no impact on the server itself apart from the disk space it requires. Why not merge it wth the print server?

 

KVM out performs Server 2k8 and HyperV many times over IMHO

[sted]

I may be too used to vSphere, but why would you have separate, full server images for each of your servers rather than having a single 'core' image with the base OS, then several separate linked clones with only the extra software installed on each?

 

because as far as im aware thats how hyperv works the machines are 100% independant so you cant share core files

 

KVM out performs Server 2k8 and HyperV many times over IMHO

 

I have never used it so cant omment ive only really used hyperv, hyperv core(or whatever its called) and something else the name of which escapes me. My main reasn for liking hyperv is the fact you get an actual usable console on the host pc you dont need to rely on a 2nd machine to manage it

Edited April 28, 2011 by sted

[plexer]

Are you planning on running virtualisation on you 2800's as the backup?

 

does kvm support those as I know hyper-v doesn't.

 

Ben

[Mr.Ben]

With the RAM requirements why not use the Hyper-V dynamic memory feature? (Introduced with 2008R2 SP1)

 

You could apply this to all non essential VM servers, allowing them to share a pool of Memory as and when they need it.

 

I would also combine some of those roles into single VM's.

[glennda]

Are you planning on running virtualisation on you 2800's as the backup?

 

does kvm support those as I know hyper-v doesn't.

 

Ben

 

if the cpu supports hardware virtualisation and also is x64 it should work AFAIK

[sted]

if the cpu supports hardware virtualisation and also is x64 it should work AFAIK

 

often needs a few option turned on in bios i know for instance hp g5 ml350s support it but you need to alter bios settings turn on virtualization and somthing else memory reaalted iirc (its been a while once i found out i enabled it on all my servers so if i ever wanted hyper v it was there)

[dhicks]

Over the last few years our server numbers have grown to the extent that we are over provisioned and most servers are under utilized (but have insufficient RAM for running multiple VMs).

 

For your older servers, double-check whether their processors have hardware support for virtualisation - you won't be able to use KVM on them if not. Xen will run paravirtualised (i.e. Linux-based) VMs on them, though, and you should be able to use Xen alongside KVM if you need to.

 

run the two machines as mirrored openfilers

 

Mirorred as in real-time (DRBD-style) mirrored? That's not a backup - delete a file from one and it deletes the same file from the other. I'd have your backup file server take a backup copy of your main file server once a day (or more often if you feel that's neccesary), storing several day's backups, and have the backup file server ready to take over as the file server if the first machine goes down for some reason. If you have a server outage (which should, hopefully, be very rare) people might loose a day's work, but I think having a versioned backup is more useful than a straight mirror.

 

VMs will be stored on the VM host and mirrored to their backup machine with DRBD

 

Good plan. Don't mirror your DC like this, though, use a secondary DC to mirror the first (you can still run them as virtual machines just fine, so you can move them around physical servers). Windows gets very confused if a previous backup of a domain controller is restarted, it's best to let active directory take care of mirroring itself.

 

Virtualization will be provided by KVM on Debian

 

I've had best results with Debian - my home servers are running Debian, it all seems very reliable.

Edited April 28, 2011 by dhicks

[jamesb]

because as far as im aware thats how hyperv works the machines are 100% independant so you cant share core files

 

Differencing disks should allow for multiple instances based off the same parent disk in Hyper-V. The instances themselves will be 100% independent - the parent just gives them a common starting system installation rather than having to install twelve copies of the same OS and consume twelve times the disk space.

 

I have never used it so cant omment ive only really used hyperv, hyperv core(or whatever its called) and something else the name of which escapes me. My main reasn for liking hyperv is the fact you get an actual usable console on the host pc you dont need to rely on a 2nd machine to manage it

 

But it does mean you're using all the system resources of a full OS with GUI just in order to have that console - probably why Hyper-V requires such a scandalous amount of memory for itself.

 

With the RAM requirements why not use the Hyper-V dynamic memory feature? (Introduced with 2008R2 SP1)

 

Sadly not in KVM yet (at least not for Windows guests - Linux guests support balloon drivers and they're working on a Windows one).

[Irazmus]

Firstly, thank you to everyone for all the advice, I'll go through again later and thank individual posts.

 

@jamesb

Is that a bit fine just on the RAM or the CPU as well?

I can probably persuade our Business Manager to cover an upgrade to 24GB, but I'm working on the assumption of 16GB for the time being.

 

Since we're currently running those services (except Syslog) as separate hosts we should be able to manage. But we can lose IIS, WDS and the Intranet without much issue. And if absolutely necessary I can always press-gang my desktop to assist.

 

We currently have 2 DCs, 1 of which will be staying physical as I'm too paranoid to virtualize them both.

 

@sted

The host OS will either be Debian or Gentoo running KVM as hypervisor. I'm more comfortable with Debain, but Gentoo has support for newer versions of Ganeti.

 

The print server is a Debian, CUPS, Pykota box, so merging with WDS isn't possible.

I guess I should have mentioned which OS each VM is running, I may go back and update that.

 

@FN-GM

Merging IIS into the SM box is an idea I'd not considered, although I'd like to try to segregate services if at all possible, since that's one of the things that attracts me to virtualization. I'd planned syslog to run on Debian, but I'll look into Windows options as a possible alternative if RAM becomes an issue.

 

@glennda

That sounds like a nice setup. I looked into SANs but they're simply out of our price range, although it's definately something to aspire to and your CPU numbers are reassuring.

 

@sted

It's probably rather fortunate we're still running 2K3 then. I'd like to at least upgrade our DCs to 2K8 but the cost of new CALs would prevent us buying the VM server. Our budget really is that tight this year.

 

@plexer

No, the 2800s will be mirrored file servers running OpenFiler. Not ideal since Dell will be classifying them End of Life later in the year, but we should be able to keep one going by robbing parts from the other if needed.

 

@Mr.Ben

If and when KVM supports it I may well do that. But even if KVM did support it, for planning I'd still assume them to each have dedicated RAM, at least that way I shouldn't under spec.

 

@dhicks

I think I already checked that, but I'll double check to be safe.

 

Yes mirrored as in DRDB style, just in case the server itself fails. We have a separate backup for disaster recovery (External HDDs stored in a firesafe in a separate building, with 3 TeraStations for server/VM backups).

 

My home server and half the servers here run Debian, the only reason I'm considering Gentoo is that Ganeti seems to be better supported there.

[plexer]

@Irazmus ok thanks good to know as I still have 2800 as my second DC here and I was going to run it as an additional 2008 R2 Dc but if it's going end of life and no more warranty available then I wont

 

Ben

[jamesb]

@jamesb

Is that a bit fine just on the RAM or the CPU as well?

 

A bit fine on both - I'd recommend making sure you've got more cores to play with before virtualising anything else onto it - 1 core/VM should be fine though. The RAM upgrade wouldn't hurt, but again so long as you're planning to upgrade before expanding the virtualisation project then it can be done later.

 

Since a large part of virtualisation is intended to maximise resource usage, cutting things a bit fine is the best way to do it.

 

Since we're currently running those services (except Syslog) as separate hosts we should be able to manage. But we can lose IIS, WDS and the Intranet without much issue. And if absolutely necessary I can always press-gang my desktop to assist.

 

We currently have 2 DCs, 1 of which will be staying physical as I'm too paranoid to virtualize them both.

 

Good call. As @dhicks said you don't really want to have any DC mirrored - much better either to have a separate virtual one on the other host, or a separate physical one, or both.

 

@Mr.Ben

If and when KVM supports it I may well do that. But even if KVM did support it, for planning I'd still assume them to each have dedicated RAM, at least that way I shouldn't under spec.

 

With well-managed dynamic memory and ballooning drivers, you can actually provide about 50% more memory than actually exists (assuming that the guests never try to actually use it all at once, then things get messy). KVM does actually support balloon drivers for your Linux guests apparently, so it may be worth looking into.

 

Out of curiousity - is there a specific reason you're going for KVM or is it purely an expense thing? If it's purely an expense, did you have a look at VMWare's ESXi Hypervisor (free)? I'm mainly asking since ESXi does provide balloon drivers for all of the OSs I've ever had a chance to try it with.

[jamesfed]

I'd strongly suggest you never cut corners on a Virtulisation project - there are just to many things which could go wrong when it comes to using existing (and potentialy aging) hard disks and expecting them to work harder than before (with your SIMS/File Server).

 

I know the budget is tight and we just had the same problem here. Ultimately we cut corners on a physical to virtual conversion by way of not spending enough on the SAN (which was a repurposed Dell PowerEdge 2900) and are now feeling the pain of corrupted disks and non working VMs.

 

Not that I want to be a doom sayer but you might be better off to wait a year for the money to get a propper SAN and a second virtual host before you start your virtual journey.

 

On a ligter note that server sounds pretty much like our HP DL165s but with a 8 core processor, we also have a DL385 G7 which has another 8 core (this one runs our virtual desktops) and both models run a treat.

 

Also have a look at Hyper-V Server 2008 R2 from Microsoft - its free and you get the things like Dynamic Memory which would help on your RAM requirements.

[jamesb]

Also have a look at Hyper-V Server 2008 R2 from Microsoft - its free and you get the things like Dynamic Memory which would help on your RAM requirements.

 

Microsoft Hyper-V, VMWare vSphere Hypervisor (ESXi) and KVM are all free, and all perfectly fine for simple server virtualisation (except for KVM's lack of balloon drivers for Windows).

[NS_tech]

Sounds like a good well thought out plan to me, the 24GB ram upgrade would be good if you can get it from the money side of things.

Nice server choice

 

Hope all goes well, be sure to let us know when its up an running how its going

[Irazmus]

With well-managed dynamic memory and ballooning drivers, you can actually provide about 50% more memory than actually exists (assuming that the guests never try to actually use it all at once, then things get messy). KVM does actually support balloon drivers for your Linux guests apparently, so it may be worth looking into.

 

I'll definitely look into that, especially for those (nix) servers that could do with more RAM anyway.

 

It's partly an expense thing, but it's also the fact that it's GPL.

We looked into ESXi but decided against it for a number of reasons.

I couldn't see any way to get High Availability and auto failover, which I think I can probably manage with KVM/Ganeti/LHAP. It's not vital but I'd like to have the option for later.

I also don't think it's possible to have local VM storage using DRDB (or similar) to clone between VM hosts. Without the DRDB cloning I can't avoid using network storage for VM images while maintaining quick recovery.

I was initially planning to use Xen with Ganeti, but KVM has replaced it as it's kernel based. I may still use some Xen hosts if some of my older kit doesn't support HVM, but that's a decision I can finalize later.

 

@jamesfed

I don't mind a little doom saying, the more information/opinions I get, the better.

 

I take your warning on board, but SIMS is too important to run on unsupported hardware so we have to act now, either with a new SIMS server, or by starting the virtualization. The upside is that most of the load will be put on the new server, the 2800s will actually get less of a hammering since they'll be sharing the load (unless one decides to fail) and will only be serving user files. Plus the drives they'll be using are only ~3 years old since they aren't original to either server, they were added as an upgrade.

 

But if this all goes to pot because we're cutting corners, you have full permission to say 'I warned you' in a vaguely Scottish accent and then throw a rabbit at me

 

Not to nit-pick, but Hyper-V isn't free, it's included in the server licence cost, and we have no 2K8 licences, nor 2K8 CALs either (Which I assume I'd need to allow clients to connect to the VMs). To sort the above plan using Hyper-V, we'd need a 2K8 licence for the new host and each of the backup hosts. When adding the CALs on (250 computers @ £3.70) , we're looking at ~£1200 - £1500, which I'd rather put towards the hardware. That and I'm happier managing *nix machines.

 

@NS_tech

I'm confident I can get the upgrade, but prefer to err on the side of caution until it's confirmed.

The HP DL3xx range come up often in the virtualization threads here so it seemed the obvious choice, especially when you compare the prices to Dell.

 

I'll definitely report back when we get it running, but it'll probably be the summer before we really get going with it.

[jamesfed]

But if this all goes to pot because we're cutting corners, you have full permission to say 'I warned you' in a vaguely Scottish accent and then throw a rabbit at me

 

Rabbit at the ready (said in the best of faith)

 

Not to nit-pick, but Hyper-V isn't free, it's included in the server licence cost, and we have no 2K8 licences, nor 2K8 CALs either (Which I assume I'd need to allow clients to connect to the VMs). To sort the above plan using Hyper-V, we'd need a 2K8 licence for the new host and each of the backup hosts. When adding the CALs on (250 computers @ £3.70) , we're looking at ~£1200 - £1500, which I'd rather put towards the hardware. That and I'm happier managing *nix machines.

 

Hyper-V Server is free http://www.microsoft.com/hyper-v-server/en/us/default.aspx (its a seprate thing from Windows Server) basicly Microsoft realised VMWare and Citrix were making a killing from 'free' products and they jumped on the band wagon.

Just looking through your requirements again I don't think Debin works to well on Hyper-V so my bad on mentioning it.

[jamesb]

I'll definitely look into that, especially for those (nix) servers that could do with more RAM anyway.

 

It's important to remember that over-committing memory (ballooning) doesn't actually create more RAM from nowhere. Instead it'll grab memory back from machines which aren't using it by having a fake system driver request the memory - then it can be parcelled out to a machine which does need it at the time. Ballooning is good, paging to disk is a bad thing, and is what'll happen if too much over-committing goes on.

 

It's partly an expense thing, but it's also the fact that it's GPL.

 

Fair enough.

 

We looked into ESXi but decided against it for a number of reasons.

I couldn't see any way to get High Availability and auto failover, which I think I can probably manage with KVM/Ganeti/LHAP. It's not vital but I'd like to have the option for later.

 

It can be done (for free, obviously it can be done with the full vSphere stack but that gets pricey). You'd need a separate monitoring server and a bit of scripting knowledge, and it wouldn't be the instantaneous HA you get with the full stack but would take maybe a minute to fire up. I don't know about doing it with Ganeti though, which is probably easier.

 

@jamesfed

Not to nit-pick, but Hyper-V isn't free, it's included in the server licence cost, and we have no 2K8 licences, nor 2K8 CALs either (Which I assume I'd need to allow clients to connect to the VMs).

 

Actually, to be nit-picky, Microsoft Hyper-V Server 2008 R2 is free - Microsoft Hyper-V Server: How to Get It, it's just that most people assume you can only get Hyper-V with Windows attached.

 

To sort the above plan using Hyper-V, we'd need a 2K8 licence for the new host and each of the backup hosts. When adding the CALs on (250 computers @ £3.70) , we're looking at ~£1200 - £1500, which I'd rather put towards the hardware. That and I'm happier managing *nix machines.

 

No CALs needed. And ESXi is Linux-based. I'd say stick with KVM though, since it looks like you've researched it and there's very little difference between the hypervisors.