New Virtualized Servers - how would you do it?

[sonofsanta]

More or less as the title says, tbh.

 

Background:

Current servers are old and running on 2k3, I want to upgrade to 2k8 R2 ahead of going W7 in Summer '12. Server room is currently full of fat HP towers of varying ages, and it couold do with tidying up and improving.

 

I want to move to rack-mounted, and I want to go down the virtualized route for disaster recovery and reliability purposes. I'm planning on having 2 DC's, 1 Exchange 2010, 1 print server and 1 general apps server to run the AV console, WSUS, intranet site etc.

 

We run about 400 computers with about 1200 users.

 

I know enough about virtualization to talk about it and appreciate what's being said, but not enough to plan this out properly. I've got a few competing quotes from companies already, mostly based around 2 Xeon-powered servers with buckets of RAM and an iSCSI SAN, but there are variations.

 

For what I want to do budget shouldn't be an issue, based on existing quotes, but obviously I need to be saving money where reasonable in the current climate. There's no need for corners to be cut though.

 

 

So if you were virtualizing the 5 servers above - how would you do it?

[strawberry]

physical - dc, virtual host 1, virtual host 2, san, standalone virtual host.

 

dc for disaster recovery, 2 clustered virtual hosts with a san for instant failover, 1 standalone incase the san fails so you can restore the vms to a standalone machine. I'd them use the enterprise lciense to stick 4 vm's on each server, have a dc on the standalone, a dc on the cluster, and do some service duplication accross the vm's.

[sonofsanta]

physical - dc, virtual host 1, virtual host 2, san, standalone virtual host.

 

dc for disaster recovery, 2 clustered virtual hosts with a san for instant failover, 1 standalone incase the san fails so you can restore the vms to a standalone machine. I'd them use the enterprise lciense to stick 4 vm's on each server, have a dc on the standalone, a dc on the cluster, and do some service duplication accross the vm's.

 

Good point - no-one mentioned "what if the SAN failed"... would you run physical DC as primary or backup though? I figure a virtual DC must be easier to recover so would be better as primary, and if physical DC goes down and is only a backup, it could just pick everything up again from replication.

[nicholab]

I would run a software SAN in a VM in HA mode accross two physical boxes. So the SAN is mirrored across the two virtual hosts and the other VM run on the same 2 hosts.

[sonofsanta]

I would run a software SAN in a VM in HA mode accross two physical boxes. So the SAN is mirrored across the two virtual hosts and the other VM run on the same 2 hosts.

 

One supplier did quote on that possibility - two very well specced servers, each running a virtual FalconStor appliance in a HA pair, and each also running the VMs. Which was the cheaper option, and I got the impression that it wasn't as powerful/recommended (although of course, that may be because they wanted me to spend more... )

[glennda]

We have the following

 

3 x DL380 G5 (2 x quad core xeon) 2 of them with 64gb ram and 1 with 8gb (testing host). Connected via Fiber Channel to a MSA2000 with multiple controllers and automatic fail over between controllers if one dies (which it did and only one server crashed which we put down to trying to write to a system file at the time or something like that)

 

We currently run on the 2 main hosts (using Qemu/KVM on Ubuntu with oracle file system cluster) 1 x dc, 3 x file server, 1 x sims server, 1 x ecplise library server, 2 x terminal servers, wsus server, av server, xibo server, print server. The Processor never gets above 45/50 % usage and the ram hits around 30gb average.

 

We then run 1 x physical primary DC, a physical proxy (using squid) our firewall (isa) and then a couple more file servers.

[Duke]

Just throwing in my two pence...

 

If you have two virtual hosts then can a single host handle the whole load if one fails? Usually everyone recommends having a minimum of three hosts for this reason, but if you've only got 5-or-so virtual servers then one host could probably handle it.

 

As strawberry said, you must keep one DC physical, I'm not 100% sure on what the best practice is for making it primary or not.

 

Clustered pool of hosts with VMware ESXi (my preference), Oracle Sun S7000 SAN (again my preference) with a budget for either a second mirrored SAN or a physical backup solution - probably a server running DAS or with a tape drive and Veeam.

 

Chris

[TechMonkey]

2 points, which may be me being simple. (1) if you get a decent SAN then surely it has built in fail over (duplicated NIC, power, disks with RAID etc) so failover for the failover is commendable but maybe overkill (2) can someone point me to the docs/discussion/theory behind having a physical DC as it seems counter-productive to virtual up and then have a single point of failure with one physical DC.

[Duke]

2 points, which may be me being simple. (1) if you get a decent SAN then surely it has built in fail over (duplicated NIC, power, disks with RAID etc) so failover for the failover is commendable but maybe overkill (2) can someone point me to the docs/discussion/theory behind having a physical DC as it seems counter-productive to virtual up and then have a single point of failure with one physical DC.

 

The SAN will have redundant individual components, but the whole head itself can die or you could get a bad firmware/OS update that kills it. Clustered heads or a failover SAN with replication should solve this, and while it seems overkill the SAN is obviously a big component if it's hosting your VMs - one SAN failure could wipe out pretty much your entire server infrastructure.

 

I don't think anyone's suggesting you just have a single physical DC but that you must have at least one physical one (then other virtual ones too) so that when you first power up the virtual hosts and the virtual servers on them they have a server to talk to for DNS, DHCP (if applicable) and AD authentication.

 

Cheers,

Chris

[sonofsanta]

We have the following

 

3 x DL380 G5 (2 x quad core xeon) 2 of them with 64gb ram and 1 with 8gb (testing host). Connected via Fiber Channel to a MSA2000 with multiple controllers and automatic fail over between controllers if one dies (which it did and only one server crashed which we put down to trying to write to a system file at the time or something like that)

 

We currently run on the 2 main hosts (using Qemu/KVM on Ubuntu with oracle file system cluster) 1 x dc, 3 x file server, 1 x sims server, 1 x ecplise library server, 2 x terminal servers, wsus server, av server, xibo server, print server. The Processor never gets above 45/50 % usage and the ram hits around 30gb average.

 

We then run 1 x physical primary DC, a physical proxy (using squid) our firewall (isa) and then a couple more file servers.

 

Crikey - how much did that lot all cost you? (ballpark)

 

Do you find it better separating everything out so much? I'd have thought running a separate instance for each web app (WSUS, AV etc.) would introduce a lot of overhead from all the 2k8 installations running.

[AngryTechnician]

Counterpoint: you do not need to keep a physical DC.

 

Just don't put it in the cluster.

 

I have a DC on every physical host, but none of them are in a clustered disk on the SAN. Their virtual HDs reside on local RAID1 disks on the host. If the SAN fails, they will still work. If a host fails, one of the others will still work. Even if the Hyper-V environment goes pear-shaped on every host, I can simply reinstall Hyper-V from scratch, create a new VM and attach the locally-stored DC disks, and it will boot.

 

You do not need to keep a physical DC. Just keep at least one that isn't in the SAN, and there are no scenarios you can't recover from that you could if it was physical.

[sonofsanta]

Counterpoint: you do not need to keep a physical DC.

 

Just don't put it in the cluster.

 

I have a DC on every physical host, but none of them are in a clustered disk on the SAN. Their virtual HDs reside on local RAID1 disks on the host. If the SAN fails, they will still work. If a host fails, one of the others will still work. Even if the Hyper-V environment goes pear-shaped on every host, I can simply reinstall Hyper-V from scratch, create a new VM and attach the locally-stored DC disks, and it will boot.

 

You do not need to keep a physical DC. Just keep at least one that isn't in the SAN, and there are no scenarios you can't recover from that you could if it was physical.

 

I like that idea. A cunning plan, my lord...

 

Do you have any DC's running from the SAN then, or do you have one DC per physical host running from local storage and just rely on AD's natural resilience-through-replication?

[glennda]

Crikey - how much did that lot all cost you? (ballpark)

 

Do you find it better separating everything out so much? I'd have thought running a separate instance for each web app (WSUS, AV etc.) would introduce a lot of overhead from all the 2k8 installations running.

 

I don't know how much the SAN was but each set of 64gb ram is around £3,500 (although i upgraded each from 32 to 64) We only have seperate servers as I haven't had a chance to move the WSUS server to the AV server yet (av was converted from physical to virtual). The xibo server is linux and uses not alot but the sims server and each TS use 10gb ram each.

 

Also running on KVM is quicker then the likes of esxi/hyper-v as it runs inside the linux kernel plus its free so is the Oracle File System Cluster software.

[chazzy2501]

2 hosts, with 1 cpu each 32gb ram each host, 1 san with 3x 10k sas drives (for os) and as many 7k2 sata drives as you need for data. ISCSI SAN and 2x layer 2 gigabit switches. that with vmware essentials plus and some backup software per socket Veeme or vranger. I've 2 x 2 cpu hosts and it laughably uses less than 10% MAX. as most licensing with vm stuff is per socket just get what you need.

 

I have no physical DC and just keep both on the SAN it's got two of everything if I experience 2 simultaneous errors then tough, this is a school not a multimillion pound business. The SAN has a 5 year 4 hour call out. This is a significant improvement on my old physical hardware that was only backed up (no redundancy so turnaround was a day or two)

 

So far only 1 psu on the san has gone, was bricking it until new one arrived though.

[TechMonkey]

The SAN will have redundant individual components, but the whole head itself can die or you could get a bad firmware/OS update that kills it. Clustered heads or a failover SAN with replication should solve this, and while it seems overkill the SAN is obviously a big component if it's hosting your VMs - one SAN failure could wipe out pretty much your entire server infrastructure.

 

I don't think anyone's suggesting you just have a single physical DC but that you must have at least one physical one (then other virtual ones too) so that when you first power up the virtual hosts and the virtual servers on them they have a server to talk to for DNS, DHCP (if applicable) and AD authentication.

 

Cheers,

Chris

 

Aha, didn't realise that there was something major that could take the SAN down & didn't have a redundant partner. I was thinking I would like 2 SANs but the cost may be prohibitive.

 

Sorry, bad typing on my part. I realised that everything wouldn't be on DC, but wasn't understanding why you had to have 1 physical DC alongside your virtual servers. Would not just starting the first VM & letting that start up before initialising the others be enough? they would then see the first server with the services adn be happy? Or am I being a VM N00b?

[sonofsanta]

I realised that everything wouldn't be on DC, but wasn't understanding why you had to have 1 physical DC alongside your virtual servers. Would not just starting the first VM & letting that start up before initialising the others be enough? they would then see the first server with the services adn be happy? Or am I being a VM N00b?

 

I think the idea of the physical DC is that if the SAN goes down, you still have something running. At least, that's my understanding.

[Cools]

for a sans i would go for Dell Equallogic you can have half ssd and half sas

the SSD will give you the boot speed for the RD's in the morn... and SAS for storage..

the more its used the better it learns to place that start up profiles, so you get max speed from it at client boot up.. smart bit of kit..

[Duke]

I realised that everything wouldn't be on DC, but wasn't understanding why you had to have 1 physical DC alongside your virtual servers. Would not just starting the first VM & letting that start up before initialising the others be enough? they would then see the first server with the services adn be happy? Or am I being a VM N00b?

 

You're not being a n00b, and I'm just going by what I've been told - I'm not claiming to have perfect answers.

 

The virtual hosts will likely need DNS at some point (you could do everything through IP addresses), and it's very likely that vCenter (if you're using it) will want it too. As such, if everything is completely down you want to bring it up in a logical order - DC w/DNS/DHCP/AD, then the v-hosts, then v-Center (which can be virtual) to manage them, then the virtual machines themselves.

 

I think the panic would come when you start up a virtual host, it wants to talk to something (e.g. your SAN) that needs DNS, has no DNS server, all falls over and you can't get any of your VMs running!

 

People used to have the same issue with vCenter if they virtualised it - you need vCenter running to handle the licenses for the virtual hosts, you can't boot the hosts without vCenter running, and you can't run the vCenter virtual machine because the virtual hosts aren't up! You can get around this now because there's a grace period in which you can run the hosts before they need to check in with vCenter for licensing (at least that's what I was told).

 

Regarding hosting a copy of DCs on each virtual host locally - only works if you have local storage on the hosts (try using an SD card or PXE booting them) and it does mean you've get an extra storage pool to manage. Just a thought.

 

Chris

Edited April 5, 2011 by Duke

[glennda]

for a sans i would go for Dell Equallogic you can have half ssd and half sas

the SSD will give you the boot speed for the RD's in the morn... and SAS for storage..

the more its used the better it learns to place that start up profiles, so you get max speed from it at client boot up.. smart bit of kit..

 

I've looked at these at a seminar - are they not stupidly expensive (although this was 2+ years ago)

[dhicks]

can someone point me to the docs/discussion/theory behind having a physical DC

 

I think we might be talking about this:

 

Things to consider when you host Active Directory domain controllers in virtual hosting environments

 

If you snapshot, backup, and restore a domain controller then much confusion will ensue. I think the current advice is by all means, have two domain controllers, just make sure they are outside any snapshot/imaging backups system - back them up via some other method, or have replicated DCs.

 

For the original setting-up-a-new-set-of-servers question, I'd skip all this business about having a SAN and simply use network-mirrored storage (which is what your SAN salesman is trying to sell you, with a bunch of technical waffle and a large price tag thrown in). You can mirror sotrage volumes over the network for free with DRBD - I find it works well on Debian, and Debian handily includes the open source version of Xen as well, so you don't have to spend anything on SANs or virtualisation solutions, you can just spend all your cash on hardware.

[Avalon]

I implented vSphere 4 last summer:

 

- 3 virtual hosts with two quad-core Xeon's in each and 32Gb of RAM in each host

- HP MSA 2000 SAN with additional shelf (mixture of hard drives using 15k drives for production data and 7.2k drives for test data and backup)

- vRanger Pro

- vCenter

 

I didn't see the point in keeping one DC physical, could not see any feasible reason to do so.

 

I had a chat with a HP engineer who had just passed the MSA SAN course. He explained to me the redundancy that is engineered into the SAN's and how SAN redundancy is overkill and not necessary.

 

I was also concerned about only having two hosts, looking at performance now one host would not cope with all of the VM's and then you are down to a single point of failure running just one host (you never know!)

 

We all have different infrastructures and budgets to start with so there is definitely no 'one size fits all' implementation, but it is interesting to see what options exist ...

[Soulfish]

I've implemented a few virtual solutions now and one thing I will say is that until you get to the HP XP range/EMC Symmetrix (£££££££££) no SAN is 100% foolproof, and even with their hefty price tags those bits of kit can still fallover. Now there are various ways of mitigating this. If you have the cash a second SAN is one, alternatively good backup procedures and local storage could get you back online in a few hours.

 

I will say that I've seen MSAs taken out entirely with all data lost before and heard of EVAs have similar happen. OK it's not common, but it is possible . I've used both XenServer and VMWare. XenServer is great and I love the product, but VMWare has more features at the high end but you're paying a lot more for the privilege. My previous solution was based around XenServer, a Sun 7110, an EMC Celerra NS-480 and 4 x Sun Fire x4150's (32gb each, dual quad core) and another 4 x Dell R715s for VDI (64gb each, dual 12-core).

 

The solution that's going into my new school is 3 x HP 380's (64gb each, dual 6-core) and a HP EVA6400 based around VMWare. The VM hosts don't have any local storage. VMWare will be boot from SAN, so we'll also be running a separate physical DC.

 

It's all swings and roundabouts on the kit you buy, but if you do go for a SAN/NAS don't scrimp. As others have said it becomes the core of your network and a single point of failure. I would also for the reasons outlined by others be running a DC outside of the SAN based storage. If you do that as a separate physical DC or locally to one of the VM-Host boxes just make sure that you can bring it online first .

[gshaw]

The way we're doing it for our project...

 

- 3 x DL 380 G7 hosts

- 1 x SAN with dual controllers, multipath network connections etc

- 1 x physical backup server (DL 180 G6) running Veeam, will probably also be a physical DC

 

Veeam can in theory run server images directly from the backup files so that's a handy disaster recovery method. One thing I would say is make sure you go for high-quality branded kit i.e. HP, Dell, IBM etc. No point saving a few £££ now but getting hurt in the long run by poorer quality hardware \ support when you need it.

[nicholab]

What about using an MSA60 with dual controllers as shared DAS storage to two VM hosts?

[glennda]

What about using an MSA60 with dual controllers as shared DAS storage to two VM hosts?

 

Not sure if the MSA 60 would have the performance of the others - i have one but its used for backups only as it only supports sata/sas drives.