Whether to virtualise our main servers

[localzuk]

I run multiple RDS servers in VMs very successfully. Hyper-V is specifically optimised to handle it in fact, like Xen Server is optimised to handle XenApp.

[zag]

I run multiple RDS servers in VMs very successfully. Hyper-V is specifically optimised to handle it in fact, like Xen Server is optimised to handle XenApp.

 

Yeh we run a Remote Desktop Server here over Hyper-V and it works perfectly.

[fiza]

Won't go in to specifics as the CPU(s) was end of line and I badgered heavily on price But it's from @VeryPC. Novtech have managed similar for me in the past. Both good bespoke manufacturers who know how to treat their customers.

 

 

As far as I am aware that's still true. Sounds like a bonkers thing to try IMHO. Bear in mind I was talking about running Terminal Server in a VM which doesn't rely on Hyper-V.

 

My mistake - confusion reigns in my head at the moment. As the thread was about virtualising I assumed you meant Hyperv.

[Jollity]

Let's see if I have learned anything. Any thoughts on the plan I have come up with would be valued.

 

The omens suggest the finance gods are in fairly favourable mood, so I am pitching for a more Seawolf-solution, but I may fall back on a more Glenda-solution if things are looking dicey. I am going to try and sell getting two powerful main servers now as an alternative to replacing the current (rather underutilised) MIS database and web servers, and absorb their roles on the main servers. I plan to use the current web server as a backup server with Microsoft DPM and use one of current main servers as a dedicated physical DC (until it dies of old age at least).

 

The main servers would be Hyper-V hosts, have identical specs and each should be able to run one of each of the essential virtual machines on its own. The files would be replicated between the two virtual file servers using DFSR as they are currently. I have divided the server roles up as follows:

[table=width: 800]

[tr]

[td]Roles[/td]

[td]Number needed[/td]

[td]Essential?[/td]

[td]Memory (GB)[/td]

[td]System Disk Space (GB)[/td]

[/tr]

[tr]

[td]host operating system[/td]

[td]2[/td]

[td]TRUE[/td]

[td]4[/td]

[td]80[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]File[/td]

[td]2[/td]

[td]TRUE[/td]

[td]6[/td]

[td]80[/td]

[td]Separate RAID 5 array for files - passthrough to VM.[/td]

[/td]

[/tr]

[tr]

[td]DC, DHCP, DNS[/td]

[td]2[/td]

[td]TRUE[/td]

[td]3[/td]

[td]80[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]Print[/td]

[td]1[/td]

[td]TRUE[/td]

[td]4[/td]

[td]100[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]WSUS, internal

web apps[/td]

[td]1[/td]

[td]TRUE[/td]

[td]4[/td]

[td]80[/td]

[td]Additional dynamic VHD for update files on SATA disk[/td]

[/tr]

[tr]

[td]AV Management,

Log Monitoring, Unifi Controller[/td]

[td]1[/td]

[td]TRUE[/td]

[td]4[/td]

[td]100[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]AV Updates

only[/td]

[td]1[/td]

[td]FALSE[/td]

[td]2[/td]

[td]100[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]WDS[/td]

[td]2[/td]

[td]FALSE[/td]

[td]6[/td]

[td]160[/td]

[td]Dynamic VHD[/td]

[td][/td]

[/tr]

[tr]

[td]Certificate

Services[/td]

[td]1[/td]

[td]TRUE[/td]

[td]2[/td]

[td]40[/td]

[td]Server Core[/td]

[/td]

[/tr]

[tr]

[td]3Sys Web Server[/td]

[td]1[/td]

[td]TRUE[/td]

[td]6[/td]

[td]80[/td]

[td][/td]

[/td]

[/tr]

[tr]

[td]PASS Database

server[/td]

[td]1[/td]

[td]TRUE[/td]

[td]8[/td]

[td]100[/td]

[td][/td]

[/td]

[/tr]

[/table]

 

I have been playing with specifications for a PowerEdge T620 on the Dell website, but I will get an equivalent quote for an HP M350p. Two main servers something like:

2 x Intel® Xeon® E5-2620v2, 2.1GHz, 15M Cache, 7.2GT/s QPI, Turbo, HT, 6C, 80W, DDR3-1600MHz

14 x 4GB UDIMM, 1600 MHz, Low Volt, Dual Rank, x8 Data Width - total RAM: 56GB

PERC H710 Adapter RAID Controller, 512MB NV Cache

System Array (for host and guest images): RAID 1 - 2 x 1.2TB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-plug)

Data Array (passthrough to the file server): RAID 5 - 3 x 600GB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-Plug)

Extras Disk (80GB for WSUS files, rest for previous versions files): 500GB, SATA, 2.5in, 7.2K RPM Hard Drive (Hot-Plug)

Dual, Hot-plug, Redundant Power Supply (1+1), 750W, Titanium

Additional network card: Broadcom 5720 DP 1Gb Network Interface Card

5Yr ProSupport and Next Business Day On-Site Service

 

Any suggestions? Would you divide up the roles differently? Anywhere you would you suggest increasing the specification? Anywhere I would be able to cut back if I have to?

 

I have specified RAID 5 on the data disk as an additional layer of redundancy on top of replication to the other server by DFSR. Is this unnecessary? I have a colleague who is not keen on using RAID, so I will need to justify this.

[Gardinho]

Virtualisation is a total no brainer, modern servers are totally bored with the load of a standard windows server, just add more ram and make them run 10 or 12. I've got a big environment, but I would virtualise in any environment just for the benefits of role separation and the ability to add a new server for any new roles that come along.

 

Veeam for backup is an absoloute marvel. Mine runs through about 6TB of servers every night in about 6 hours.

 

 

What hardware do you backup to? 6 hours is good!

[glennda]

What hardware do you backup to? 6 hours is good!

 

With Veeam it depends on the rate of change so it could be running through 6TB but actually only backing up 30gb

[glennda]

Any reason you need two file servers with DFSR?

 

If you use Veeam as a backup tool instead of DPM you could instantly boot a failed file server if the requirement/need was there

[Gardinho]

With Veeam it depends on the rate of change so it could be running through 6TB but actually only backing up 30gb

 

This is true. We have FC connection to our backup server. A full backup (not differential) on a weekend normally takes us about 12 hours for 3TB. The bottleneck for us is the USB3 dual external drive. This works for us but quicker speeds would be nice at some stage.

[glennda]

This is true. We have FC connection to our backup server. A full backup (not differential) on a weekend normally takes us about 12 hours for 3TB. The bottleneck for us is the USB3 dual external drive. This works for us but quicker speeds would be nice at some stage.

 

Do you use Veeam? What transport methods do you use? USB3.0 should be able in theory write 5GBs/S to a disk so the bottleneck is probably the Sata drive in the device.

 

With My clients I setup Veeam with Direct SAN access (for VMware) and Local storage in the Backup server. And then Servers are also replicated offsite.

 

EDIT: Does Veeam with Hyper-V have any change block tracking?

[seawolf]

Let's see if I have learned anything. Any thoughts on the plan I have come up with would be

 

I have been playing with specifications for a PowerEdge T620 on the Dell website, but I will get an equivalent quote for an HP M350p. Two main servers something like:

2 x Intel® Xeon® E5-2620v2, 2.1GHz, 15M Cache, 7.2GT/s QPI, Turbo, HT, 6C, 80W, DDR3-1600MHz

14 x 4GB UDIMM, 1600 MHz, Low Volt, Dual Rank, x8 Data Width - total RAM: 56GB

PERC H710 Adapter RAID Controller, 512MB NV Cache

System Array (for host and guest images): RAID 1 - 2 x 1.2TB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-plug)

Data Array (passthrough to the file server): RAID 5 - 3 x 600GB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-Plug)

Extras Disk (80GB for WSUS files, rest for previous versions files): 500GB, SATA, 2.5in, 7.2K RPM Hard Drive (Hot-Plug)

Dual, Hot-plug, Redundant Power Supply (1+1), 750W, Titanium

Additional network card: Broadcom 5720 DP 1Gb Network Interface Card

5Yr ProSupport and Next Business Day On-Site Service

 

Any suggestions? Would you divide up the roles differently? Anywhere you would you suggest increasing the specification? Anywhere I would be able to cut back if I have to?

 

I have specified RAID 5 on the data disk as an additional layer of redundancy on top of replication to the other server by DFSR. Is this unnecessary? I have a colleague who is not keen on using RAID, so I will need to justify this.

 

I only have a few comments, but overall this looks pretty good.

 

1. I would never use RAID5 except for backup storage as I find is too unreliable for primary storage. You didn't specify whether you planned to use a hot spare or not. If you do plan to, don't. Besides my own experience (and others I know) with RAID5, technical reasons for not using it:

 

http://www.smbitjournal.com/2012/07/hot-spare-or-a-hot-mess/

http://miracleas.com/BAARF/RAID5_versus_RAID10.txt

 

So, what to use instead? RAID10 ideally, but if not in your budget at least use RAID6, it only costs you one more drive.

 

2. Of the other specs I might bump up if the money were there, it would be the CPUs and NICs.

 

NICs - how many total NICs will you have? You should aim for 6-8 per server ideally. If I had to choose between more CPU grunt and more 1Gb NICs in a virtualisation solution, I would spend the money on the extra NICs.

 

CPUs - I would go for E-2650s for the extra two cores (if I could afford them), which can become important in virtualisation. If budget doesn't allow, the 2620s will certainly suffice unless you experience a lot of growth.

 

As for the roles of the VM servers, your plan looks good. I do question the need for the AV management server and AV update servers to be on their own separate servers. We combine those here. Also, I think it would be OK to run the print server on one of your file servers with the number of clients you have, shouldn't be a problem at all.

 

3. Re-purposing older servers is a great plan. We do the same and it is partially this process over time that allows you to build up a rack or two of great kit. If you buy quality, well spec'd servers, they are still useful after 4-5 years in non-mission critical roles, especially of you have a spare one of similar spec to fire up if something goes down. Our backup server now runs on one of our 4.5 year old previous ESXi hosts, so it is a very well spec'd backup server, which means backups run fast!

 

4. What do you plan to use for backup storage? This is the one place where RAID5 using good quality SATA drives (WD SE for instance) can be used relatively safely and save you a few dollars. Another option is to do what we did, build your own server with at least 4 drives, 16GB+ RAM and install FreeNAS for rock solid ZFS backup storage. If you have an older server with several SATA drive bays then you have a great inexpensive backup storage solution as long as you use quality drives.

[Gardinho]

Yes we use Veeam v6.1. We have direct SAN access too, backing up 20 odd VMs from 3 ESXi hosts.

 

I'd love to add a 4th host to our VMWare infrastructure but the licence pricing for this is complicated. I haven't yet got my head around it.

 

To be honest I'm not too concerned about the write speeds as we are well within the allotted time for completing backups. I think our Veeam processes at about 200MB so I can't complain.

[Jollity]

Any reason you need two file servers with DFSR?

 

If you use Veeam as a backup tool instead of DPM you could instantly boot a failed file server if the requirement/need was there

It is a fair question, and I will think more about the alternative.

 

A significant reason is perhaps that this is what we do currently. It has the advantage of also splitting the load, but I do not really know that this is necessary. It means no technician action is required if one server goes down, though there is a slowdown while the connections to the server that's down time out. Does the Veeam set up you are talking about do automatic failover?

 

Though if we have the disks space available to store all the data files on the second server, it occurs to me it might as well act as a DFSR replica server. Though I suppose we could use slower, cheaper disks if it was only to be used in emergencies.

[zag]

It is a fair question, and I will think more about the alternative.

 

A significant reason is perhaps that this is what we do currently. It has the advantage of also splitting the load, but I do not really know that this is necessary. It means no technician action is required if one server goes down, though there is a slowdown while the connections to the server that's down time out. Does the Veeam set up you are talking about do automatic failover?

 

Though if we have the disks space available to store all the data files on the second server, it occurs to me it might as well act as a DFSR replica server. Though I suppose we could use slower, cheaper disks if it was only to be used in emergencies.

 

Its not automatic, but this is how we do it and it doesnt take long to get the server back up and running.

 

If you absolutely need 0% downtime then DFSR is the way forward, but in reality for a school a couple of hours downtime isn't the biggest problem in the world.

[Jollity]

1. I would never use RAID5 except for backup storage as I find is too unreliable for primary storage. You didn't specify whether you planned to use a hot spare or not.

I will read over the links you have sent this evening. I have read about the inadequacy of RAID 5 as redundancy due to the significant risk of another drive failing before the array can be rebuilt, but I was thinking it would be okay if used in conjunction with DFSR. If the array does fail completely we can rereplicate from the other server. But if there are other concerns as well over RAID5 that might change that.

 

NICs - how many total NICs will you have?

I was going to go for the built in one + one extra. I am assuming they have two ports each, but will check. We might add in another NIC later, but there did not seem to be any point in putting any more at the moment as our network backbone is still only 1GBit.

 

I do question the need for the AV management server and AV update servers to be on their own separate servers. We combine those here.

I was not clear. The AV Management server does updates as well. The other one is a fallback server that only does updates. Not totally sure that is necessary, or maybe I can combine it with another.

 

4. What do you plan to use for backup storage? This is the one place where RAID5 using good quality SATA drives (WD SE for instance) can be used relatively safely and save you a few dollars.

I was leaving it for another post. But yes I was thinking RAID 5 of SATA drives. I am concerned about the RAID controller: I have been reading posts about issues with the Dell PERC H200 particularly with SATA.

[Jollity]

Its not automatic, but this is how we do it and it doesnt take long to get the server back up and running.

 

If you absolutely need 0% downtime then DFSR is the way forward, but in reality for a school a couple of hours downtime isn't the biggest problem in the world.

True enough on the downtime. If the host for your file server would not boot, would you host the file server VM on the backup server?

[zag]

True enough on the downtime. If the host for your file server would not boot, would you host the file server VM on the backup server?

 

In our case yes, we have a "backup" empty Hyper-V host that we could restore the entire file server to in about 45mins.

 

We use SSD drives here so its a bit quicker

[tmcd35]

I have been playing with specifications for a PowerEdge T620 on the Dell website, but I will get an equivalent quote for an HP M350p. Two main servers something like:

2 x Intel® Xeon® E5-2620v2, 2.1GHz, 15M Cache, 7.2GT/s QPI, Turbo, HT, 6C, 80W, DDR3-1600MHz

14 x 4GB UDIMM, 1600 MHz, Low Volt, Dual Rank, x8 Data Width - total RAM: 56GB

PERC H710 Adapter RAID Controller, 512MB NV Cache

System Array (for host and guest images): RAID 1 - 2 x 1.2TB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-plug)

Data Array (passthrough to the file server): RAID 5 - 3 x 600GB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive (Hot-Plug)

Extras Disk (80GB for WSUS files, rest for previous versions files): 500GB, SATA, 2.5in, 7.2K RPM Hard Drive (Hot-Plug)

Dual, Hot-plug, Redundant Power Supply (1+1), 750W, Titanium

Additional network card: Broadcom 5720 DP 1Gb Network Interface Card

5Yr ProSupport and Next Business Day On-Site Service

 

Looks like pretty good specs. Doing a quick addup it seems to cover running all VM's on 1 machine if the worst happens, pretty much the point of virtualisation. Personally I'd spec up the RAM and CPU's a little to allow for growth other whys, looks fine.

 

Hard drives, I'd be tempted to have 1 large storage pool and allocate space for each virtual hard drive from there rather than segregating space. In deed, I know others will likely disagree, but I'd have the "Data Array" as an expanding VHDX image attached to the file server rather than passing through a physical array.

 

I'd question using a seperate server for storage and using iSCSI to connect the two hosts to that instead of DFSR, but that woudld cost more...

 

Any suggestions? Would you divide up the roles differently? Anywhere you would you suggest increasing the specification? Anywhere I would be able to cut back if I have to?

 

I like to split roles as much as possible, I'd put the DHCP server in it's own VM away from the DC/DNS servers.

 

I have specified RAID 5 on the data disk as an additional layer of redundancy on top of replication to the other server by DFSR. Is this unnecessary? I have a colleague who is not keen on using RAID, so I will need to justify this.

 

Personally I think RAID5 is fine how you're planning to use it, espeacially with the DFSR replication going on. For how you're planning to set it up I'd say RAID is a must, regardless of version. It's justification is simple - the server keeps running when a hard drive fails giving you time to fix the problem with minimal/no downtime.

 

If you look at Server2012 Storage Pools then hardware RAID is probably best avoided. Storage Pools and DFSR should give you a lot of flexibility.

[glennda]

It is a fair question, and I will think more about the alternative.

 

A significant reason is perhaps that this is what we do currently. It has the advantage of also splitting the load, but I do not really know that this is necessary. It means no technician action is required if one server goes down, though there is a slowdown while the connections to the server that's down time out. Does the Veeam set up you are talking about do automatic failover?

 

Though if we have the disks space available to store all the data files on the second server, it occurs to me it might as well act as a DFSR replica server. Though I suppose we could use slower, cheaper disks if it was only to be used in emergencies.

 

It is not going to equally split the load though - you can still potentially end up with all the load on a single server.

[Jollity]

It is not going to equally split the load though - you can still potentially end up with all the load on a single server.

DFS does seem to divide the clients up between the servers, but I have never tested how well the balances the load. It is true that there is nothing to stop it putting all the heavy users on one server by chance. I suppose I am making an assumption that on average things will balance out well enough.

 

Though if we have the disks space available to store all the data files on the second server, it occurs to me it might as well act as a DFSR replica server.

On reflection this is wrong, because I am already talking about three copies of the data in total, one on each main server and one on the backup server. If the backup server acts as a file server in emergencies then we would only need two.

[Jollity]

In our case yes, we have a "backup" empty Hyper-V host that we could restore the entire file server to in about 45mins.

Does the "backup" server do anything most of the time or is it just there waiting for emergencies?

[zag]

Does the "backup" server do anything most of the time or is it just there waiting for emergencies?

 

Its just there for disaster recovery purposes, so is online and ready to be used but does nothing in the mean time.

 

The beauty of virtualiztion and using VEEAM for backups is they can be restored so quickly, having a spare server makes this an even quicker process.

Edited February 24, 2014 by zag

[zag]

Another option would be to overspec your backup server and use that to run a VM in an emergency.

[tmcd35]

DFS does seem to divide the clients up between the servers, but I have never tested how well the balances the load. It is true that there is nothing to stop it putting all the heavy users on one server by chance. I suppose I am making an assumption that on average things will balance out well enough.

 

A couple of thoughts (I may be wrong, but can't find detail for 2012R2 on this)...

 

Hyper-V can't use DFS shares to store VM's for live migration - the file system will lock the VM files preventing the handover

 

Hyper-V doesn't do live automatic VM load balancing (or if it does, I can't find the detail). It can do automatic failover using Hyper-V Replica. And you can manually load balance using Live Migration.

[ChrisH]

A couple of thoughts (I may be wrong, but can't find detail for 2012R2 on this)...

 

Hyper-V doesn't do live automatic VM load balancing (or if it does, I can't find the detail). It can do automatic failover using Hyper-V Replica. And you can manually load balance using Live Migration.

 

You need SCVMM integrated with SCOM for this.

[Jollity]

I am doing some more research on the backup software and am giving Veeam a better look.

 

To use Veeam I would need to do the data storage as a virtual disk file rather than passthrough. Is there any other reason why Veeam would not work for the plan I laid out?

 

Would I need a Veeam license for the backup server if it was not actually being backed up? Is there any good reason for considering a version other than Veeam Backup Essentials for an organisation of our size?