Hyper-V Live Migration and CPU Allocation / Oversubscription

[sonofsanta]

Still trying to wrap my head around all this Hyper-V stuff.

 

I've got two hosts, both dual-socketed with hexacore Xeons, so 12 cores/ 24 threads each, and 32GB of RAM. They'll operate in a failover cluster with live migration in case anything goes wrong.

 

I've got it planned so that the startup RAM for every VM totals 30Gb (leaving two for the host) so that in the case of a failover, one host can manage every VM. I'm then planning on using the dynamic memory to let VMs grow to a higher value when both hosts are up and there's 60Gb of RAM available to use overall.

 

However, I cannot, for the life of me, find out if I can do a similar thing with CPU cores. Does this mean I need to plan my core allocation to fit into the 22 available threads and waste half my processing power for a just-in-case scenario? Or will Hyper-V/VMM allow for CPU oversubscription, where in normal day-to-day activity there's no oversubscription taking place except when one host failsover? Potentially it's not as wasteful as it might seem as it would at least mean that VMs have real cores allocated rather than HyperThreaded Pretend Cores, except when failing over, but it still strikes me as under-utilising resources.

 

Has anyone set this up to know better than Google?

[DavidYoung]

You don't need to setup each vCPU so that it relates to a real core. We only run quad-cores with hyperthreading in our servers, and run way more than 8 vCPUs on each. You'll find that CPU is very rarely the bottleneck as much as disk and RAM. I would just give each VM a single core, unless it has high CPU needs and monitor it for a while.

 

Note: my experience is VMware, I thought it was similar, but turns out I'm wrong (posts below).

 

David.

Edited February 2, 2012 by DavidYoung

[Achandler]

I can tell you right now, if you have to many CPUs handed out or to much RAM for that case, then shoudl 1 host fail then one or more of your VMs won't boot up/ migrate over as it ill say not enough resources available.

 

best bet is allocate like you have 22 CPUs and 30Gb RAM available, then just split your VMs across both hosts to share the load.

[DavidYoung]

I can tell you right now, if you have to many CPUs handed out or to much RAM for that case, then shoudl 1 host fail then one or more of your VMs won't boot up/ migrate over as it ill say not enough resources available.

 

Does Hyper-V require one physical core per virtual CPU then? I must admit I don't have much experiance with Hyper-V as we use VMware here, but we did trial an early version a while ago and I remember the memory limits.

[Achandler]

Nope 1 thread is a core, so you have 24 virtual cores (12 cores with hyperthreading (2 theads)) in total. So I would allocate upto 22 cores to virtual machines. This leave 1 core/ 2 threads left for the host.

[DavidYoung]

Nope 1 thread is a core, so you have 24 virtual cores in total (including the host). So I would allocate upto 22.

 

But does that mean that you can't have more than 24 VMs, each with only one CPU, 12 VMs with dual-CPU, etc?

[Achandler]

Yes that is true, one host could support upto 22 VMs with 1 CPU or 11 with 2 CPUs etc. The combination would need to be less then 22 CPUs ofcourse.

 

The dynamic memory allocation for the VMs only came into play in the last Service pack, so maybe dynamic cores numbers will be introduced soon.

[DavidYoung]

Yes that is true, one host could support upto 22 VMs with 1 CPU or 11 with 2 CPUs etc. The combination would need to be less then 22 CPUs ofcourse.

 

The dynamic memory allocation for the VMs only came into play in the last Service pack, so maybe dynamic cores numbers will be introduced soon.

 

Wow, did not know that, glad we went for VMware!

[sonofsanta]

I don't need massive numbers of virtual servers and the licensing for Hyper-V was much cheaper on an educational licence than VMWare seemed to be, so the limitation isn't a big issue. It's just annoying that effectively half the processing capacity is going wasted.

 

I fear that CPU cores won't ever be possible to dynamically alter without a reboot of a server. Although I'd have instinctively thought that of RAM as well, so maybe.

 

Lesson for future: more low powered servers better than a few high-spec servers. If I have three servers dual socketed with quad cores I'd be able to allocate up to 28 threads...

[Achandler]

Three lower spec servers would be better in this instance as a number of server possible, but might not be the best choice long term.

 

Im sure dynamic cores will be possible as it would be an underlying software software change to "insert" the cores or maybe they will work using 1 big core where you could allocate a percentage maximum and minimum of it (allowing over 100%). I haven't use VMware but David seems to be hinting it is entirely possible to overallocate already in VMware so I assume if a node fails the allocations get sorted out to keep all VMs working. If this is the case, then the technolgoy must exist.

[Arthur]

HyperV R2 has an 8:1 logical to virtual processor limit. HyperV in Windows Server 8 on the other hand won't have this limit.

[Arthur]

Lesson for future: more low powered servers better than a few high-spec servers. If I have three servers dual socketed with quad cores I'd be able to allocate up to 28 threads...

... or fewer servers with more cores per processor.

 

A server with two Xeon E5-2650's would give you 32 threads.

[sonofsanta]

HyperV R2 has an 8:1 logical to virtual processor limit. HyperV in Windows Server 8 on the other hand won't have this limit.

 

Wait... my virtual servers can have no more than a single quad core assigned to them? That's a bit pants, I was hoping to give Exchange 8 threads total :/

 

 

... or fewer servers with more cores per processor.

 

If I have two hosts in a cluster, I can only safely use up to 1/2 of available resources; with three, I could use 2/3; with four, 3/4 etc.

[Arthur]

The formula used to work out the maximum number of virtual processors your Hyper-V server supports is this...

 

(Number of processors) * (Number of cores) * (Number of threads per core) * 8

 

Source: http://blogs.technet.com/b/virtualization/archive/2011/04/25/hyper-v-vm-density-vp-lp-ratio-cores-and-threads.aspx

 

Therefore, if you have a dual quad core Xeon with hyper-threading, you should be able to use upto 128 virtual processors per host.

[gshaw]

This is a great advert for N+1 virtual host design... if your capacity suits 2 servers get 3 so in case of failure the remaining host doesn't get overwhelmed... does depend on budget I guess but always worth bearing in mind...

[bossman]

@sonofsanta:

 

We went down the Xenserver route with 3 hosts 32GB Ram in each and worked on the theory of utilising a third of the memory on each host so in the event of 2 hosts failing (Chances are slim) the remaining host could still deal with the load.

 

I am working on getting our hosts up to 128GB Ram in each.

[Achandler]

Arthur, although I haven't added a VM with to many processors recently (since SP1) I can say that when I tried to add a dual core VM to a host with 8 threads and already 6 virtual cores in use but loads of RAM free, I was met with the error that there was not enough resources available. I actually took a core from another VM on the host taking it to 5 of 8 used and then the VM could be made.

 

This was an older server but the problem existed. Recently I also had a host fail overnight (my newest and best host) and the next day one server was down, there was plenty of RAM free on other servers but a lack of virtual cores and this server refused to start, infact it actuall moved itself (when starting with the cluster manager) to a server that had spare cores (even with though sufficient RAM was available on the host it was on) once I moved some VMs about.

 

Maybe this is due to System Centre Virtual Machine Manager or maybe these limits are due to Failover Cluster Manager, and not actually the physical hardware limitation I don't know and I haven't looked into it as I have sufficient cores available anyway as long as I don't have VMs all over the place. I work with 3 older hosts and 3 new hosts, so it makes life a little bit trickier.

 

I also before that this core problem is one of the major benefits of the AMD processors with physical 12 cores (no hyperthreading), as you get more cores for your money and as such they make a better host. All my servers are intel based though, so no experience with them.

[jamesfed]

16 core AMD Opterons....dual socket - 32 REAL cores how easy can it be - with 8:1 that is 256 VMs.

 

To be honest we over subscribe our Hyper-V CPUs and as others have said CPU is rarely the bottleneck in virtualisation, most often its in RAM (where dynamic allocation and just having loads of the stuff comes in use) and in the disk IO (where SSDs come in use).

[sonofsanta]

The formula used to work out the maximum number of virtual processors your Hyper-V server supports is this...

 

(Number of processors) * (Number of cores) * (Number of threads per core) * 8

 

Source: Hyper-V VM Density, VP:LP Ratio, Cores and Threads... - Windows Virtualization Team Blog - Site Home - TechNet Blogs

 

Therefore, if you have a dual quad core Xeon with hyper-threading, you should be able to use upto 128 virtual processors per host.

 

I don't think I'll get anywhere near that limit - I think there'll only be a half dozen VMs running to be honest - I'm just annoyed/surprised that the VM guest OS can only have 4 threads given over to it, seems a strange thing to limit.

[sonofsanta]

To be honest we over subscribe our Hyper-V CPUs and as others have said CPU is rarely the bottleneck in virtualisation, most often its in RAM (where dynamic allocation and just having loads of the stuff comes in use) and in the disk IO (where SSDs come in use).

 

How do you allocate them then? As said before, I've got 24 threads total on each of two hosts, so 22 available to guests on each - if you allocate (say) 30 cores, what happens when one server falls over or is taken down for maintenance? What @Achandler is saying jives with what I've found on TechNet and around the web so far, and there's not much point in having a failover cluster if not all your VMs failover, I fear :/

Edited February 3, 2012 by sonofsanta
gramammar!

[Achandler]

One of the things I woudl lvoe to see, if that you can specify unneeded machines in event of a failure, to allow oversubscribing resources better. I would like to specify that if I lost say 2 hosts or 3 hosts then shutdown all servers except the exchange server, main DC and fileserver for example. As this is the core of what I need to keep people working, they can not print, and have printers mapped locally if that was really necessary while work is completed.

[garethEds]

The formula used to work out the maximum number of virtual processors your Hyper-V server supports is this...

 

(Number of processors) * (Number of cores) * (Number of threads per core) * 8

 

Source: Hyper-V VM Density, VP:LP Ratio, Cores and Threads... - Windows Virtualization Team Blog - Site Home - TechNet Blogs

 

Therefore, if you have a dual quad core Xeon with hyper-threading, you should be able to use upto 128 virtual processors per host.

 

So if we have E5645 chip with 'IPMI X2 6 Core E5645 2.4Ghz Xeon CPU's' then it would be:

 

2 x 6 x 12 x 8 = 1152 virtual processors? Seems a lot. That would also be *2 because we have two servers.

 

Or am I missing the point.

 

GJE

[Achandler]

As I have stated earlier, I don't believe Microsoft at all, one of my Virtual Machines Properties, it has 4 Virtual Processors assigned, and clearly states it is using 25% of the machines resources. This machine has 8 cores and 16 threads, so 25% would be right.