Hyper-V Live Migration and CPU Allocation / Oversubscription

[sonofsanta]

Still trying to wrap my head around all this Hyper-V stuff.

I've got two hosts, both dual-socketed with hexacore Xeons, so 12 cores/ 24 threads each, and 32GB of RAM. They'll operate in a failover cluster with live migration in case anything goes wrong.

I've got it planned so that the startup RAM for every VM totals 30Gb (leaving two for the host) so that in the case of a failover, one host can manage every VM. I'm then planning on using the dynamic memory to let VMs grow to a higher value when both hosts are up and there's 60Gb of RAM available to use overall.

However, I cannot, for the life of me, find out if I can do a similar thing with CPU cores. Does this mean I need to plan my core allocation to fit into the 22 available threads and waste half my processing power for a just-in-case scenario? Or will Hyper-V/VMM allow for CPU oversubscription, where in normal day-to-day activity there's no oversubscription taking place except when one host failsover? Potentially it's not as wasteful as it might seem as it would at least mean that VMs have real cores allocated rather than HyperThreaded Pretend Cores, except when failing over, but it still strikes me as under-utilising resources.

Has anyone set this up to know better than Google?

[DavidYoung]

You don't need to setup each vCPU so that it relates to a real core. We only run quad-cores with hyperthreading in our servers, and run way more than 8 vCPUs on each. You'll find that CPU is very rarely the bottleneck as much as disk and RAM. I would just give each VM a single core, unless it has high CPU needs and monitor it for a while.

Note: my experience is VMware, I thought it was similar, but turns out I'm wrong (posts below).

David.

[Achandler]

I can tell you right now, if you have to many CPUs handed out or to much RAM for that case, then shoudl 1 host fail then one or more of your VMs won't boot up/ migrate over as it ill say not enough resources available.

best bet is allocate like you have 22 CPUs and 30Gb RAM available, then just split your VMs across both hosts to share the load.

[DavidYoung]

I can tell you right now, if you have to many CPUs handed out or to much RAM for that case, then shoudl 1 host fail then one or more of your VMs won't boot up/ migrate over as it ill say not enough resources available.

Does Hyper-V require one physical core per virtual CPU then? I must admit I don't have much experiance with Hyper-V as we use VMware here, but we did trial an early version a while ago and I remember the memory limits.

[Achandler]

Nope 1 thread is a core, so you have 24 virtual cores (12 cores with hyperthreading (2 theads)) in total. So I would allocate upto 22 cores to virtual machines. This leave 1 core/ 2 threads left for the host.

[DavidYoung]

Nope 1 thread is a core, so you have 24 virtual cores in total (including the host). So I would allocate upto 22.

But does that mean that you can't have more than 24 VMs, each with only one CPU, 12 VMs with dual-CPU, etc?

[Achandler]

Yes that is true, one host could support upto 22 VMs with 1 CPU or 11 with 2 CPUs etc. The combination would need to be less then 22 CPUs ofcourse.

The dynamic memory allocation for the VMs only came into play in the last Service pack, so maybe dynamic cores numbers will be introduced soon.

[DavidYoung]

Yes that is true, one host could support upto 22 VMs with 1 CPU or 11 with 2 CPUs etc. The combination would need to be less then 22 CPUs ofcourse.

The dynamic memory allocation for the VMs only came into play in the last Service pack, so maybe dynamic cores numbers will be introduced soon.

Wow, did not know that, glad we went for VMware!

[sonofsanta]

I don't need massive numbers of virtual servers and the licensing for Hyper-V was much cheaper on an educational licence than VMWare seemed to be, so the limitation isn't a big issue. It's just annoying that effectively half the processing capacity is going wasted.

I fear that CPU cores won't ever be possible to dynamically alter without a reboot of a server. Although I'd have instinctively thought that of RAM as well, so maybe.

Lesson for future: more low powered servers better than a few high-spec servers. If I have three servers dual socketed with quad cores I'd be able to allocate up to 28 threads...

[Achandler]

Three lower spec servers would be better in this instance as a number of server possible, but might not be the best choice long term.

Im sure dynamic cores will be possible as it would be an underlying software software change to "insert" the cores or maybe they will work using 1 big core where you could allocate a percentage maximum and minimum of it (allowing over 100%). I haven't use VMware but David seems to be hinting it is entirely possible to overallocate already in VMware so I assume if a node fails the allocations get sorted out to keep all VMs working. If this is the case, then the technolgoy must exist.

[Arthur]

HyperV R2 has an 8:1 logical to virtual processor limit. HyperV in Windows Server 8 on the other hand won't have this limit.

[Arthur]

Lesson for future: more low powered servers better than a few high-spec servers. If I have three servers dual socketed with quad cores I'd be able to allocate up to 28 threads...

... or fewer servers with more cores per processor.

A server with two Xeon E5-2650's would give you 32 threads.

[sonofsanta]

HyperV R2 has an 8:1 logical to virtual processor limit. HyperV in Windows Server 8 on the other hand won't have this limit.

Wait... my virtual servers can have no more than a single quad core assigned to them? That's a bit pants, I was hoping to give Exchange 8 threads total :/

... or fewer servers with more cores per processor.

If I have two hosts in a cluster, I can only safely use up to 1/2 of available resources; with three, I could use 2/3; with four, 3/4 etc.

[Arthur]

The formula used to work out the maximum number of virtual processors your Hyper-V server supports is this...

(Number of processors) * (Number of cores) * (Number of threads per core) * 8

Source: http://blogs.technet.com/b/virtualiz...d-threads.aspx

Therefore, if you have a dual quad core Xeon with hyper-threading, you should be able to use upto 128 virtual processors per host.

[gshaw]

This is a great advert for N+1 virtual host design... if your capacity suits 2 servers get 3 so in case of failure the remaining host doesn't get overwhelmed... does depend on budget I guess but always worth bearing in mind...