[VMware ESX] Server CPU Resource Allocations - Confused

[CHiLL]

We have 2x HP DL360 Gen9, which both have 2x Intel Xeon E5-2630 v3 CPUs and 192GB RAM per host. The hosts are both running ESXi 6.7 (coupled with VCSA 6.7). Since we have scheduled downtime, I'm looking at the resource allocation for our servers and have managed to get our RAM allocations sorted. However I'm not sure how to assign the CPUs. When modifying the CPU values, what's the difference between CPU and Cores per socket? Could someone recommend the CPU allocation for the following types of servers?

 

Server RAM allocation:

• File server with shared applications - 4GB
  
• Backup Server - 8GB
  
• Biometrics server - 4GB
  
• Domain Controller - 8GB
  
• Management server (includes Impero Server, DirSync and SalamanderAD) - 16GB
  
• SIMS - 16GB
  
• Web Server - Spiceworks - 4GB
  
• File Server - Students - 12GB
  
• ADFS Web Application Proxy server - 4GB
  
• Old CMIS Server - 8GB
  
• ADFS FED Server - 4GB
  
• RDS -24GB
  
• RDS Gateway - 4GB
  
• Web server - Home Access Plus - 4GB
  
• SCCM Server - 16GB
  
• Print server - 4GB
  
• File Server - Staff - 12GB
  

 

What sorts of CPU allocations (both CPU and core per socket) should I be setting for the above servers?

[liam_impero]

Hi CHiLL,

 

I’m Liam, Infrastructure and Systems Manager at Impero Software. I'm answering this in capacity as part of the Edugeek community.

 

In short, there is no difference between vCPU cores and sockets. The idea was included in VMWare as a way to get around limitations of the guest OS. For example in Windows Server 2008 (yes we have to go back that far!), there was a limitation of four PHYSICAL CPUs. In VMWare, no matter how many sockets you assign this server (for example 8), it will not recognise more than 4 of the vCPUs assigned to it. So in comes the theory of logical cores per socket (ie, one physical CPU acting as many CPUs). We can assign the same 2008 server a single socket with 8 cores, it’ll happily use all 8 of the vCPUs assigned to it. This limitation has all but disappeared in more modern OS’s (from memory Server 2016 supports 512 CPUs!).

 

CPU is always heavily over provisioned on the Hypervisors, so you can really assign as much as you like to the guest servers . I find a single socket with 4 cores works happily for most use cases. That being said, look up the minimum specs for each application/server role you’re running to be sure they are resourced appropriately.

 

Many thanks,

Liam

[CHiLL]

Hi CHiLL,

 

I’m Liam, Infrastructure and Systems Manager at Impero Software. I'm answering this in capacity as part of the Edugeek community.

 

In short, there is no difference between vCPU cores and sockets. The idea was included in VMWare as a way to get around limitations of the guest OS. For example in Windows Server 2008 (yes we have to go back that far!), there was a limitation of four PHYSICAL CPUs. In VMWare, no matter how many sockets you assign this server (for example 8), it will not recognise more than 4 of the vCPUs assigned to it. So in comes the theory of logical cores per socket (ie, one physical CPU acting as many CPUs). We can assign the same 2008 server a single socket with 8 cores, it’ll happily use all 8 of the vCPUs assigned to it. This limitation has all but disappeared in more modern OS’s (from memory Server 2016 supports 512 CPUs!).

 

CPU is always heavily over provisioned on the Hypervisors, so you can really assign as much as you like to the guest servers . I find a single socket with 4 cores works happily for most use cases. That being said, look up the minimum specs for each application/server role you’re running to be sure they are resourced appropriately.

 

Many thanks,

Liam

Thanks for the reply. Whilst I had the servers down, I decided to take the following approach:

 

Servers with <=8GB RAM - 4 CPUs and 4 cores.

Servers with >8GB RAM - 8 CPUs and 8 cores.

[Steve21]

I'd add a caveat to the above though, remember to check things like licensing especially regarding SQL etc to see what it needs.

 

A prime example is anything using SQL express is limited to 1 socket, or 4 core (Using the lower of the two), If you have 4x 1 Sockets, you'll only use 25% of the CPU etc compared to 1x 4 socket.

 

Steve

[Domino]

Thanks for the reply. Whilst I had the servers down, I decided to take the following approach:

 

Servers with <=8GB RAM - 4 CPUs and 4 cores.

Servers with >8GB RAM - 8 CPUs and 8 cores.

 

Errr what? 4CPUs and 4Cores per socket? One of those uses one whole physical server's worth of allocation. The level of contention there must be epic, what's the current CPU Ready time for a server?

 

You're also crossing NUMA nodes with the larger ones - Which is a performance hit.

 

You should really be thinking what a server *needs* to do the work it's given, starting low and working up in allocation, there's more on this here: http://www.edugeek.net/forums/enterprise-software/185012-want-compare-my-esxi-ram-allocation-others.html

Edited August 2, 2018 by Domino

[CHiLL]

Errr what? 4CPUs and 4Cores per socket? One of those uses one whole physical server's worth of allocation. The level of contention there must be epic, what's the current CPU Ready time for a server?

 

You're also crossing NUMA nodes with the larger ones - Which is a performance hit.

 

You should really be thinking what a server *needs* to do the work it's given, starting low and working up in allocation, there's more on this here: http://www.edugeek.net/forums/enterprise-software/185012-want-compare-my-esxi-ram-allocation-others.html

This is why I've created this thread, to help me understand the situation. I've read through that thread and I'm still none the wiser on how to allocate the CPUs.

 

What's the best way to spread the CPUs without one VM using one whole physical server's worth of allocation?

 

Here's our CPU performance for out cluster over the past 24 hours (with practically nobody on-site).

 

[ATTACH=CONFIG]49895[/ATTACH]

Edited August 2, 2018 by CHiLL

[Domino]

This is why I've created this thread, to help me understand the situation. I've read through that thread and I'm still none the wiser on how to allocate the CPUs.

 

The right-sizing link examples should give an indication.

 

There's no hard and fast rules as to exactly how much a vm should have, Size for the workload. You also need to have an understanding that this is an ecosystem of machines working on the same hardware. More vCPUs != more power.

 

I'd always advise starting low and working up. Consider the physical CPUs on a host. This is how many vCPUs can do work *at the same time* any others needing CPU time are then scheduled and have to wait till they can get a turn on the physcial hardware. This is value is reflected in the performance stats as "CPU Ready" time.

 

If your physical hardware only has enough room for one or two of these large VMs, then that is how many concurrent machines can be using CPUs simultaneously. Many 2vCPU machines are easier to schedule than 4 16vCPU.

 

In real terms having a massive overallocation of resource will slow everything down. This becomes more apparent under load, as machines have to wait longer for other systems to finish.

 

My Standard at this business is 2vCPU by standard, 4 if needed after performance tuning. Anything above that is a special case and needs investigation.

 

Additionally, your graph will show low usage when overallocated, it literally can't run enough VMs concurrently to make usage of the CPU power.

Edited August 2, 2018 by Domino

[Steve21]

What's the best way to spread the CPUs without one VM using one whole physical server's worth of allocation?

 

Generally I work backwards from the CPUs, if you mean these ones from your link: https://ark.intel.com/products/83356/Intel-Xeon-Processor-E5-2630-v3-20M-Cache-2_40-GHz

 

You got 16 threads per CPU, so 64 across the cluster to play with (for 0 contention, generally I go max about 2x if it's for some servers that aren't always in use e.g. backup etc).

 

To put this in perspective compared to ours (and I've tried to match the closest style of server we have to yours:

 

File server with shared applications - 4GB 2

Backup Server - 8GB 4

Biometrics server - 4GB 4

Domain Controller - 8GB 2

Management server (includes Impero Server, DirSync and SalamanderAD) - 16GB 4

SIMS - 16GB 8

Web Server - Spiceworks - 4GB 2

File Server - Students - 12GB 2

ADFS Web Application Proxy server - 4GB 4

Old CMIS Server - 8GB 4

ADFS FED Server - 4GB 4

RDS -24GB 4

RDS Gateway - 4GB 1

Web server - Home Access Plus - 4GB 1

SCCM Server - 16GB 6

Print server - 4GB 4

File Server - Staff - 12GB 2

 

Total 58

 

We have a few styles of servers you don't as I break things down a lot more, but remember a lot of these things e.g. file servers don't really use CPU much until dedup is run which is normally overnight etc. Maybe you'd want more on RDS etc if it's higher too, but I wouldn't go overboard unless you're maxing out an individuals VM vCPU all the time.

 

Steve

[RobD]

Here's what I would do;

 

Assign each server with one core and 2 VCPUs then leave them for a period of time and take a look at the performance of the server. If its suffering then add more vcpus and then more sockets.

 

 

I've not watched this all but looks pretty good guide to give you an idea on what to look for:

[CHiLL]

Thanks for the replies. I think I'm getting confused with vCPUs. When assigning CPU resources to a VM and I want to use say 2 vCPUs, how do I configure that in vSphere? I can see the CPU drop down with 1-32 as an option and I can see cores per socket, which has a multiple of the selected CPU value. What do I set these to to get 2 vCPUs for that VM?

[Domino]

Just set the number of Virtual Processors and leave cores per socket at 1.

[Steve21]

That depends if you actually want 2 CPus or 1x 2core

 

For example,

 

1 CPU with 2 Core = 2 VCPU + 2 Socket

2 CPU with 1 Core = 2 VCPU + 1 Socket

 

1 CPU with 4 Core = 4 VCPU + 4 Socket

2 CPU with 2 Core = 4 VCPU + 2 Socket

4 CPU with 1 Core = 4 VCPU + 1 Socket

 

If that makes sense?

 

Steve

[CHiLL]

That depends if you actually want 2 CPus or 1x 2core

 

For example,

 

1 CPU with 2 Core = 2 VCPU + 2 Socket

2 CPU with 1 Core = 2 VCPU + 1 Socket

 

1 CPU with 4 Core = 4 VCPU + 4 Socket

2 CPU with 2 Core = 4 VCPU + 2 Socket

4 CPU with 1 Core = 4 VCPU + 1 Socket

 

If that makes sense?

 

Steve

So for overall performance without over-committing the cores (as Domino was suggesting), what would be best? I've been comparing it to standard desktop CPUs, where I have 1 CPU with 4, but I guess it doesn't work that way with vCPUs!

[Steve21]

As long as you're not going massively over the 64 threads mentioned above you shouldn't hit any major issues (but as with everything you need to check it over time as might need to tweak them)

 

Whether you do the 2 or 1 CPU route won't affect anything, as they're all "fake" based on the thread counts

 

Steve

[Domino]

^ this. They could just put "CPUs" - but they added cores to deal with licensing/software constraints of things running within the VM

[CHiLL]

Thanks for the responses. I now have the CPUs set in what I think is a more optimised way.