Hyper-V Guests Randomly Stick on Stopping - vmcompute

[ChrisMiles]

We have some new HP DL380s and DL580s running server 2016 with Hyper-V role. On all the servers I have a problem where randomly the vmcompute service seems to lock up preventing any vm state changes like starting, stopping and rebooting. Restarting this service forcibly seems to fix it for a while but upsets the cluster service.

 

Has anyone else seen this issue? I've browsed the web and found loads of similar issues with 2012 but the fixes for those such as disabling VMQ and QoS don't seem to work.

 

I have the latest drivers and firmware as well as the latest windows updates.

[ChrisMiles]

Getting this crash:

 

Version=1

EventType=APPCRASH

EventTime=131357883734624932

ReportType=2

Consent=1

UploadTime=131357883734944929

ReportIdentifier=dc053f50-193f-11e7-a954-e0071b6e83e1

IntegratorReportIdentifier=724f102a-7a92-448e-9301-8fbed4a8a82f

NsAppName=vmcompute.exe

AppSessionGuid=00001628-0000-001d-a761-f89749add201

TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!00009a90425c5aa0499eabe1958da033616d3ba5bd28!vmcompute.exe

TargetAppVer=2017//03//04:06:26:29!1e0fd5!vmcompute.exe

BootId=4294967295

Response.BucketId=ae3769d70f9079aa24b7e02c3e9ce458

Response.BucketTable=4

Response.LegacyBucketId=120717097638

Response.type=4

Sig[0].Name=Application Name

Sig[0].Value=vmcompute.exe

Sig[1].Name=Application Version

Sig[1].Value=10.0.14393.953

Sig[2].Name=Application Timestamp

Sig[2].Value=58ba5e15

Sig[3].Name=Fault Module Name

Sig[3].Value=vmcompute.exe

Sig[4].Name=Fault Module Version

Sig[4].Value=10.0.14393.953

Sig[5].Name=Fault Module Timestamp

Sig[5].Value=58ba5e15

Sig[6].Name=Exception Code

Sig[6].Value=c0000005

Sig[7].Name=Exception Offset

Sig[7].Value=00000000000c86ca

DynamicSig[1].Name=OS Version

DynamicSig[1].Value=10.0.14393.2.0.0.400.8

DynamicSig[2].Name=Locale ID

DynamicSig[2].Value=2057

DynamicSig[22].Name=Additional Information 1

DynamicSig[22].Value=84e3

DynamicSig[23].Name=Additional Information 2

DynamicSig[23].Value=84e3bcc16d77029e73f0d8c077d5fdd9

DynamicSig[24].Name=Additional Information 3

DynamicSig[24].Value=0673

DynamicSig[25].Name=Additional Information 4

DynamicSig[25].Value=06732e203c3436ba1c74ee3be1ffe61a

UI[2]=C:\Windows\system32\vmcompute.exe

UI[5]=Check online for a solution (recommended)

UI[6]=Check for a solution later (recommended)

UI[7]=Close

UI[8]=Hyper-V Host Compute Service stopped working and was closed

UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.

UI[10]=&Close

LoadedModule[0]=C:\Windows\system32\vmcompute.exe

LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll

LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL

LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll

LoadedModule[4]=C:\Windows\System32\ucrtbase.dll

LoadedModule[5]=C:\Windows\System32\sechost.dll

LoadedModule[6]=C:\Windows\System32\RPCRT4.dll

LoadedModule[7]=C:\Windows\System32\combase.dll

LoadedModule[8]=C:\Windows\System32\bcryptPrimitives.dll

LoadedModule[9]=C:\Windows\System32\shcore.dll

LoadedModule[10]=C:\Windows\System32\msvcrt.dll

LoadedModule[11]=C:\Windows\System32\advapi32.dll

LoadedModule[12]=C:\Windows\system32\vmsif.dll

LoadedModule[13]=C:\Windows\system32\VirtDisk.dll

LoadedModule[14]=C:\Windows\system32\vmprox.dll

LoadedModule[15]=C:\Windows\System32\OLEAUT32.dll

LoadedModule[16]=C:\Windows\System32\msvcp_win.dll

LoadedModule[17]=C:\Windows\System32\CRYPT32.dll

LoadedModule[18]=C:\Windows\System32\MSASN1.dll

LoadedModule[19]=C:\Windows\System32\cfgmgr32.dll

LoadedModule[20]=C:\Windows\System32\WS2_32.dll

LoadedModule[21]=C:\Windows\system32\CCGLaunchPad.dll

LoadedModule[22]=C:\Windows\system32\vmwpctrl.dll

LoadedModule[23]=C:\Windows\system32\IPHLPAPI.DLL

LoadedModule[24]=C:\Windows\system32\XmlLite.dll

LoadedModule[25]=C:\Windows\system32\wevtapi.dll

LoadedModule[26]=C:\Windows\system32\FLTLIB.DLL

LoadedModule[27]=C:\Windows\system32\DEVOBJ.dll

LoadedModule[28]=C:\Windows\system32\NetSetupApi.dll

LoadedModule[29]=C:\Windows\system32\fwpuclnt.dll

LoadedModule[30]=C:\Windows\system32\MPR.dll

LoadedModule[31]=C:\Windows\system32\bcrypt.dll

LoadedModule[32]=C:\Windows\SYSTEM32\ntmarta.dll

LoadedModule[33]=C:\Windows\system32\container.dll

LoadedModule[34]=C:\Windows\system32\container_xml.dll

LoadedModule[35]=C:\Windows\system32\wc_storage.dll

LoadedModule[36]=C:\Windows\system32\vid.dll

LoadedModule[37]=C:\Windows\system32\SspiCli.dll

LoadedModule[38]=C:\Windows\system32\CRYPTBASE.DLL

LoadedModule[39]=C:\Windows\System32\kernel.appcore.dll

LoadedModule[40]=C:\Windows\System32\user32.dll

LoadedModule[41]=C:\Windows\System32\win32u.dll

LoadedModule[42]=C:\Windows\System32\GDI32.dll

LoadedModule[43]=C:\Windows\System32\gdi32full.dll

LoadedModule[44]=C:\Windows\System32\clbcatq.dll

State[0].Key=Transport.DoneStage1

State[0].Value=1

FriendlyEventName=Stopped working

ConsentKey=APPCRASH

AppName=Hyper-V Host Compute Service

AppPath=C:\Windows\system32\vmcompute.exe

NsPartner=windows

NsGroup=windows8

ApplicationIdentity=1D4B05C3DD517AD2FE7F352CE5DF9819

MetadataHash=-1371436347

[Terens]

I have the same Problem with a Dell PowerEdge 530.

I think, this error may occur, when there is heavy load on the disk (-array), where the hypervisor resides. e.g. if a guest VM on the same disk (-array) downloads and installs windows updates.... .

 

I went back to 2012 R2 and have no problems anymore.

 

 

Can you verify this relation between the crash and the load on the disk (-array)?

Edited April 30, 2017 by Terens

[tomsimmons]

We are having the same problem using Hyper-V 2016 Server (the non GUI set up).

 

It does seem to tie into Terens' theory that it is update related, it always seems to happen after updates have been applied to the guest, nothing to do with the host. Has anyone found a solution to this yet?

 

I tried the restarting the vmcompute service, it fails to stop, but terminating it through task manager then restarting I can the see all the stuck machines shown with a status of Off. However you can't restart them, they all response with a catastrophic failure message. restarting the host clears this.

 

 

Tom

[Steve21]

Might be worth checking your NUMA settings on the 16 host, there was a KB on MS about it affecting it sometimes especially if it was upgraded from 2012r2 where it wasn't enabled by default

 

Steve

[ChrisMiles]

We are having the same problem using Hyper-V 2016 Server (the non GUI set up).

 

It does seem to tie into Terens' theory that it is update related, it always seems to happen after updates have been applied to the guest, nothing to do with the host. Has anyone found a solution to this yet?

 

I tried the restarting the vmcompute service, it fails to stop, but terminating it through task manager then restarting I can the see all the stuck machines shown with a status of Off. However you can't restart them, they all response with a catastrophic failure message. restarting the host clears this.

 

 

Tom

 

You can fix the catastrophic error by terminating the vmcompute service again once the machines are stopped. I still have this problem now. Nothing fixes it

[tomsimmons]

Might be worth checking your NUMA settings on the 16 host, there was a KB on MS about it affecting it sometimes especially if it was upgraded from 2012r2 where it wasn't enabled by default

 

Steve

 

Steve

 

We have NUMA turned off, not sure if we should or not, wasn't my decision don't know the reasoning behind the decision.

 

I do recall someone mentioning this before, and seeing a KB, but can I find it now.

 

We have 2 identical servers in a singe rack case, they don't work as a cluster, they are fully independent. I don't have the CPU & RAM specs to hand, but I do know that all the memory is identical performance in each server, so not sure NUMA would offer much to us.

 

Do you have the KB number to hand, or can you recall the jist of what was being said?

 

 

Tom

[tomsimmons]

You can fix the catastrophic error by terminating the vmcompute service again once the machines are stopped. I still have this problem now. Nothing fixes it

 

Chris

 

When you terminate vmcompute the second time, do you have to resort to a brutal process kill again, or does the service stop cleanly from the services manager?

 

 

Tom

[ChrisMiles]

I admit I gave up trying to restart the service, I just terminate it again from task manager. Never seen any evidence of problems related to this but if a proper restart works the 2nd time then I guess it's safer.

[ChrisMiles]

It may also help to know that for us the problem improved significantly ever since we stopped using dynamic memory.

[tomsimmons]

It may also help to know that for us the problem improved significantly ever since we stopped using dynamic memory.

 

Chris

 

You haven't seen anything to suggest it is related to Windows updates of the guests? It is upon the restart of guests post updates that ours seems most likely to fail.

 

When you say you stopped using dynamic memory, that is per machine you have set a fixed and permanently assign RAM size?

 

 

Tom

[ChrisMiles]

Yes, we changed to a fixed amount of RAM per machine. The problem did often rear it's head when VMs tried to restart for updates but I think this is because the problem only shows symptoms when VMs try to change state during a restart and this is mostly the only time it happens, not because of the updates themselves. I did have the issue at other times too, for example when trying to shutdown servers myself. From what I've seen, the vmcompute process borking up doesn't affect running servers except that it prevents them being assigned additional memory when using dynamic memory. This is what lead me to trying without dynamic memory.

[tomsimmons]

Yes, we changed to a fixed amount of RAM per machine. The problem did often rear it's head when VMs tried to restart for updates but I think this is because the problem only shows symptoms when VMs try to change state during a restart and this is mostly the only time it happens, not because of the updates themselves. I did have the issue at other times too, for example when trying to shutdown servers myself. From what I've seen, the vmcompute process borking up doesn't affect running servers except that it prevents them being assigned additional memory when using dynamic memory. This is what lead me to trying without dynamic memory.

 

Chris

 

When you say that it prevents additional memory being assigned under dynamic, does that include after the service has been restarted on machines that were still running?

 

Some of our guests don't get touched for a period of time, so Hyper-V will had cut their resources right the way back. If that machine is isn't stuck in the stopping state, but vmcompute is restarted to resolve others that are, I'm wondering if that machine won't be assigned more resource (RAM) when it does gets used unless it has also been restarted after the vmcompute restarts.

 

Where did you find the crash log you posted?

 

 

Tom

Edited November 1, 2017 by tomsimmons

[ChrisMiles]

If I remember correctly, restarting the service didn't fix the machines that wouldn't expand their memory any more. They had to be shut off and restarted. I could see in the Hyper-V Manager that the memory demand exceeded the allocated memory but it would not allocate more without a reboot. I'm not 100% sure though, it's been a while. The log came from the Application event log on the host server.