ChrisMiles Posted April 4, 2017 Posted April 4, 2017 We have some new HP DL380s and DL580s running server 2016 with Hyper-V role. On all the servers I have a problem where randomly the vmcompute service seems to lock up preventing any vm state changes like starting, stopping and rebooting. Restarting this service forcibly seems to fix it for a while but upsets the cluster service. Has anyone else seen this issue? I've browsed the web and found loads of similar issues with 2012 but the fixes for those such as disabling VMQ and QoS don't seem to work. I have the latest drivers and firmware as well as the latest windows updates.
ChrisMiles Posted April 4, 2017 Author Posted April 4, 2017 Getting this crash: Version=1 EventType=APPCRASH EventTime=131357883734624932 ReportType=2 Consent=1 UploadTime=131357883734944929 ReportIdentifier=dc053f50-193f-11e7-a954-e0071b6e83e1 IntegratorReportIdentifier=724f102a-7a92-448e-9301-8fbed4a8a82f NsAppName=vmcompute.exe AppSessionGuid=00001628-0000-001d-a761-f89749add201 TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!00009a90425c5aa0499eabe1958da033616d3ba5bd28!vmcompute.exe TargetAppVer=2017//03//04:06:26:29!1e0fd5!vmcompute.exe BootId=4294967295 Response.BucketId=ae3769d70f9079aa24b7e02c3e9ce458 Response.BucketTable=4 Response.LegacyBucketId=120717097638 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=vmcompute.exe Sig[1].Name=Application Version Sig[1].Value=10.0.14393.953 Sig[2].Name=Application Timestamp Sig[2].Value=58ba5e15 Sig[3].Name=Fault Module Name Sig[3].Value=vmcompute.exe Sig[4].Name=Fault Module Version Sig[4].Value=10.0.14393.953 Sig[5].Name=Fault Module Timestamp Sig[5].Value=58ba5e15 Sig[6].Name=Exception Code Sig[6].Value=c0000005 Sig[7].Name=Exception Offset Sig[7].Value=00000000000c86ca DynamicSig[1].Name=OS Version DynamicSig[1].Value=10.0.14393.2.0.0.400.8 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=2057 DynamicSig[22].Name=Additional Information 1 DynamicSig[22].Value=84e3 DynamicSig[23].Name=Additional Information 2 DynamicSig[23].Value=84e3bcc16d77029e73f0d8c077d5fdd9 DynamicSig[24].Name=Additional Information 3 DynamicSig[24].Value=0673 DynamicSig[25].Name=Additional Information 4 DynamicSig[25].Value=06732e203c3436ba1c74ee3be1ffe61a UI[2]=C:\Windows\system32\vmcompute.exe UI[5]=Check online for a solution (recommended) UI[6]=Check for a solution later (recommended) UI[7]=Close UI[8]=Hyper-V Host Compute Service stopped working and was closed UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available. UI[10]=&Close LoadedModule[0]=C:\Windows\system32\vmcompute.exe LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll LoadedModule[4]=C:\Windows\System32\ucrtbase.dll LoadedModule[5]=C:\Windows\System32\sechost.dll LoadedModule[6]=C:\Windows\System32\RPCRT4.dll LoadedModule[7]=C:\Windows\System32\combase.dll LoadedModule[8]=C:\Windows\System32\bcryptPrimitives.dll LoadedModule[9]=C:\Windows\System32\shcore.dll LoadedModule[10]=C:\Windows\System32\msvcrt.dll LoadedModule[11]=C:\Windows\System32\advapi32.dll LoadedModule[12]=C:\Windows\system32\vmsif.dll LoadedModule[13]=C:\Windows\system32\VirtDisk.dll LoadedModule[14]=C:\Windows\system32\vmprox.dll LoadedModule[15]=C:\Windows\System32\OLEAUT32.dll LoadedModule[16]=C:\Windows\System32\msvcp_win.dll LoadedModule[17]=C:\Windows\System32\CRYPT32.dll LoadedModule[18]=C:\Windows\System32\MSASN1.dll LoadedModule[19]=C:\Windows\System32\cfgmgr32.dll LoadedModule[20]=C:\Windows\System32\WS2_32.dll LoadedModule[21]=C:\Windows\system32\CCGLaunchPad.dll LoadedModule[22]=C:\Windows\system32\vmwpctrl.dll LoadedModule[23]=C:\Windows\system32\IPHLPAPI.DLL LoadedModule[24]=C:\Windows\system32\XmlLite.dll LoadedModule[25]=C:\Windows\system32\wevtapi.dll LoadedModule[26]=C:\Windows\system32\FLTLIB.DLL LoadedModule[27]=C:\Windows\system32\DEVOBJ.dll LoadedModule[28]=C:\Windows\system32\NetSetupApi.dll LoadedModule[29]=C:\Windows\system32\fwpuclnt.dll LoadedModule[30]=C:\Windows\system32\MPR.dll LoadedModule[31]=C:\Windows\system32\bcrypt.dll LoadedModule[32]=C:\Windows\SYSTEM32\ntmarta.dll LoadedModule[33]=C:\Windows\system32\container.dll LoadedModule[34]=C:\Windows\system32\container_xml.dll LoadedModule[35]=C:\Windows\system32\wc_storage.dll LoadedModule[36]=C:\Windows\system32\vid.dll LoadedModule[37]=C:\Windows\system32\SspiCli.dll LoadedModule[38]=C:\Windows\system32\CRYPTBASE.DLL LoadedModule[39]=C:\Windows\System32\kernel.appcore.dll LoadedModule[40]=C:\Windows\System32\user32.dll LoadedModule[41]=C:\Windows\System32\win32u.dll LoadedModule[42]=C:\Windows\System32\GDI32.dll LoadedModule[43]=C:\Windows\System32\gdi32full.dll LoadedModule[44]=C:\Windows\System32\clbcatq.dll State[0].Key=Transport.DoneStage1 State[0].Value=1 FriendlyEventName=Stopped working ConsentKey=APPCRASH AppName=Hyper-V Host Compute Service AppPath=C:\Windows\system32\vmcompute.exe NsPartner=windows NsGroup=windows8 ApplicationIdentity=1D4B05C3DD517AD2FE7F352CE5DF9819 MetadataHash=-1371436347
Terens Posted April 30, 2017 Posted April 30, 2017 (edited) I have the same Problem with a Dell PowerEdge 530. I think, this error may occur, when there is heavy load on the disk (-array), where the hypervisor resides. e.g. if a guest VM on the same disk (-array) downloads and installs windows updates.... . I went back to 2012 R2 and have no problems anymore. Can you verify this relation between the crash and the load on the disk (-array)? Edited April 30, 2017 by Terens
tomsimmons Posted October 30, 2017 Posted October 30, 2017 We are having the same problem using Hyper-V 2016 Server (the non GUI set up). It does seem to tie into Terens' theory that it is update related, it always seems to happen after updates have been applied to the guest, nothing to do with the host. Has anyone found a solution to this yet? I tried the restarting the vmcompute service, it fails to stop, but terminating it through task manager then restarting I can the see all the stuck machines shown with a status of Off. However you can't restart them, they all response with a catastrophic failure message. restarting the host clears this. Tom
Steve21 Posted October 30, 2017 Posted October 30, 2017 Might be worth checking your NUMA settings on the 16 host, there was a KB on MS about it affecting it sometimes especially if it was upgraded from 2012r2 where it wasn't enabled by default Steve
ChrisMiles Posted October 30, 2017 Author Posted October 30, 2017 We are having the same problem using Hyper-V 2016 Server (the non GUI set up). It does seem to tie into Terens' theory that it is update related, it always seems to happen after updates have been applied to the guest, nothing to do with the host. Has anyone found a solution to this yet? I tried the restarting the vmcompute service, it fails to stop, but terminating it through task manager then restarting I can the see all the stuck machines shown with a status of Off. However you can't restart them, they all response with a catastrophic failure message. restarting the host clears this. Tom You can fix the catastrophic error by terminating the vmcompute service again once the machines are stopped. I still have this problem now. Nothing fixes it
tomsimmons Posted October 31, 2017 Posted October 31, 2017 Might be worth checking your NUMA settings on the 16 host, there was a KB on MS about it affecting it sometimes especially if it was upgraded from 2012r2 where it wasn't enabled by default Steve Steve We have NUMA turned off, not sure if we should or not, wasn't my decision don't know the reasoning behind the decision. I do recall someone mentioning this before, and seeing a KB, but can I find it now. We have 2 identical servers in a singe rack case, they don't work as a cluster, they are fully independent. I don't have the CPU & RAM specs to hand, but I do know that all the memory is identical performance in each server, so not sure NUMA would offer much to us. Do you have the KB number to hand, or can you recall the jist of what was being said? Tom
tomsimmons Posted October 31, 2017 Posted October 31, 2017 You can fix the catastrophic error by terminating the vmcompute service again once the machines are stopped. I still have this problem now. Nothing fixes it Chris When you terminate vmcompute the second time, do you have to resort to a brutal process kill again, or does the service stop cleanly from the services manager? Tom
ChrisMiles Posted October 31, 2017 Author Posted October 31, 2017 I admit I gave up trying to restart the service, I just terminate it again from task manager. Never seen any evidence of problems related to this but if a proper restart works the 2nd time then I guess it's safer.
ChrisMiles Posted November 1, 2017 Author Posted November 1, 2017 It may also help to know that for us the problem improved significantly ever since we stopped using dynamic memory.
tomsimmons Posted November 1, 2017 Posted November 1, 2017 It may also help to know that for us the problem improved significantly ever since we stopped using dynamic memory. Chris You haven't seen anything to suggest it is related to Windows updates of the guests? It is upon the restart of guests post updates that ours seems most likely to fail. When you say you stopped using dynamic memory, that is per machine you have set a fixed and permanently assign RAM size? Tom
ChrisMiles Posted November 1, 2017 Author Posted November 1, 2017 Yes, we changed to a fixed amount of RAM per machine. The problem did often rear it's head when VMs tried to restart for updates but I think this is because the problem only shows symptoms when VMs try to change state during a restart and this is mostly the only time it happens, not because of the updates themselves. I did have the issue at other times too, for example when trying to shutdown servers myself. From what I've seen, the vmcompute process borking up doesn't affect running servers except that it prevents them being assigned additional memory when using dynamic memory. This is what lead me to trying without dynamic memory.
tomsimmons Posted November 1, 2017 Posted November 1, 2017 (edited) Yes, we changed to a fixed amount of RAM per machine. The problem did often rear it's head when VMs tried to restart for updates but I think this is because the problem only shows symptoms when VMs try to change state during a restart and this is mostly the only time it happens, not because of the updates themselves. I did have the issue at other times too, for example when trying to shutdown servers myself. From what I've seen, the vmcompute process borking up doesn't affect running servers except that it prevents them being assigned additional memory when using dynamic memory. This is what lead me to trying without dynamic memory. Chris When you say that it prevents additional memory being assigned under dynamic, does that include after the service has been restarted on machines that were still running? Some of our guests don't get touched for a period of time, so Hyper-V will had cut their resources right the way back. If that machine is isn't stuck in the stopping state, but vmcompute is restarted to resolve others that are, I'm wondering if that machine won't be assigned more resource (RAM) when it does gets used unless it has also been restarted after the vmcompute restarts. Where did you find the crash log you posted? Tom Edited November 1, 2017 by tomsimmons
ChrisMiles Posted November 1, 2017 Author Posted November 1, 2017 If I remember correctly, restarting the service didn't fix the machines that wouldn't expand their memory any more. They had to be shut off and restarted. I could see in the Hyper-V Manager that the memory demand exceeded the allocated memory but it would not allocate more without a reboot. I'm not 100% sure though, it's been a while. The log came from the Application event log on the host server.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now