Koldov Posted August 6, 2022 Posted August 6, 2022 Anybody got a sure fire way of stopping this remotely?
Koldov Posted August 6, 2022 Author Posted August 6, 2022 No, it's the HOST for my VM PDC! This is all I managed to get before it rebooted again... The computer has rebooted from a bugcheck. The bugcheck was: 0x0000007e (0xffffffffc0000005, 0xfffff800acd5f58f, 0xffffd000213b8288, 0xffffd000213b7aa0). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 080622-28109-01.
Koldov Posted August 6, 2022 Author Posted August 6, 2022 I also managed to briefly look through the SYSTEM event log but nothing really stood out as a reason...
georgeescott Posted August 6, 2022 Posted August 6, 2022 You able to grab the MEMORY.DMP file before it reboots again? Is there iDRAC/iLO you could get to?
3s-gtech Posted August 6, 2022 Posted August 6, 2022 I faulting host is usually best fixed by reinstalling the OS - you can’t afford for it to be flaky. You can re-attach the VMs. There may be some fun if it’s a member server, but not if you have another independent DC (no PDCs, only FSMO role holders). If that’s not possible right now, try to stop it booting any VMs on startup in case they’re bringing it down. It’s an inherently stable platform so quite unusual for it to play up.
Koldov Posted August 6, 2022 Author Posted August 6, 2022 Unfortunately the iDRAC seems to be version 7 and Basic Management... which somehow seems to mean NO MANAGEMENT... There are no options in there at all. I missed the last reboot, but did shutdown the VM before that - it still rebooted... Will try to get to the .dmp file next time round...
Koldov Posted August 6, 2022 Author Posted August 6, 2022 (edited) The last command I ran before the restart was to reset the local security settings if that helps...? secedit /configure /cfg %windir%\inf\defltbase.inf /db defltbase.sdb /verbose In the event logs I did see: svchost (8640) Database recovery/restore failed with unexpected error -551. svchost (8640) Database recovery failed with error -551 because it encountered references to a database, 'C:\Windows\system32\LogFiles\Sum\SystemIdentity.mdb', which does not match the current set of logs. The database engine will not permit recovery to complete for this instance until the mismatching database is re-instated. If the database is truly no longer available or no longer required, procedures for recovering from this error are available in the Microsoft Knowledge Base or by following the "more information" link at the bottom of this message. Don't actually know what that means... Not sure if that would cause a reboot though... Edited August 6, 2022 by Koldov
3s-gtech Posted August 6, 2022 Posted August 6, 2022 I can put myself in your shoes, and have experience of failed hosts. You’re trying to think of the best course of action - repair or rebuild - but I think you will need to reinstall the host OS then reattach the VMs to it. You could cost yourself the weekend otherwise.
Koldov Posted August 6, 2022 Author Posted August 6, 2022 Yeah quite possibly but I have no access, so I'm not sure I can reinstall the OS remotely...? Also that means I leave it rebooting every few minutes until Monday morning...
Koldov Posted August 6, 2022 Author Posted August 6, 2022 Also this is going to hurt even more considering it is the SIMS server...
georgeescott Posted August 6, 2022 Posted August 6, 2022 (edited) If the server reboots too fast for you to get to the \\hostname\C$\Windows\ share from another server you could probably write a PowerShell script to loop until the host pings, then Test-Path the C$ share until it’s reachable, then copy the MEMORY.DMP file from it to another server so that you can catch it as soon as it’s available. You can then load it up in WinDbg (https://apps.microsoft.com/store/detail/windbg-preview/9PGJGD53TN86?hl=en-gb&gl=GB) and analyse it to see what might be the cause. You could also attempt to set the host to go in to safe mode with networking on the next reboot. Something like this would probably do it if you’re quick enough (not tested this so use at your own risk!): Enter-PSSession hostname & bcdedit /set '{default}' safeboot network Edited August 6, 2022 by georgeescott
psydii Posted August 6, 2022 Posted August 6, 2022 I say this a lot, but might be worth dropping £440 on a call to Microsoft support, which is something I've done a couple of times on my on credit card when something big breaks at the weekend.
Koldov Posted August 7, 2022 Author Posted August 7, 2022 (edited) Well, a bit bleary eyed this morning... I tried changing it to boot into safe mode with networking, but it didn't stop the boot loop, it did however stop me getting in via RDP... Then I tried to: Seize all FMSO roles to a different VM DC (luckily I went with 2 on different hosts) Clean-up Metadata Then remembering... Administrator isn't a member of Schema Admins (failed on last role seize) Create Time Server on new PDC Clean up of DNS might have to wait to see what I can salvage as it was a pain to change so if I can isolate the old VM PDC from the network I will demote and promote... I seem to have been able to remotely shutdown the host (so might save a bit of disk wear, until Monday morning when I can go in and assess the damage and start the real work... Luckily I got the MEMORY.DMP file first, I didn't have the time to get it from the server directly before the reboot but managed to get it to another server (as that was much quicker) and then copied it from there. I didn't have the store on my work laptop to get 'WinDbg Preview' so that was another issue to overcome trying to find in GitHub what file and what code to use (seems like it is fairly easy now though with no script or files to extract anymore), just a simple command.... genius whoever found that out! Worked for LTSC 2019, not entirely sure how much it enabled (Xbox crud, etc.) but it did the job. https://www.reddit.com/r/Windows10LTSC/comments/s88jre/guide_activateinstall_windows_store_without_an/ If anyone can make sense of the MEMORY.DMP file it is as follows: Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [C:\Users\Administrator\Desktop\MEMORY.DMP] Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available. ************* Path validation summary ************** Response Time (ms) Location Deferred srv* Symbol search path is: srv* Executable search path is: Windows 8.1 Kernel Version 9600 MP (16 procs) Free x64 Product: Server, suite: TerminalServer SingleUserTS Edition build lab: 9600.20475.amd64fre.winblue_ltsb_escrow.220622-1747 Machine Name: Kernel base = 0xfffff802`1de11000 PsLoadedModuleList = 0xfffff802`1e0d4650 Debug session time: Sat Aug 6 13:09:36.319 2022 (UTC + 1:00) System Uptime: 0 days 0:06:02.053 Loading Kernel Symbols ............................................................... ................................................................ ........................... Loading User Symbols Loading unloaded module list .... For analysis of this file, run !analyze -v nt!KeBugCheckEx: fffff802`1df504c0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffd000`21f222c0=000000000000007e 7: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e) This is a very common BugCheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Arguments: Arg1: ffffffffc0000005, The exception code that was not handled Arg2: fffff800904c458f, The address that the exception occurred at Arg3: ffffd00021f23288, Exception Record Address Arg4: ffffd00021f22aa0, Context Record Address Debugging Details: ------------------ KEY_VALUES_STRING: 1 Key : AV.Dereference Value: NullPtr Key : AV.Fault Value: Read Key : Analysis.CPU.mSec Value: 3952 Key : Analysis.DebugAnalysisManager Value: Create Key : Analysis.Elapsed.mSec Value: 3961 Key : Analysis.Init.CPU.mSec Value: 3405 Key : Analysis.Init.Elapsed.mSec Value: 10096 Key : Analysis.Memory.CommitPeak.Mb Value: 88 Key : Bugcheck.Code.DumpHeader Value: 0x7e Key : Bugcheck.Code.KiBugCheckData Value: 0x7e Key : Bugcheck.Code.Register Value: 0x7e Key : WER.OS.Branch Value: winblue_ltsb_escrow Key : WER.OS.Timestamp Value: 2022-06-22T17:47:00Z Key : WER.OS.Version Value: 8.1.9600.20475 FILE_IN_CAB: MEMORY.DMP BUGCHECK_CODE: 7e BUGCHECK_P1: ffffffffc0000005 BUGCHECK_P2: fffff800904c458f BUGCHECK_P3: ffffd00021f23288 BUGCHECK_P4: ffffd00021f22aa0 EXCEPTION_RECORD: ffffd00021f23288 -- (.exr 0xffffd00021f23288) ExceptionAddress: fffff800904c458f (NDIS!ndisCreateStringStreamEntry+0x000000000000002f) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 0000000000000000 Parameter[1]: 0000000000000000 Attempt to read from address 0000000000000000 CONTEXT: ffffd00021f22aa0 -- (.cxr 0xffffd00021f22aa0) rax=ffffd00021f23790 rbx=0000000000000015 rcx=ffffd00021f23540 rdx=ffffe00121e66ad8 rsi=ffffd00021f23680 rdi=ffffd00021f23540 rip=fffff800904c458f rsp=ffffd00021f234c0 rbp=0000000000000000 r8=ffffd00021f23680 r9=0000000000000000 r10=0000000000000000 r11=ffffd00021f23790 r12=fffff800904b1950 r13=ffffe0011e17c040 r14=ffffe00121e66990 r15=0000000000000000 iopl=0 nv up ei ng nz na po nc cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010286 NDIS!ndisCreateStringStreamEntry+0x2f: fffff800`904c458f 66413929 cmp word ptr [r9],bp ds:002b:00000000`00000000=???? Resetting default scope PROCESS_NAME: System READ_ADDRESS: 0000000000000000 ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE_STR: c0000005 EXCEPTION_PARAMETER1: 0000000000000000 EXCEPTION_PARAMETER2: 0000000000000000 EXCEPTION_STR: 0xc0000005 STACK_TEXT: ffffd000`21f234c0 fffff800`904c46a1 : 00000000`00000015 00000000`00000002 ffffe001`21e66ad8 ffffe001`2a1eea98 : NDIS!ndisCreateStringStreamEntry+0x2f ffffd000`21f234f0 fffff800`9043faa8 : ffffe001`21e66900 ffffd000`21f239d0 fffff800`904b2e00 ffffd000`21f239d0 : NDIS!ndisSqmLogDriverVersion+0xb9 ffffd000`21f238d0 fffff802`1de6a15f : fffff800`9043f958 fffff800`904b2ea8 00000000`00000000 fffff802`1e0ad4c0 : NDIS!ndisSqmTimerWorkerRoutine+0x150 ffffd000`21f23b50 fffff802`1dee27aa : ffffc000`000a0000 ffffd000`20fc0180 00000000`00000080 ffffe001`1d3968c0 : nt!ExpWorkerThread+0x69f ffffd000`21f23c00 fffff802`1df57f66 : ffffd000`20fc0180 ffffe001`1e17c040 ffffd000`20fd0680 00000000`00000000 : nt!PspSystemThreadStartup+0x18a ffffd000`21f23c60 00000000`00000000 : ffffd000`21f24000 ffffd000`21f1e000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16 SYMBOL_NAME: NDIS!ndisCreateStringStreamEntry+2f MODULE_NAME: NDIS IMAGE_NAME: NDIS.SYS STACK_COMMAND: .cxr 0xffffd00021f22aa0 ; kb BUCKET_ID_FUNC_OFFSET: 2f FAILURE_BUCKET_ID: AV_NDIS!ndisCreateStringStreamEntry OS_VERSION: 8.1.9600.20475 BUILDLAB_STR: winblue_ltsb_escrow OSPLATFORM_TYPE: x64 OSNAME: Windows 8.1 FAILURE_ID_HASH: {a6009a6f-0469-bc4c-27f7-a8fa2f293092} Followup: MachineOwner --------- Edited August 7, 2022 by Koldov
Koldov Posted August 7, 2022 Author Posted August 7, 2022 (edited) Just a random thought.... Looking through the MEMORY.DMP it notes NDIS.SYS and a lot of Google results suggest NIC drivers need updating... but it has been solid for so long and any number of reboots before this happened... Then I saw a thread somewhere about NDIS.SYS causing BSOD and the answer was about the following: Replace: WinPcap to npcap Then it hit me, the only change I made to the server yesterday was updating Wireshark (which is probably not a good idea on a prod server but I was trying to deal with another error about NTLM being blocked stopping 'scan to folder')..... That also had references to WinPcap and npcap during the install! Here's an old GitHub thread with references: https://github.com/nmap/npcap/issues/565 Specifically: "Bluescreen (ndis.sys) on Windows 8.1 with npcap 1,60" "Hi, have the same issue. Fails in ndisCreateStringStreamEntry." "I have the same issue. Npcap 1.60 and Windows Server 2012 R2. I will send you a minidump. Oddly, this problem only started approximately 1 day after initial installation. Without Driver Verifier: The machine will bluescreen within a few minutes after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with a reference to ndis.sys. With Driver Verifier: The machine will bluescreen within a few seconds after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with no reference to any driver. I can confirm that "Raw 802.11 Packet Capture Support" seems to be the problem, as everything seems to work fine after I reinstalled without that option." Edited August 7, 2022 by Koldov
georgeescott Posted August 8, 2022 Posted August 8, 2022 Just a random thought.... Looking through the MEMORY.DMP it notes NDIS.SYS and a lot of Google results suggest NIC drivers need updating... but it has been solid for so long and any number of reboots before this happened... Then I saw a thread somewhere about NDIS.SYS causing BSOD and the answer was about the following: Replace: WinPcap to npcap Then it hit me, the only change I made to the server yesterday was updating Wireshark (which is probably not a good idea on a prod server but I was trying to deal with another error about NTLM being blocked stopping 'scan to folder')..... That also had references to WinPcap and npcap during the install! Here's an old GitHub thread with references: https://github.com/nmap/npcap/issues/565 Specifically: "Bluescreen (ndis.sys) on Windows 8.1 with npcap 1,60" "Hi, have the same issue. Fails in ndisCreateStringStreamEntry." "I have the same issue. Npcap 1.60 and Windows Server 2012 R2. I will send you a minidump. Oddly, this problem only started approximately 1 day after initial installation. Without Driver Verifier: The machine will bluescreen within a few minutes after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with a reference to ndis.sys. With Driver Verifier: The machine will bluescreen within a few seconds after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with no reference to any driver. I can confirm that "Raw 802.11 Packet Capture Support" seems to be the problem, as everything seems to work fine after I reinstalled without that option." Nice find! This sounds exactly like what it could be. A reboot every 6 minutes. Hopefully you can modify the install within 6 minutes to remove the "Raw 802.11 Packet Capture Support" or remove WireShark/Npcap and the reboots will stop. Safe mode probably won't load this driver so you could do it then too. If you were troubleshooting a VM NTLM issues, You can always use Hyper-V Port Mirroring to capture packets from one VM to another running WireShark rather than installing it on the host.
Koldov Posted August 8, 2022 Author Posted August 8, 2022 Turns out that seemed to be the issue (I completely uninstalled Wireshark), I've always been vary wary of installing 3rd party software on my servers (and now I'll be even more so). Safe mode didn't stop the boot loop unfortunately (but it did kill any chance of using RDP), luckily I live quite close so went in and did it. It was just NTLM in general I was looking at, as a setting I had to deny it (and which I'd had set for ages) obviously wasn't applying with our old 2012R2 DCs, now we've moved to 2019 it seems to have kicked in and killed scan to folder. 1
3s-gtech Posted August 8, 2022 Posted August 8, 2022 You can put third-party software on your servers, but your Hyper-V hosts should have the absolute minimum on them - backup mechanism, maybe UPS software, little else - and this is a great case study! 1
Koldov Posted August 8, 2022 Author Posted August 8, 2022 this is a great case study! Glad I could be of service.... 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now