2012R2 server just gone into boot loop - working remotely!!!!

[Koldov]

Anybody got a sure fire way of stopping this remotely?

[3s-gtech]

Got the stop code? Is it a VM?

[Koldov]

No, it's the HOST for my VM PDC!

 

This is all I managed to get before it rebooted again...

 

The computer has rebooted from a bugcheck. The bugcheck was: 0x0000007e (0xffffffffc0000005, 0xfffff800acd5f58f, 0xffffd000213b8288, 0xffffd000213b7aa0). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 080622-28109-01.

[Koldov]

I also managed to briefly look through the SYSTEM event log but nothing really stood out as a reason...

[georgeescott]

You able to grab the MEMORY.DMP file before it reboots again?

 

Is there iDRAC/iLO you could get to?

[3s-gtech]

I faulting host is usually best fixed by reinstalling the OS - you can’t afford for it to be flaky. You can re-attach the VMs. There may be some fun if it’s a member server, but not if you have another independent DC (no PDCs, only FSMO role holders).

 

If that’s not possible right now, try to stop it booting any VMs on startup in case they’re bringing it down. It’s an inherently stable platform so quite unusual for it to play up.

[Koldov]

Unfortunately the iDRAC seems to be version 7 and Basic Management... which somehow seems to mean NO MANAGEMENT... There are no options in there at all.

 

I missed the last reboot, but did shutdown the VM before that - it still rebooted...

 

Will try to get to the .dmp file next time round...

[Koldov]

The last command I ran before the restart was to reset the local security settings if that helps...?

 

secedit /configure /cfg %windir%\inf\defltbase.inf /db defltbase.sdb /verbose

 

In the event logs I did see:

 

svchost (8640) Database recovery/restore failed with unexpected error -551.

 

svchost (8640) Database recovery failed with error -551 because it encountered references to a database, 'C:\Windows\system32\LogFiles\Sum\SystemIdentity.mdb', which does not match the current set of logs. The database engine will not permit recovery to complete for this instance until the mismatching database is re-instated. If the database is truly no longer available or no longer required, procedures for recovering from this error are available in the Microsoft Knowledge Base or by following the "more information" link at the bottom of this message.

 

Don't actually know what that means... Not sure if that would cause a reboot though...

Edited August 6, 2022 by Koldov

[3s-gtech]

I can put myself in your shoes, and have experience of failed hosts. You’re trying to think of the best course of action - repair or rebuild - but I think you will need to reinstall the host OS then reattach the VMs to it. You could cost yourself the weekend otherwise.

[Koldov]

Yeah quite possibly but I have no access, so I'm not sure I can reinstall the OS remotely...?

 

Also that means I leave it rebooting every few minutes until Monday morning...

[Koldov]

Also this is going to hurt even more considering it is the SIMS server...

[georgeescott]

If the server reboots too fast for you to get to the \\hostname\C$\Windows\ share from another server you could probably write a PowerShell script to loop until the host pings, then Test-Path the C$ share until it’s reachable, then copy the MEMORY.DMP file from it to another server so that you can catch it as soon as it’s available.

 

You can then load it up in WinDbg (https://apps.microsoft.com/store/detail/windbg-preview/9PGJGD53TN86?hl=en-gb&gl=GB) and analyse it to see what might be the cause.

 

You could also attempt to set the host to go in to safe mode with networking on the next reboot. Something like this would probably do it if you’re quick enough (not tested this so use at your own risk!):

Enter-PSSession hostname
& bcdedit /set '{default}' safeboot network

Edited August 6, 2022 by georgeescott

[psydii]

I say this a lot, but might be worth dropping £440 on a call to Microsoft support, which is something I've done a couple of times on my on credit card when something big breaks at the weekend.

[Koldov]

Well, a bit bleary eyed this morning...

 

I tried changing it to boot into safe mode with networking, but it didn't stop the boot loop, it did however stop me getting in via RDP...

 

Then I tried to:

 

Seize all FMSO roles to a different VM DC (luckily I went with 2 on different hosts)

Clean-up Metadata

 

Then remembering...

 

Administrator isn't a member of Schema Admins (failed on last role seize)

Create Time Server on new PDC

Clean up of DNS might have to wait to see what I can salvage as it was a pain to change so if I can isolate the old VM PDC from the network I will demote and promote...

 

I seem to have been able to remotely shutdown the host (so might save a bit of disk wear, until Monday morning when I can go in and assess the damage and start the real work...

 

Luckily I got the MEMORY.DMP file first, I didn't have the time to get it from the server directly before the reboot but managed to get it to another server (as that was much quicker) and then copied it from there. I didn't have the store on my work laptop to get 'WinDbg Preview' so that was another issue to overcome trying to find in GitHub what file and what code to use (seems like it is fairly easy now though with no script or files to extract anymore), just a simple command.... genius whoever found that out! Worked for LTSC 2019, not entirely sure how much it enabled (Xbox crud, etc.) but it did the job.

 

https://www.reddit.com/r/Windows10LTSC/comments/s88jre/guide_activateinstall_windows_store_without_an/

 

If anyone can make sense of the MEMORY.DMP file it is as follows:

 

Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\Administrator\Desktop\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
Windows 8.1 Kernel Version 9600 MP (16 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 9600.20475.amd64fre.winblue_ltsb_escrow.220622-1747
Machine Name:
Kernel base = 0xfffff802`1de11000 PsLoadedModuleList = 0xfffff802`1e0d4650
Debug session time: Sat Aug  6 13:09:36.319 2022 (UTC + 1:00)
System Uptime: 0 days 0:06:02.053
Loading Kernel Symbols
...............................................................
................................................................
...........................
Loading User Symbols

Loading unloaded module list
....
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff802`1df504c0 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:ffffd000`21f222c0=000000000000007e
7: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common BugCheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff800904c458f, The address that the exception occurred at
Arg3: ffffd00021f23288, Exception Record Address
Arg4: ffffd00021f22aa0, Context Record Address

Debugging Details:
------------------


KEY_VALUES_STRING: 1

   Key  : AV.Dereference
   Value: NullPtr

   Key  : AV.Fault
   Value: Read

   Key  : Analysis.CPU.mSec
   Value: 3952

   Key  : Analysis.DebugAnalysisManager
   Value: Create

   Key  : Analysis.Elapsed.mSec
   Value: 3961

   Key  : Analysis.Init.CPU.mSec
   Value: 3405

   Key  : Analysis.Init.Elapsed.mSec
   Value: 10096

   Key  : Analysis.Memory.CommitPeak.Mb
   Value: 88

   Key  : Bugcheck.Code.DumpHeader
   Value: 0x7e

   Key  : Bugcheck.Code.KiBugCheckData
   Value: 0x7e

   Key  : Bugcheck.Code.Register
   Value: 0x7e

   Key  : WER.OS.Branch
   Value: winblue_ltsb_escrow

   Key  : WER.OS.Timestamp
   Value: 2022-06-22T17:47:00Z

   Key  : WER.OS.Version
   Value: 8.1.9600.20475


FILE_IN_CAB:  MEMORY.DMP

BUGCHECK_CODE:  7e

BUGCHECK_P1: ffffffffc0000005

BUGCHECK_P2: fffff800904c458f

BUGCHECK_P3: ffffd00021f23288

BUGCHECK_P4: ffffd00021f22aa0

EXCEPTION_RECORD:  ffffd00021f23288 -- (.exr 0xffffd00021f23288)
ExceptionAddress: fffff800904c458f (NDIS!ndisCreateStringStreamEntry+0x000000000000002f)
  ExceptionCode: c0000005 (Access violation)
 ExceptionFlags: 00000000
NumberParameters: 2
  Parameter[0]: 0000000000000000
  Parameter[1]: 0000000000000000
Attempt to read from address 0000000000000000

CONTEXT:  ffffd00021f22aa0 -- (.cxr 0xffffd00021f22aa0)
rax=ffffd00021f23790 rbx=0000000000000015 rcx=ffffd00021f23540
rdx=ffffe00121e66ad8 rsi=ffffd00021f23680 rdi=ffffd00021f23540
rip=fffff800904c458f rsp=ffffd00021f234c0 rbp=0000000000000000
r8=ffffd00021f23680  r9=0000000000000000 r10=0000000000000000
r11=ffffd00021f23790 r12=fffff800904b1950 r13=ffffe0011e17c040
r14=ffffe00121e66990 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
NDIS!ndisCreateStringStreamEntry+0x2f:
fffff800`904c458f 66413929        cmp     word ptr [r9],bp ds:002b:00000000`00000000=????
Resetting default scope

PROCESS_NAME:  System

READ_ADDRESS:  0000000000000000 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000000

EXCEPTION_STR:  0xc0000005

STACK_TEXT:  
ffffd000`21f234c0 fffff800`904c46a1     : 00000000`00000015 00000000`00000002 ffffe001`21e66ad8 ffffe001`2a1eea98 : NDIS!ndisCreateStringStreamEntry+0x2f
ffffd000`21f234f0 fffff800`9043faa8     : ffffe001`21e66900 ffffd000`21f239d0 fffff800`904b2e00 ffffd000`21f239d0 : NDIS!ndisSqmLogDriverVersion+0xb9
ffffd000`21f238d0 fffff802`1de6a15f     : fffff800`9043f958 fffff800`904b2ea8 00000000`00000000 fffff802`1e0ad4c0 : NDIS!ndisSqmTimerWorkerRoutine+0x150
ffffd000`21f23b50 fffff802`1dee27aa     : ffffc000`000a0000 ffffd000`20fc0180 00000000`00000080 ffffe001`1d3968c0 : nt!ExpWorkerThread+0x69f
ffffd000`21f23c00 fffff802`1df57f66     : ffffd000`20fc0180 ffffe001`1e17c040 ffffd000`20fd0680 00000000`00000000 : nt!PspSystemThreadStartup+0x18a
ffffd000`21f23c60 00000000`00000000     : ffffd000`21f24000 ffffd000`21f1e000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16


SYMBOL_NAME:  NDIS!ndisCreateStringStreamEntry+2f

MODULE_NAME: NDIS

IMAGE_NAME:  NDIS.SYS

STACK_COMMAND:  .cxr 0xffffd00021f22aa0 ; kb

BUCKET_ID_FUNC_OFFSET:  2f

FAILURE_BUCKET_ID:  AV_NDIS!ndisCreateStringStreamEntry

OS_VERSION:  8.1.9600.20475

BUILDLAB_STR:  winblue_ltsb_escrow

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8.1

FAILURE_ID_HASH:  {a6009a6f-0469-bc4c-27f7-a8fa2f293092}

Followup:     MachineOwner
---------

Edited August 7, 2022 by Koldov

[Koldov]

Just a random thought....

 

Looking through the MEMORY.DMP it notes NDIS.SYS and a lot of Google results suggest NIC drivers need updating... but it has been solid for so long and any number of reboots before this happened...

 

Then I saw a thread somewhere about NDIS.SYS causing BSOD and the answer was about the following:

 

Replace:

WinPcap to npcap

 

Then it hit me, the only change I made to the server yesterday was updating Wireshark (which is probably not a good idea on a prod server but I was trying to deal with another error about NTLM being blocked stopping 'scan to folder').....

 

That also had references to WinPcap and npcap during the install!

 

Here's an old GitHub thread with references:

 

https://github.com/nmap/npcap/issues/565

 

Specifically:

 

"Bluescreen (ndis.sys) on Windows 8.1 with npcap 1,60"

 

"Hi, have the same issue. Fails in ndisCreateStringStreamEntry."

 

"I have the same issue. Npcap 1.60 and Windows Server 2012 R2. I will send you a minidump. Oddly, this problem only started approximately 1 day after initial installation.

 

Without Driver Verifier: The machine will bluescreen within a few minutes after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with a reference to ndis.sys.

With Driver Verifier: The machine will bluescreen within a few seconds after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with no reference to any driver.

 

I can confirm that "Raw 802.11 Packet Capture Support" seems to be the problem, as everything seems to work fine after I reinstalled without that option."

Edited August 7, 2022 by Koldov

[georgeescott]

Just a random thought....

 

Looking through the MEMORY.DMP it notes NDIS.SYS and a lot of Google results suggest NIC drivers need updating... but it has been solid for so long and any number of reboots before this happened...

 

Then I saw a thread somewhere about NDIS.SYS causing BSOD and the answer was about the following:

 

Replace:

WinPcap to npcap

 

Then it hit me, the only change I made to the server yesterday was updating Wireshark (which is probably not a good idea on a prod server but I was trying to deal with another error about NTLM being blocked stopping 'scan to folder').....

 

That also had references to WinPcap and npcap during the install!

 

Here's an old GitHub thread with references:

 

https://github.com/nmap/npcap/issues/565

 

Specifically:

 

"Bluescreen (ndis.sys) on Windows 8.1 with npcap 1,60"

 

"Hi, have the same issue. Fails in ndisCreateStringStreamEntry."

 

"I have the same issue. Npcap 1.60 and Windows Server 2012 R2. I will send you a minidump. Oddly, this problem only started approximately 1 day after initial installation.

 

Without Driver Verifier: The machine will bluescreen within a few minutes after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with a reference to ndis.sys.

With Driver Verifier: The machine will bluescreen within a few seconds after rebooting with a SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error with no reference to any driver.

 

I can confirm that "Raw 802.11 Packet Capture Support" seems to be the problem, as everything seems to work fine after I reinstalled without that option."

 

 

Nice find! This sounds exactly like what it could be. A reboot every 6 minutes. Hopefully you can modify the install within 6 minutes to remove the "Raw 802.11 Packet Capture Support" or remove WireShark/Npcap and the reboots will stop. Safe mode probably won't load this driver so you could do it then too.

 

If you were troubleshooting a VM NTLM issues, You can always use Hyper-V Port Mirroring to capture packets from one VM to another running WireShark rather than installing it on the host.

[Koldov]

Turns out that seemed to be the issue (I completely uninstalled Wireshark), I've always been vary wary of installing 3rd party software on my servers (and now I'll be even more so).

 

Safe mode didn't stop the boot loop unfortunately (but it did kill any chance of using RDP), luckily I live quite close so went in and did it.

 

It was just NTLM in general I was looking at, as a setting I had to deny it (and which I'd had set for ages) obviously wasn't applying with our old 2012R2 DCs, now we've moved to 2019 it seems to have kicked in and killed scan to folder.

[3s-gtech]

You can put third-party software on your servers, but your Hyper-V hosts should have the absolute minimum on them - backup mechanism, maybe UPS software, little else - and this is a great case study!

[Koldov]

this is a great case study!

 

Glad I could be of service....