Looks like January windows updates are causing all sorts of issues..

[Jcx500]

 

From Reddit :

Looks KB5009557 (2019) and KB5009555 (2022) are causing something to fail on domain controllers, which then keep rebooting every few minutes.

[Koldov]

KB5009624 Breaks 2012R2 HyperV

 

https://www.reddit.com/r/msp/comments/s269qy/kb5009624_breaks_2012r2_hyperv/

 

That was close!

 

Sitting at home but on a remote session to update all the servers 'out of hours'...

 

Then I notice the time is quite close to the back-up running so I decide not to do the Hyper-V host tonight!

Edited January 12, 2022 by Koldov

[3s-gtech]

Microsoft: “here’s this month’s updates!”

WSUS: “I’ll just nope that for you. You’re welcome”

[TwistedHelixis]

I have my server updates auto installed from the 24th of each month.

 

How do we know if this update has been fixed? Might it get pulled and replaced or something.

 

Always been curious, if I block this update, does that cause any issues for future updates? What I mean is, surely this update is included in the next update anyway, so the problem will come back then.

[free780]

Microsoft: “here’s this month’s updates!”

WSUS: “I’ll just nope that for you. You’re welcome”

 

2 Week deadline under Cyber Essentials [emoji16]

 

Why are people still running 2012 R2?

[Jcx500]

2 Week deadline under Cyber Essentials [emoji16]

 

Why are people still running 2012 R2?

 

I know a few places what are , main reason is cause it’s still supported by MS

[caffrey]

It's good til October 2023

[3s-gtech]

Because it’s still in (extended) support, still does the job and is a good server OS (with a bad UI). I still have 1 or 2 Hyper-V servers on it, as well as a few service VMs.

[kennysarmy]

Can't see of any reported issues with VMWare - is this bad patch only linked to HyperV ?

[jthompson]

Regarding KB5009557 (2022-01 CU for Server 2019), some of our servers have installed it successfully, some are yet to try. One of our DCs rebooted overnight but is showing KB5009557 as having failed. Thankfully not boot looping. The .NET CU installed successfully on that one, though.

 

All our servers are 2019. It took an age to install when I was doing a couple manually yesterday, but that could just be down to some coincidental issues in our environment at present. Not seeing any issues with the Win10 CU.

 

Really don't need a wonky update messing with the servers at the minute.

Edited January 13, 2022 by jthompson

[Koldov]

Well, unfortunately the one DC that I did update was our SIMS server...

 

Being very slow to log in to a remote session and nobody can access SIMS or FMS...

 

Edit: I'm in now but lots of errors to work through and 'Shutdown Event Tracker' has come up asking why it shutdown unexpectedly...?

 

Looking through the event logs it seems to have shutdown at 08:17, 08:28, 09:25 this morning.... so, not exactly boot looping I guess, but....?

 

"A critical system process, C:\Windows\system32\lsass.exe, failed with status code c0000005. The machine must now be restarted."

 

"The process wininit.exe has initiated the restart of computer SERVER on behalf of user for the following reason: No title for this reason could be found

Reason Code: 0x50006

Shutdown Type: restart

Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart."

 

Strangely enough it doesn't seem to have been doing it all night though... Once last night for the updates and then randomly those 3 this morning...

Edited January 13, 2022 by Koldov

[mshakespeare]

I've had this same issue this morning. Two domain controller running Windows Server 2019 stuck in an endless loop of restarting after one minute of being on. I have a VMware infrastructure so have managed to fire up the replicas for now to get everyone back online.

[Scrumpy]

Hi,

 

2012 R2 host running a few VM's here with the VM's failing to start this morning after last nights update - "hypervisor not running".

 

Stumped at first what the problem was but a bit of searching led me to this - https://www.jordanmalcolm.com/virtual-machine-could-not-be-started-because-the-hypervisor-is-not-running/

 

The host had kb:5009624 installed (which I removed) but not kb:5009595. Once the update was removed and the server rebooted, all was fine.

 

Hope this helps someone.

[Koldov]

I've noticed that just before the reboots event viewer records this:

 

Application popup: lsass.exe - Application Error : The instruction at 0xfbb12663 referenced memory at 0x00000058. The memory could not be read.

 

Click on OK to terminate the program

Click on CANCEL to debug the program

 

Fairly generic and as usual the internet is therefore leading me down a lot of rabbit holes (a lot of it is quite old info and ends up with suggesting some years old hotfix)... from Virtual Machines, LDAP queries, DNS, changing a user name or permissions,

[kennysarmy]

I've noticed that just before the reboots event viewer records this:

 

Application popup: lsass.exe - Application Error : The instruction at 0xfbb12663 referenced memory at 0x00000058. The memory could not be read.

 

Click on OK to terminate the program

Click on CANCEL to debug the program

 

Fairly generic and as usual the internet is therefore leading me down a lot of rabbit holes (a lot of it is quite old info and ends up with suggesting some years old hotfix)... from Virtual Machines, LDAP queries, DNS, changing a user name or permissions,

 

Don't you just need to uninstall the update?

[Koldov]

2022-01 Security Monthly Quality Rollup for Windows Server 2012 R2 for x64-based Systems (KB5009624)

 

This is the January Monthly Quality Rollup for 2012R2.

 

KB5009595 is the Security-only update.

 

Some info here (lots of adverts though):

 

https://www.kiratas.com/security-updates-from-january-2022-with-massive-collateral-damage-in-windows-3/

 

Main info:

 

The process lsass.exe (or wininit.exe) triggers a cyclical restart of the affected Windows server on affected machines because a Access conflict generates the error 0xc0000005. The interval at which these restarts are forced can be up to 15 minutes. At first only Windows Server 2012 R2 in connection with domain controllers was mentioned, but now there are also reports that domain controllers (DC) under Windows Server 2016, 2019 and 2022 are affected. The problem does not occur in all scenarios, the exact conditions that trigger the problem are not yet known.

Edited January 13, 2022 by Koldov

[Koldov]

Don't you just need to uninstall the update?

 

Yes probably, but you know... DC & SIMS server in the middle of the day... hasn't rebooted since the last one at 09:25 this morning, so it doesn't appear to be in any kind of fatal boot loop or anything. Was hoping it might have had a setting or something that would stop the lsass.exe crashing and restarting the server...

 

Is everybody doing 'wusa /uninstall /kb:5009624 /quiet' and will that avoid a restart?

 

Or go through the 'Installed Updates > Uninstall' in Control Panel and do a restart?

[Duranis]

So it's not just me!

 

Had this come up yesterday after I had got home remoted in and saw the LSASS.exe errors and a ton of of errors of services failing to start, spent some time looking up the errors but didn't see anything relevant at the time. Assumed that it was one of the windows updates though considering when it started.

 

Came in this morning, uninstalled the updates, restarted and everything is back to normal again.

 

Regarding why am I still using 2012 R2? I don't have the budget to change it and because of how things are set up here it will mean a massive change (and cost) to switch to anything else now. I have no way to justify the cost and disruption to replace something that is "working fine".

 

In the last year pretty much every "major" issue I have had to deal with has been due to a windows update. I have debated with myself over delaying installs but then I worry that I'm leaving security holes open.

[LeMarchand]

Also problems with Hypervisors: http://www.edugeek.net/forums/windows-server-2012/225786-dell-server-raid-problem.html#post1935046

[paulkerton]

I find it interesting that the attitude towards Windows bad updates is very different from other OS vendors. If this was a MacOS or Chrome problem there would be way more uproar. Maybe Microsoft's got people conditioned to be used to updates going wrong?

[Koldov]

I find it interesting that the attitude towards Windows bad updates is very different from other OS vendors. If this was a MacOS or Chrome problem there would be way more uproar. Maybe Microsoft's got people conditioned to be used to updates going wrong?

 

Good question, but I'm not sure why you believe people aren't outraged... The other question is what is a large % of the world's infrastructure/business supposed to do about it exactly? Start using Macs...?

 

I'm not sure if MS have ever been quoted as saying 'it just works" though... Apple do have an advantage that a great deal of their ecosphere (technosphere) is run/developed/maintained/controlled by them on their hardware. MS certainly don't have that luxury as a lot of their products are shoehorned into unsuitable systems, run in configurations that are 'not recommended' and pulled apart by all sorts of 'unqualified' users etc.

 

This is NOT a defence of MS as such and yes I agree we are conditioned to expect issues with every update (but it is certainly the last one I will be doing without a good week or so scouring the forums first)...

Edited January 13, 2022 by Koldov

[paulkerton]

The other question is what is a large % of the world's infrastructure/business supposed to do about it exactly? Start using Macs...?

 

Demand better?

That's the resignation to inevitable I'm talking about. Microsoft have done a fantastic job of setting up the expectations.

[mullet_man]

Were on 2016 here and updates are due to install on the 19th, am guessing they will get pulled before then these updates.

[Koldov]

Demand better?

That's the resignation to inevitable I'm talking about. Microsoft have done a fantastic job of setting up the expectations.

 

Well, yes... don't get me wrong, to a greater extent I do agree with you (nobody wants updates that grind our systems to a halt) but I'm not sure how every Windows user in the world could get together and hold Bill accountable though to demand better. I still maintain though that I really can't imagine that every update can be tested on every (even basic) configuration, there is always going to be an issue for someone - I mean even if this one is admittedly pretty big, I have got servers that are unaffected and so will a lot of sysadmins.

 

Were on 2016 here and updates are due to install on the 19th, am hoping they will get pulled before then these updates.

 

FTFY... This is what we hoped every month about the Print Nightmare updates though!

 

Anyway, it is definitely a DC thing for us I think (managed not to install it on the Virtual Server Host so that's one good thing), my DC has just crashed again so uninstalling the update and taking the flak for the reboot... However, the lowly print server is (with update) still happily chugging away like a trooper (no restarts).

 

EDIT: The unfortunate thing is that I thought I was being very clever installing the updates on my test VMs (just really checking I could still print more than anything else) and was happy that they seemed to work ok. How could I have known that the test VMs also need to be DCs and have a VM themselves to fully test for issues...

Edited January 13, 2022 by Koldov

[jonnykewell1]

Has it been pulled yet, there's no way that this will happen on every single 2019 DC so it must be in conjunction with something else that causes it to happen surely?