Jcx500 Posted January 12, 2022 Posted January 12, 2022 From Reddit : Looks KB5009557 (2019) and KB5009555 (2022) are causing something to fail on domain controllers, which then keep rebooting every few minutes. 1
Koldov Posted January 12, 2022 Posted January 12, 2022 (edited) KB5009624 Breaks 2012R2 HyperV https://www.reddit.com/r/msp/comments/s269qy/kb5009624_breaks_2012r2_hyperv/ That was close! Sitting at home but on a remote session to update all the servers 'out of hours'... Then I notice the time is quite close to the back-up running so I decide not to do the Hyper-V host tonight! Edited January 12, 2022 by Koldov
3s-gtech Posted January 12, 2022 Posted January 12, 2022 Microsoft: “here’s this month’s updates!” WSUS: “I’ll just nope that for you. You’re welcome”
TwistedHelixis Posted January 12, 2022 Posted January 12, 2022 I have my server updates auto installed from the 24th of each month. How do we know if this update has been fixed? Might it get pulled and replaced or something. Always been curious, if I block this update, does that cause any issues for future updates? What I mean is, surely this update is included in the next update anyway, so the problem will come back then.
free780 Posted January 12, 2022 Posted January 12, 2022 Microsoft: “here’s this month’s updates!” WSUS: “I’ll just nope that for you. You’re welcome” 2 Week deadline under Cyber Essentials [emoji16] Why are people still running 2012 R2?
Jcx500 Posted January 13, 2022 Author Posted January 13, 2022 2 Week deadline under Cyber Essentials [emoji16] Why are people still running 2012 R2? I know a few places what are , main reason is cause it’s still supported by MS
3s-gtech Posted January 13, 2022 Posted January 13, 2022 Because it’s still in (extended) support, still does the job and is a good server OS (with a bad UI). I still have 1 or 2 Hyper-V servers on it, as well as a few service VMs. 2
kennysarmy Posted January 13, 2022 Posted January 13, 2022 Can't see of any reported issues with VMWare - is this bad patch only linked to HyperV ?
jthompson Posted January 13, 2022 Posted January 13, 2022 (edited) Regarding KB5009557 (2022-01 CU for Server 2019), some of our servers have installed it successfully, some are yet to try. One of our DCs rebooted overnight but is showing KB5009557 as having failed. Thankfully not boot looping. The .NET CU installed successfully on that one, though. All our servers are 2019. It took an age to install when I was doing a couple manually yesterday, but that could just be down to some coincidental issues in our environment at present. Not seeing any issues with the Win10 CU. Really don't need a wonky update messing with the servers at the minute. Edited January 13, 2022 by jthompson
Koldov Posted January 13, 2022 Posted January 13, 2022 (edited) Well, unfortunately the one DC that I did update was our SIMS server... Being very slow to log in to a remote session and nobody can access SIMS or FMS... Edit: I'm in now but lots of errors to work through and 'Shutdown Event Tracker' has come up asking why it shutdown unexpectedly...? Looking through the event logs it seems to have shutdown at 08:17, 08:28, 09:25 this morning.... so, not exactly boot looping I guess, but....? "A critical system process, C:\Windows\system32\lsass.exe, failed with status code c0000005. The machine must now be restarted." "The process wininit.exe has initiated the restart of computer SERVER on behalf of user for the following reason: No title for this reason could be found Reason Code: 0x50006 Shutdown Type: restart Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart." Strangely enough it doesn't seem to have been doing it all night though... Once last night for the updates and then randomly those 3 this morning... Edited January 13, 2022 by Koldov
mshakespeare Posted January 13, 2022 Posted January 13, 2022 I've had this same issue this morning. Two domain controller running Windows Server 2019 stuck in an endless loop of restarting after one minute of being on. I have a VMware infrastructure so have managed to fire up the replicas for now to get everyone back online. 1
Scrumpy Posted January 13, 2022 Posted January 13, 2022 Hi, 2012 R2 host running a few VM's here with the VM's failing to start this morning after last nights update - "hypervisor not running". Stumped at first what the problem was but a bit of searching led me to this - https://www.jordanmalcolm.com/virtual-machine-could-not-be-started-because-the-hypervisor-is-not-running/ The host had kb:5009624 installed (which I removed) but not kb:5009595. Once the update was removed and the server rebooted, all was fine. Hope this helps someone.
Koldov Posted January 13, 2022 Posted January 13, 2022 I've noticed that just before the reboots event viewer records this: Application popup: lsass.exe - Application Error : The instruction at 0xfbb12663 referenced memory at 0x00000058. The memory could not be read. Click on OK to terminate the program Click on CANCEL to debug the program Fairly generic and as usual the internet is therefore leading me down a lot of rabbit holes (a lot of it is quite old info and ends up with suggesting some years old hotfix)... from Virtual Machines, LDAP queries, DNS, changing a user name or permissions,
kennysarmy Posted January 13, 2022 Posted January 13, 2022 I've noticed that just before the reboots event viewer records this: Application popup: lsass.exe - Application Error : The instruction at 0xfbb12663 referenced memory at 0x00000058. The memory could not be read. Click on OK to terminate the program Click on CANCEL to debug the program Fairly generic and as usual the internet is therefore leading me down a lot of rabbit holes (a lot of it is quite old info and ends up with suggesting some years old hotfix)... from Virtual Machines, LDAP queries, DNS, changing a user name or permissions, Don't you just need to uninstall the update?
Koldov Posted January 13, 2022 Posted January 13, 2022 (edited) 2022-01 Security Monthly Quality Rollup for Windows Server 2012 R2 for x64-based Systems (KB5009624) This is the January Monthly Quality Rollup for 2012R2. KB5009595 is the Security-only update. Some info here (lots of adverts though): https://www.kiratas.com/security-updates-from-january-2022-with-massive-collateral-damage-in-windows-3/ Main info: The process lsass.exe (or wininit.exe) triggers a cyclical restart of the affected Windows server on affected machines because a Access conflict generates the error 0xc0000005. The interval at which these restarts are forced can be up to 15 minutes. At first only Windows Server 2012 R2 in connection with domain controllers was mentioned, but now there are also reports that domain controllers (DC) under Windows Server 2016, 2019 and 2022 are affected. The problem does not occur in all scenarios, the exact conditions that trigger the problem are not yet known. Edited January 13, 2022 by Koldov
Koldov Posted January 13, 2022 Posted January 13, 2022 Don't you just need to uninstall the update? Yes probably, but you know... DC & SIMS server in the middle of the day... hasn't rebooted since the last one at 09:25 this morning, so it doesn't appear to be in any kind of fatal boot loop or anything. Was hoping it might have had a setting or something that would stop the lsass.exe crashing and restarting the server... Is everybody doing 'wusa /uninstall /kb:5009624 /quiet' and will that avoid a restart? Or go through the 'Installed Updates > Uninstall' in Control Panel and do a restart?
Duranis Posted January 13, 2022 Posted January 13, 2022 So it's not just me! Had this come up yesterday after I had got home remoted in and saw the LSASS.exe errors and a ton of of errors of services failing to start, spent some time looking up the errors but didn't see anything relevant at the time. Assumed that it was one of the windows updates though considering when it started. Came in this morning, uninstalled the updates, restarted and everything is back to normal again. Regarding why am I still using 2012 R2? I don't have the budget to change it and because of how things are set up here it will mean a massive change (and cost) to switch to anything else now. I have no way to justify the cost and disruption to replace something that is "working fine". In the last year pretty much every "major" issue I have had to deal with has been due to a windows update. I have debated with myself over delaying installs but then I worry that I'm leaving security holes open.
LeMarchand Posted January 13, 2022 Posted January 13, 2022 Also problems with Hypervisors: http://www.edugeek.net/forums/windows-server-2012/225786-dell-server-raid-problem.html#post1935046
paulkerton Posted January 13, 2022 Posted January 13, 2022 I find it interesting that the attitude towards Windows bad updates is very different from other OS vendors. If this was a MacOS or Chrome problem there would be way more uproar. Maybe Microsoft's got people conditioned to be used to updates going wrong?
Koldov Posted January 13, 2022 Posted January 13, 2022 (edited) I find it interesting that the attitude towards Windows bad updates is very different from other OS vendors. If this was a MacOS or Chrome problem there would be way more uproar. Maybe Microsoft's got people conditioned to be used to updates going wrong? Good question, but I'm not sure why you believe people aren't outraged... The other question is what is a large % of the world's infrastructure/business supposed to do about it exactly? Start using Macs...? I'm not sure if MS have ever been quoted as saying 'it just works" though... Apple do have an advantage that a great deal of their ecosphere (technosphere) is run/developed/maintained/controlled by them on their hardware. MS certainly don't have that luxury as a lot of their products are shoehorned into unsuitable systems, run in configurations that are 'not recommended' and pulled apart by all sorts of 'unqualified' users etc. This is NOT a defence of MS as such and yes I agree we are conditioned to expect issues with every update (but it is certainly the last one I will be doing without a good week or so scouring the forums first)... Edited January 13, 2022 by Koldov
paulkerton Posted January 13, 2022 Posted January 13, 2022 The other question is what is a large % of the world's infrastructure/business supposed to do about it exactly? Start using Macs...? Demand better? That's the resignation to inevitable I'm talking about. Microsoft have done a fantastic job of setting up the expectations.
mullet_man Posted January 13, 2022 Posted January 13, 2022 Were on 2016 here and updates are due to install on the 19th, am guessing they will get pulled before then these updates.
Koldov Posted January 13, 2022 Posted January 13, 2022 (edited) Demand better? That's the resignation to inevitable I'm talking about. Microsoft have done a fantastic job of setting up the expectations. Well, yes... don't get me wrong, to a greater extent I do agree with you (nobody wants updates that grind our systems to a halt) but I'm not sure how every Windows user in the world could get together and hold Bill accountable though to demand better. I still maintain though that I really can't imagine that every update can be tested on every (even basic) configuration, there is always going to be an issue for someone - I mean even if this one is admittedly pretty big, I have got servers that are unaffected and so will a lot of sysadmins. Were on 2016 here and updates are due to install on the 19th, am hoping they will get pulled before then these updates. FTFY... This is what we hoped every month about the Print Nightmare updates though! Anyway, it is definitely a DC thing for us I think (managed not to install it on the Virtual Server Host so that's one good thing), my DC has just crashed again so uninstalling the update and taking the flak for the reboot... However, the lowly print server is (with update) still happily chugging away like a trooper (no restarts). EDIT: The unfortunate thing is that I thought I was being very clever installing the updates on my test VMs (just really checking I could still print more than anything else) and was happy that they seemed to work ok. How could I have known that the test VMs also need to be DCs and have a VM themselves to fully test for issues... Edited January 13, 2022 by Koldov
jonnykewell1 Posted January 13, 2022 Posted January 13, 2022 Has it been pulled yet, there's no way that this will happen on every single 2019 DC so it must be in conjunction with something else that causes it to happen surely?
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now