Windows update caused unbootable servers

[howartp]

This is a note for closing the stable door after the horse has bolted, in case it affects anyone else.

 

Due to a new install of WSUS by our engineers as part of domain upgrade, some windows updates which I'd previously declined were pushed out to servers and PCs alike.

 

My finance and payroll server went down Wednesday night to finish installing the updates - but never came back up as something was corrupt. It booted to the windows system repair screen.

 

Skipping past the several hours trying to combine the previous days backup with the live data/databases on Wednesdays HDD (and currently waiting 24 hours since I raised a "critical - no functionality" ticket with Civica), I finally managed to revert the windows updates and boot the server back up about 2.30pm, minus the windows updates.

 

For anyone facing same issue, let it boot into the repair wizard that comes up (it'll be asking for keyboard language), and open the command prompt.

 

Determine which drive is currently your C:\Windows drive (it won't be C:, it's been either D: or G: on mine depending how many drives you have) by typing:

 

d:

dir

e:

dir

f:

dir

g:

dir

 

until you find the one with Windows on it.

 

Then the main command you need is (substitute F: for D: or G: etc):

 

dism /image:F: /cleanup-image /revertpendingactions

 

It might fail with error 5 (I think) about closing/finishing some revert procedures and tell you to retry in a few minutes. If you do retry, it will give you a different error. However, restarting the server at this stage will successfully enter into "Stage 1 of 3" reverting/rolling back upgrades, and eventually boot you back into Windows.

 

Hope this helps other users, now or in the future - I believe some users had issues with the August 2019 updates if March 2019 updates hadn't been installed prior.

 

Peter

[simpsonj]

Thanks for that, stored under in a wiki just in case!

 

Do you know which update was the culprit?

[howartp]

It's one of the following 6 updates:

 

KB4507456 2019-07 Security Only Quality Update

KB4474419 2019-09 Security

KB4099950 2018-04 Update

KB4507704 2019-07 Update

KB4503269 2019-06 Security Only Quality Update

KB3140245 Update

 

My suspicion is KB4474419 as that KB number seems to be re-released periodically with new content.

 

I've declined all 6 in our WSUS for now.

 

Peter

[psydii]

Good tips on troubleshoot this scenario thanks for sharing.

 

If anyone finds themselves in a similar hole without support, I recommend raising a ticket with Microsoft directly (you will need a credit card) they treat production server being down with the sort of response times you’d expect. I had one call back in less that 10 minutes! But usually it is within a couple of hours.

[googlemad]

If it is the same issue we had, there is a much simpler fix available (which also enables you then to actually install the updates rather than rolling them back)

 

Boot into the recovery environment (hope you know your local server administrator account password!), open a command prompt and type in the following command :-

 

bcdedit /Set {default} nointegritychecks ON

 

This should take a couple of seconds to execute and report success, then you can reboot your server successfully all being well...apply all 'Important' updates (at a minimum) until you're sure fully up to date, once you are sure you are up to date you can open a command prompt within Windows and reverse the command, no reboot required (this is of course an optional step, you could just leave no integrity checks on but probably a good idea to turn it back to off from a security point of view)

 

bcdedit /Set {default} nointegritychecks OFF

 

Now if anyone could fund my GoFundMe for the clean pair of underpants I had to acquire after Windows startup repair popping up on one of our main production SQL servers and Exchange servers it would be very much appreciated!

Edited March 2, 2020 by googlemad