Updating servers

[harry]

Hi Guys, not long now.....1 more week 'til the holidays start.

 

Just a daft question, we update our servers monthly and log it on a spreadsheet that it was done. Do you document what updates were applied? Do you document? Or do you update and hope that the server comes back up after an update and if it doesn't safe mode/uninstall the last updates?

 

Cheers

Harry

[synaesthesia]

Pretty much the same - I log the build number changes between each. There's too many individual update KB's to log!

[NegativeKillDeath]

Hi Guys, not long now.....1 more week 'til the holidays start.

 

Just a daft question, we update our servers monthly and log it on a spreadsheet that it was done. Do you document what updates were applied? Do you document? Or do you update and hope that the server comes back up after an update and if it doesn't safe mode/uninstall the last updates?

 

Cheers

Harry

 

Record in change log which servers are updated. Snapshot the servers VM so that if it does fail to come back up we will revert at the end of the maintenance window.

[mrbios]

MECM automatic deployment rule setup for monthly updates here with various servers staggered. I delay by a few days to identify any issue updates and remove them if need be...Done by setting a "Custom severity: none" on the rule that installs them, then if i need to negate an update i'll just apply a custom severity of low to it. (Might be a better way to do that, but i've been doing it this way for years)

 

Personally in the process of setting up a lab environment to do Veeam restores into and do testing pre-deployment of things like updates, that's on my to-do list for 2025 anyway.

Edited December 13, 2024 by mrbios

[dmj]

Infrastructure as code, for updates gives us:

 

• A testing environment with the same server config gets deployed to test the updates prior to changing the live server (it's all VM's, so why not as it costs next to nothing to do this)
  
• An audit trail for who made the changes and when
  
• A review process by other techs, both to authorise the changes and inform others that the change is being made
  
• Deployment frequency stats
  
• A rollback method for bad changes.
  

[TheRobins]

Check that Veeam has ran the night before, then click update and see what happens.

 

If it goes wrong just restore that VM

 

Not had one gone wrong yet all running on Server 19

[Fazza]

update and hope that the server comes back up

 

Yep, that. Sort of...

 

We use PANDA AD360 (or whatever it's now called) for patch management every 2 weeks and that has a log of what it has done.

[Kitkatninja]

You'll never get CSE by doing that once a month

 

For us our patch management system keeps the logs of what has been installed and updated. If something goes wrong, we can remove the update if the server/machine is online. If not online but on, we can manually remove. If it doesn't come back up, then it's restore from backups.

Edited December 13, 2024 by Kitkatninja

[harry]

Many thanks to everyone, it's appreciated. Nice to learn that everyone has their own ways to do things.

 

Cheers

h

[Bedders]

For us it's relatively straightforward. Some of the servers we allow to install automatically overnight, less important ones that I don't mind if they fall over. Stuff like our monitoring server, one of our Domain Controller's, or our print server.

 

As for the others the process every weekend after Patch Tuesday is:

 

Check backups were completed last night (more importantly, that auto verification was successful).

Check for Updates on each server.

Screenshot all KB numbers that are about to be installed and save image to central location.

If it's not a Domain Controller or Exchange server, snapshot it on the host.

Click install now and pray.

Test the services on it after install. If nothing broke, remove the snapshot.

 

I've got a full tick list along with easy tests for the services for each of these 'special servers', to pass it over to my new colleague in a few weeks' time once he's ready.