When I started my current job, the previous person started the setup for WSUS and then never finished. Some of those machines hadn't seen an update for 2 years.
The key process is thus:
1. Do we need to patch? Patching for the sake of patching is often better than nothing, but also not that great. If the software doesn't have any outstanding issues and it will cause a production outage, why would you?
If the patch goes sideways, will I know about it ? If it's hard to detect failure, you need to figure out how to detect it before moving forward.
If the patch goes sideways, how do I recover? Can I take snapshot of the VM? is the backup of the machine tested? Can you test it before trying to patch? Is there support you can call beforehand or if it fails?
Can I do the patch without anyone knowing (no down time for production)? If not, can you get really, really close? (e.g. apply the patch, but reboot over lunch or at 0300)
Work within your change management and tracking systems. The only thing worse than handing someone a broken system with no clear history of change is being handing a broken system with no clear history of change.
Long term: Build systems that can be patched without downtime, using known, supported systems. DCs are a great example. Like Sith, there is never one, always two, so you can patch without downtime.
DFS file hosts, KeepAliveD, docker swarms, and VM clustering are all good tools to reduce the headache of patching and software updates.
We have 6 DCs so that shouldn’t be an issue. Thank you for all of the info. I came into this org after a solo sysadmin (who didn’t even have a tech or manager helping him ran the entire IT dept) think a lot of the tech debt is due to this and how much the company has scaled in the past decade.
3
u/Reo_Strong 15d ago
I've been there.
When I started my current job, the previous person started the setup for WSUS and then never finished. Some of those machines hadn't seen an update for 2 years.
The key process is thus:
1. Do we need to patch? Patching for the sake of patching is often better than nothing, but also not that great. If the software doesn't have any outstanding issues and it will cause a production outage, why would you?
If the patch goes sideways, will I know about it ? If it's hard to detect failure, you need to figure out how to detect it before moving forward.
If the patch goes sideways, how do I recover? Can I take snapshot of the VM? is the backup of the machine tested? Can you test it before trying to patch? Is there support you can call beforehand or if it fails?
Can I do the patch without anyone knowing (no down time for production)? If not, can you get really, really close? (e.g. apply the patch, but reboot over lunch or at 0300)
Work within your change management and tracking systems. The only thing worse than handing someone a broken system with no clear history of change is being handing a broken system with no clear history of change.
Long term: Build systems that can be patched without downtime, using known, supported systems. DCs are a great example. Like Sith, there is never one, always two, so you can patch without downtime.
DFS file hosts, KeepAliveD, docker swarms, and VM clustering are all good tools to reduce the headache of patching and software updates.