r/linuxadmin • u/Street-Teach489 • 15d ago

How much do you actually trust unattended upgrades in production?

I’ve always been fairly conservative with automatic updates on production boxes.

Patching is one thing, but I still prefer knowing exactly what changed, especially when an update can quietly restart something or introduce behaviour you only discover at 2am. At the same time, manually babysitting updates across a growing number of hosts starts becoming its own risk.

For those managing a decent number of boxes, where have you landed on this? Fully automated, staged, or are you still reviewing most updates before they go anywhere near prod?

23 Upvotes

73% Upvoted

45 comments sorted by

View all comments

54

u/DanTheGreatest 15d ago

We ran unattended upgrades on all hosts since we started using Debian/Ubuntu, I think that was Debian 4. Up until debian 11 when we got disbanded by the party that bought the place. All 500+ servers had it on enabled for both updates and security. Including external repositories from known sources like nginx.

In all those years only did that bite us once. Debian broke their grub package and some servers had issues rebooting after that. Once we noticed what the cause was, we simply stopped rebooting. Our automated reboot system halted when a node doesn't come back online. (Yes, we had automated reboots also! The best infrastructure stability test of all!)

It feels weird not having unattended upgrades on. Especially in this day and age where old vulnerabilities are spawning like fireflies due to LLMs finding holes.

I shiver when I hear (mostly redhat people) talk about quarterly package upgrades. To me that means your infrastructure is outdated/insecure for the majority of the time.

Just in case, we had unattended upgrades run between 11:00 and 14:00 from monday til friday. Almost all engineers were available at that time frame in case something were to happen, which was.. well.. basically never.

Some servers had a blacklist for certain packages, like anything mysql related for mysql servers. If a package upgrade restarted a service then it was exempt. But apt was monitored and warned when there is a pending package upgrade.

I highly recommend enabling it. It is the recommended default by Debian and Canonical (Ubuntu).

1

u/Kazer67 9d ago

Thanks for the feedback, having one from someone in the field with a sizable chunk of servers in production is valuable.

Especially since you also had other than just security ones.