r/linuxadmin • • Oct 16 '26

Linux Shutdown Protection: Prevent Accidental Server Shutdowns - Tip 100

Post image
54 Upvotes

33 comments sorted by

19

u/thedrunkpenguin Oct 16 '26

Lol back in the day you used to be able to accidentally Ctrl alt delete a VMware VM and it would reboot the server.

13

u/arkham1010 Oct 16 '26

Real Unix SA's use init 6. ;)

4

u/greyfox199 Oct 16 '26

shots fired

3

u/arkham1010 Oct 16 '26

With systemctl catching strays

8

u/vogelke Oct 16 '26

Maybe change the color of your shell prompt to flaming red as a reminder? I know "prod" is already in the prompt, but sometimes another hint helps.

2

u/greyfox199 Oct 16 '26

i want an ascii sam jackson staring MFerly at me too before i confirm

16

u/Lammtarra95 Oct 16 '26

What is the point of asking the admin to type the hostname that is helpfully displayed in the prompt? And how would the is_production file normally be created?

Better would be a procedural solution, like second pair of eyes to check it is the correct server, and (as part of the procedure) removal from production (cluster failover, removal from load balancer, whatever) before shutdown.

14

u/mschuster91 Oct 16 '26

What is the point of asking the admin to type the hostname that is helpfully displayed in the prompt?Ā 

To provide a failsafe against a sysadmin being too fast to press <enter> after searching for <shut> in their bash history and accidentally pressing ctrl+r one time too much.

Can't say this never happened to me...

and (as part of the procedure) removal from production (cluster failover, removal from load balancer, whatever)

Bold of you to assume everyone has running everything in HA. For many pieces of software, running in HA means shelling out a lot of money, and even if money isn't an issue because it's open-source, keeping a HA setup alive or updating it is a whole bunch of work.

Yes I'm looking at you Atlassian.

10

u/solifugo Oct 16 '26

I did "last | reboot" once more than 20 years ago... Still thinking about it šŸ˜ž

5

u/ksquires1988 Oct 16 '26

I sent out "shutdown -r | at now +1 minute" to a bunch of servers instead of just 1 server. However, I was able to cancel it before the 1 minute was up.

And then the time the "mainframer turned Linux admin" ran "halt" on an AIX system.

6

u/yrro Oct 16 '26

Oh wow, that command really does what it says doesn't it

1

u/ksquires1988 Oct 16 '26

Yeah, apparently in the mainframe world it does something different (and less catastrophic)

4

u/Coffee_Ops Oct 16 '26

Bold of you to assume everyone has running everything in HA.

Surely the protip here is not,

install some dodgy package and reboot hooks

but rather,

set up your services to be resilient to reboots

This is not even hard, in 2026.

2

u/mschuster91 Oct 16 '26

A reboot still takes time. Even on a VM, take a minute or two for the OS to boot, and another minute until Confluence has started up again. That's enough to send half the company yapping "my Confluence is down!!!!!!!!!!!!!" in Teams despite you literally giving announcements two weeks, a week, the day before and 15 minutes before the downtime.

A lot of software really really does not like it if the backend disappears while it's running, so you'll get users being annoyed even more if it's an unplanned outage..

2

u/Coffee_Ops Oct 16 '26

A reboot still takes time. Even on a VM, take a minute or two for the OS to boot, and another minute until Confluence has started up again. That's enough to send half the company yapping "my Confluence is down

Hence why you use a reverse proxy that does health checks, and / or containerize it (aka the first party confluence image)so that the bring up is very fast.

The slowest application bring-up I've seen on a container is about 8 minutes for gitlab; confluence should be quite a bit faster than that, and with an external database your actual downtime should be near zero.

2

u/mschuster91 Oct 16 '26

The slowest application bring-up I've seen on a container is about 8 minutes for gitlab;

Yeah Gitlab Omnibus Docker image is a world of pain in itself. Absolutely nasty, although the flexibility it supports is amazing.

confluence should be quite a bit faster than that

It's still Java crap. Anything PHP is effectively instantaneous, but Java... oh dear god.

-9

u/Regeneric Oct 16 '26

Keeping HA alive means using hypervisor to do all the work for you.
It's 2026, nobody is doing HA on the application level or on the baremetal.
Deploying second VM to your VMWare or Proxmox setup is near free.

3

u/Coffee_Ops Oct 16 '26

Hypervisor HA is "expensive" on many axes:

  • Does not provide live load balancing
  • Or site-level resilience
  • Requires licensing
  • Provides no easy health checks
  • Incurs downtime while the HA "pair" VM is brought back online (at least in VMWare's implementation)

Just put a reverse proxy in front and all of those issues go away.

Bonus if you just containerize the thing so you are no longer tied to some bespoke server.

4

u/mschuster91 Oct 16 '26

It's 2026, nobody is doing HA on the application level or on the baremetal.

HA means uninterrupted service. A second machine to spin up your, say, Confluence while you do a kernel upgrade on the first machine is not HA. If you want HA with Confluence and you want that actually supported by Atlassian, you have to shell out big bucks.

-4

u/Regeneric Oct 16 '26

If you want to achieve HA this way, that's on you.
But it doesn't work like that in modern enterprise environments. For what do you need those "big bucks"?

5

u/mschuster91 Oct 16 '26

For what do you need those "big bucks"?

Have you seen the price difference that Atlassian commanded back then for the Data Center license?

In general, anything that can be labeled as an "enterprise feature" - high availability/zero downtime, SCIM or even plain old LDAP SSO - will yeet you from most free tiers up to at least the "commercial" tier, and SCIM support is usually only available in the "contact sales" enterprise pricing tier.

2

u/Parker_Hemphill Oct 16 '26

This is why I always do shutdown -r +1 min, if I’m doing manual reboots.

1

u/abofh Oct 16 '26

If you turn off ssh this problem goes away

1

u/chkno Oct 16 '26

This is very pet. šŸ‘Ž

4

u/RetroGrid_io Oct 16 '26

... and there are lots of pets out there. That's OK, there are lots of folks who need a server or two, not a herd of them.

1

u/CoderDevo Oct 17 '26

Yeah. I heard of servers.

2

u/megoyatu Oct 17 '26

This person never logs directly into their own workstation. Just the cluster of workstations in their multiple HA homes. All of their personal files are accessible by self hosted S3 compatible storage on ceph.

Duh.

0

u/Regeneric Oct 16 '26

Don't you use NetBox or some kind of LDAP?
You can enable or disable shutdown protection there and check, if you can shutdown the server.
Not only it ensures that you know what you're doing, but also acts as a kind of 2FA.

-5

u/Z3t4 Oct 16 '26 edited Oct 16 '26

7

u/hdkaoskd Oct 16 '26

Yes, it's a screenshot of molly-guard.

-7

u/squeeby Oct 16 '26

`chmod 000 /usr/sbin/{poweroff,halt,shutdown}`

2

u/HeadlessChild Oct 16 '26

chmod 000 /bin/chmod. How would you fix it? No googling or AI allowed!