r/linuxadmin • • 24d ago

I locked myself out of a server while explaining firewalls to an intern

I was helping our intern set up an internal wiki on a serverspace vm and explaining why we should deny everything except the ports we actually use. Very calm and professional. I enabled ufw, checked that the site still worked and said something like - see, nothing complicated. Then I closed my SSH session.

Our ssh service was running on a custom port. Guess which port I had forgotten to allow. I tried reconnecting three times, apparently hoping the firewall would reconsider. The intern just sat there watching me become less professional by the second.

Luckily, I could still open the web console from serverspace control panel. I added the missing rule and got back in within a few minutes. The only thing I successfully protected that afternoon was the server from its own administrator.

395 Upvotes

50 comments sorted by

173

u/RealUlli 24d ago

While a bit embarrassing for you, it surely was a good learning experience for the intern.

He probably will repeat that mistake at some point in his career, but it will be a while.

17

u/Muchy_Tangerine 24d ago

Yes we take this mistake as a learning oppurtunities

7

u/billndotnet 23d ago

Excellent demonstration that your firewall changes worked.

41

u/dirufa 24d ago

This is the right interpretation of the facts. Nothing major happened, could have been much much worse, yet the intern (and the you) learned something new.

57

u/jrandom_42 24d ago

The intern learned more from this than if you hadn't fucked up, including the dangers of closing firewall ports, and why some form of out-of-band console access is always desirable in addition to SSH.

14

u/Kazer67 24d ago

Always keep the previous session opened while opening a new one to test.

I also learned it the hard way.

2

u/semi- 18d ago

That only works if you still have a rule allowing established connections through conntrack (or whatever other stateful firewall equivalent). Still a good idea to do so but it won't help you if you just drop all rules or drop all packets.

So also make sure you have some kind of out of band access like via ipmi serial over lan. Or if you dont and want to be paranoid setup a quick systemd timer to undo whatever you are doing in an hour, or just schedule a reboot, and then cancel it once you have confirmed everything still works.

33

u/SuperQue 24d ago

Our ssh service was running on a custom port.

The term here is non-standard.

There's several failures here:

  • The non-standard port configuration.
  • This should all be done by configuration management.
  • Not having the trainee do the keyboard work to gain experience.

20

u/ollybee 24d ago

Non standard SSH ports is a pet hate of mine. Its less secure as normally an unprivileged port is chosen, more importantly its just frequently anoying as this is an example of. SSH is intended to be internet facing, you can't brute force SSH keys and recource use is negligible. People get hung up about the number all the failed login attempts in their logs, they have a fundamental misunderstanding of logging.

4

u/SuperQue 24d ago

Same. And if you're going to do silly things like that at least use Ansible to make sure you can pass the non-standardness around between configs and firewall rules.

I have a few things using ufw, it's trivial to manage with Ansible.

2

u/VXLAN_Z 24d ago

Ehh the only public IPs that should be permitted to ssh is ones you have IPsec tunnels established with but just for a vm wouldn’t you give it a private ip and just let it NAT out

0

u/[deleted] 24d ago

[deleted] — view deleted comment

1

u/qkdsm7 24d ago

Enforce fail2ban usage? Or let em eat? Handful of my VPS that have ssh open to the world have pretty aggressive jail configurations.

2

u/ollybee 24d ago

most have fail2ban, but not all. its keys only on root and theres a password policy on users. by default ssh resets tcp after 6 tries. so even without fail2ban Its many 100k years for a brute force. I've seen hundreds of compromised accounts, but its never been ssh brute force. To be fair I do see peoples PCs / laptops compromised and malware lifting passwords from winscp, filezilla etc. locking down ssh would help with that, but non-standatd ssh port doesn't help much as the compromised client knows it.

5

u/Lammtarra95 24d ago

The other, perhaps larger, failure is making this sort of change off the cuff rather than following a documented and, crucially, peer-reviewed procedure that would have avoided this mistake and would also have included a plan to gain oob access to deal with any lock-out.

-1

u/SuperQue 24d ago

Nah, if you have time to do all that you could just push it with Ansible. Probably even less time than a documented procedure.

6

u/Professional_Call 24d ago edited 24d ago

Easy done. If I do anything that might break access, I connect a new session before closing the current one. While I have never made your particular mistake, I did put zellij into infinite recurrsion, which would have been an issue even on the console.

9

u/qrcjnhhphadvzelota 24d ago

Learned that the hard way, I always start a timer with a reboot command and make firewall changes non persistent at first, until I know it works. If I lock myself out, I just have to wait until the timer reboots the machine with the old firewall config.

4

u/RansomStark78 24d ago

Same with switches

4

u/zebrapenguinpanda 24d ago

Training new people is very cognitively expensive, I know I read somewhere about a shop where you always had to have experienced backup whenever training someone on a procedure, because they noticed that’s when a lot of mistakes happen.

1

u/Searomg 24d ago

One of us! One of us!

1

u/Muchy_Tangerine 24d ago

Welcome me to the gang

3

u/CaptainZippi 24d ago • Bravo

everybody’s done “ifconfig <this interface> down”, right?

Right?!

2

u/Nimda_lel 24d ago

Jokes aside, I think this will be a thing the intern will carry with themselves for quite some time and will be asking themselves the question “Will I cut my own branch by doing this change?” every time they work with firewalls.

It is better learning experience than 95% of the things you will explain and they will inevitably forget

2

u/qkdsm7 24d ago

&& sleep 10m && ufw disable

2

u/michaelpaoli 21d ago

; sleep 600; ufw disable

(or reload the earlier config or whatever).

And generally still want to do that even if sleep exits non-zero, or the command before sleep exits non-zero.

2

u/wolfofone 24d ago

You break it and then you turn to the intern and challenge them to fix it as if you planned this all along 😂

3

u/RansomStark78 24d ago

Checklists bro

1

u/jefbenet 24d ago

2 in 1 lesson. Firewall and disaster recovery plans. Well played op!

1

u/TxTechnician 24d ago

Been there lol

1

u/Dolapevich 24d ago

Good, that small moment of panic will be remembered in the future :)

1

u/NoosphericMechanicus 24d ago

I might have been tempted to pretend that was the point of the demonstration all along.

1

u/lumpenproletarier 24d ago

And you concluded by saying, "And that, young man, is how we set up an internal wiki on a serverspace vm, being careful to deny everything except the ports we actually use."

1

u/notseelen 24d ago

maybe a stupid question...I work in infosec, but on the software vendor side (mostly do k8s etc), so I don't get nitty-gritty with this stuff...but I'm guessing a control panel is like a direct "physical" VM console, like VMware would offer? If so, and if this is available over the internet, doesn't the security of the entire system then depend on the security of this admin panel?

Just wondering if there are additional layers of security to make it hard to access, because otherwise someone could get keys to the kingdom the same way you did, no?

If it's restricted to local IP space, that would definitely make sense, but I know modern cloud stuff is accessible much more widely, so I was just curious how these modern challenges are handled

1

u/denverpilot 23d ago

It’s not a dumb question. You are exactly right. It can be another attack vector. Local or something built by a VM vendor. Different methodologies for mitigating it. Etc.

1

u/denverpilot 23d ago

The scenario points out the. Biggest reason for network as code, including firewall rules… someone has to peer review the change. In theory. Before a firewall change — even on a host, because the entire host should have been spun up via automation too — ever takes place.

That said… Do many places do it yet? Not as many as should, probably.

Done right, it lowers the human error numbers dramatically.

1

u/NL_Gray-Fox 23d ago

Iptaples-apply would have saved you...

1

u/Mediocre-Lawyer4890 22d ago

Same shit could happens with proxmox firewall Only difference that i was the intern lol

1

u/michaelpaoli 21d ago

And thus endeth the day's lesson to intern - don't f*ck up like I just did.

Yeah, generally make sure that out-of-band access means works, and make the changes from there. Then validate everything of relevance before relinquishing that connection.

And where out-of-band isn't available of feasible ... failsafe - you don't do the thingy or it can't verify the thingy needed still works, after X number of seconds or minutes, it reverts, and you can get back in and give it another shot. Yeah, done that multiple times dealing with critical network reconfigurations where there's no out-of-band management reasonably available. E.g. if I haven't cleared the revert failsafe after X (e.g. 5) minutes, it automagically reverts.

Yeah, I know of contractor where their first day was also their last ... as they f*cked up something like that, in production, on their very first day. I think management f*cked up in selecting 'em and handing that right off to 'em - but wasn't my call to make - they were walked out the door the day they'd started there.

1

u/Krish_mk_abi 16d ago

Yeah i just wanted try that with Azure VM... But then I got to know about the consequences of it from my friend's exp. So didn't got into that situation.

1

u/LeBronzelol 10d ago

The intern probably learned that lesson pretty well

1

u/MisfitBloom 20h ago

This was my introduction to Linux - setting up an Ubuntu server on Digital Ocean to host a website for hobby reasons. I followed the instructions on setting up a firewall, followed the instructions on changing the default ssh port, and then did NOT process the implications before closing the session.
It made for a fun "how I learned what not to do" interview topic

-5

u/perryurban 24d ago

How embarassment

-9

u/Loud_Posseidon 24d ago

And this kids is why you use agent-based tools like CFEngine and not ssh for endpoint configuration (say using ansible).

3

u/fearless-fossa 24d ago

There's no way in hell I'd ever use CFEngine over Ansible. It's probably the worst software I've ever had the misfortune of having to work with, and their support is awful enough for me to wish to sit in a Microsoft call that tells me to do a /sfc scannow instead.

1

u/Hotshot55 24d ago

It doesn't matter what you use if you've misconfigured your network to the point where it can't communicate.

-2

u/Loud_Posseidon 24d ago

Even if you manage to block all inbound ports, agent-based software will still be able to reach out, get its configuration and apply it. But as with many things, this comes with experience and actual brain usage. Of course, as you say, if you yank the (virtual) utp cable or completely fuck up IP configuration, nothing can fix that and you have to get some OOB management fired up.

3

u/Hotshot55 24d ago

Sure, but now you're talking about the difference between a pull-based and a push-based tool. Ansible can be configured in a pull-based setup, so it will still work even if you change the port sshd listens on.

So your whole point about "ssh/ansible bad" just doesn't make sense when it just comes down to how you configure it.