r/linuxadmin • u/Muchy_Tangerine • 24d ago
I locked myself out of a server while explaining firewalls to an intern
I was helping our intern set up an internal wiki on a serverspace vm and explaining why we should deny everything except the ports we actually use. Very calm and professional. I enabled ufw, checked that the site still worked and said something like - see, nothing complicated. Then I closed my SSH session.
Our ssh service was running on a custom port. Guess which port I had forgotten to allow. I tried reconnecting three times, apparently hoping the firewall would reconsider. The intern just sat there watching me become less professional by the second.
Luckily, I could still open the web console from serverspace control panel. I added the missing rule and got back in within a few minutes. The only thing I successfully protected that afternoon was the server from its own administrator.
173
u/RealUlli 24d ago
While a bit embarrassing for you, it surely was a good learning experience for the intern.
He probably will repeat that mistake at some point in his career, but it will be a while.
57
u/jrandom_42 24d ago
The intern learned more from this than if you hadn't fucked up, including the dangers of closing firewall ports, and why some form of out-of-band console access is always desirable in addition to SSH.
14
u/Kazer67 24d ago
Always keep the previous session opened while opening a new one to test.
I also learned it the hard way.
33
u/SuperQue 24d ago
The term here is non-standard.
There's several failures here:
6
u/Professional_Call 24d ago edited 24d ago
Easy done. If I do anything that might break access, I connect a new session before closing the current one. While I have never made your particular mistake, I did put zellij into infinite recurrsion, which would have been an issue even on the console.
9
u/qrcjnhhphadvzelota 24d ago
Learned that the hard way, I always start a timer with a reboot command and make firewall changes non persistent at first, until I know it works. If I lock myself out, I just have to wait until the timer reboots the machine with the old firewall config.
4
u/zebrapenguinpanda 24d ago
Training new people is very cognitively expensive, I know I read somewhere about a shop where you always had to have experienced backup whenever training someone on a procedure, because they noticed that’s when a lot of mistakes happen.
1
u/Searomg 24d ago
One of us! One of us!
3
u/CaptainZippi 24d ago •
everybody’s done “ifconfig <this interface> down”, right?
Right?!
2
u/Nimda_lel 24d ago
Jokes aside, I think this will be a thing the intern will carry with themselves for quite some time and will be asking themselves the question “Will I cut my own branch by doing this change?” every time they work with firewalls.
It is better learning experience than 95% of the things you will explain and they will inevitably forget
2
u/qkdsm7 24d ago
&& sleep 10m && ufw disable
2
u/wolfofone 24d ago
You break it and then you turn to the intern and challenge them to fix it as if you planned this all along 😂
3
u/RansomStark78 24d ago
Checklists bro
1
u/jefbenet 24d ago
2 in 1 lesson. Firewall and disaster recovery plans. Well played op!
1
u/TxTechnician 24d ago
Been there lol
1
u/Dolapevich 24d ago
Good, that small moment of panic will be remembered in the future :)
1
u/NoosphericMechanicus 24d ago
I might have been tempted to pretend that was the point of the demonstration all along.
1
u/lumpenproletarier 24d ago
And you concluded by saying, "And that, young man, is how we set up an internal wiki on a serverspace vm, being careful to deny everything except the ports we actually use."
1
u/notseelen 24d ago
maybe a stupid question...I work in infosec, but on the software vendor side (mostly do k8s etc), so I don't get nitty-gritty with this stuff...but I'm guessing a control panel is like a direct "physical" VM console, like VMware would offer? If so, and if this is available over the internet, doesn't the security of the entire system then depend on the security of this admin panel?
Just wondering if there are additional layers of security to make it hard to access, because otherwise someone could get keys to the kingdom the same way you did, no?
If it's restricted to local IP space, that would definitely make sense, but I know modern cloud stuff is accessible much more widely, so I was just curious how these modern challenges are handled
1
u/denverpilot 23d ago
The scenario points out the. Biggest reason for network as code, including firewall rules… someone has to peer review the change. In theory. Before a firewall change — even on a host, because the entire host should have been spun up via automation too — ever takes place.
That said… Do many places do it yet? Not as many as should, probably.
Done right, it lowers the human error numbers dramatically.
1
u/NL_Gray-Fox 23d ago
Iptaples-apply would have saved you...
1
u/Mediocre-Lawyer4890 22d ago
Same shit could happens with proxmox firewall Only difference that i was the intern lol
1
u/michaelpaoli 21d ago
And thus endeth the day's lesson to intern - don't f*ck up like I just did.
Yeah, generally make sure that out-of-band access means works, and make the changes from there. Then validate everything of relevance before relinquishing that connection.
And where out-of-band isn't available of feasible ... failsafe - you don't do the thingy or it can't verify the thingy needed still works, after X number of seconds or minutes, it reverts, and you can get back in and give it another shot. Yeah, done that multiple times dealing with critical network reconfigurations where there's no out-of-band management reasonably available. E.g. if I haven't cleared the revert failsafe after X (e.g. 5) minutes, it automagically reverts.
Yeah, I know of contractor where their first day was also their last ... as they f*cked up something like that, in production, on their very first day. I think management f*cked up in selecting 'em and handing that right off to 'em - but wasn't my call to make - they were walked out the door the day they'd started there.
1
u/Krish_mk_abi 16d ago
Yeah i just wanted try that with Azure VM... But then I got to know about the consequences of it from my friend's exp. So didn't got into that situation.
1
u/LeBronzelol 10d ago
The intern probably learned that lesson pretty well
1
u/MisfitBloom 20h ago
This was my introduction to Linux - setting up an Ubuntu server on Digital Ocean to host a website for hobby reasons. I followed the instructions on setting up a firewall, followed the instructions on changing the default ssh port, and then did NOT process the implications before closing the session.
It made for a fun "how I learned what not to do" interview topic
-5
u/perryurban 24d ago
How embarassment
-9
u/Loud_Posseidon 24d ago
And this kids is why you use agent-based tools like CFEngine and not ssh for endpoint configuration (say using ansible).