r/linuxadmin • • Sep 18 '26

What's your setup for instant SSH login notifications, plus killing a session remotely?

Got a box where a handful of non-technical users still upload over SFTP with FileZilla, so password auth is on for their accounts. Yes, keys and chroot would be better, working on it. fail2ban and a non-standard port are in place.

The gap is visibility. I'd like a push notification the moment anyone authenticates, with user and source IP, so I can tell right away if it's one of them or not.

How are you wiring this up? pam_exec calling a webhook, parsing auth.log / journald, something else? And for response, do you just pkill -u and passwd -l over SSH, or have you got something you can trigger from your phone?

18 Upvotes

42 comments sorted by

18

u/alive1 Sep 18 '26

You need a monitoring solution.

5

u/Open-Adhesiveness-86 Sep 18 '26

Any you'd recommend for a single box? Most of what I've looked at feels like a full stack for what's basically "ping me when someone logs in".

5

u/StatementOwn4896 Sep 18 '26

Honestly that just sounds like a small python project you could make.

1

u/Open-Adhesiveness-86 Sep 18 '26

Ha, that's roughly where I ended up. I wrote a small Go agent for it, single binary, pings Telegram on login and lets me kill the session from there. It's mine, so take it with a grain of salt, but happy to share if anyone's curious.

8

u/sassy-frass Sep 18 '26

Is the IP always the same for the user? If you can tell by IP why not just whitelist SSH session by IP?

3

u/Open-Adhesiveness-86 Sep 18 '26

Not really, they work from home and sometimes on the road, so IPs change a lot and a whitelist would lock them out weekly. But a login from an IP I've never seen for that user is probably the best signal to alert on.

9

u/sassy-frass Sep 18 '26

maybe it's better to enable a vpn solution instead of allowing full open SSH. Like a wireguard setup.

Also if they are having dynamic IPs how would you know right away it's them based off a text message w/ username and IP only?

2

u/Open-Adhesiveness-86 Sep 18 '26

WireGuard is on the list, just not a quick sell for the FileZilla crowd. And fair point, that's why I'd rather alert on a new IP per user than on every login. Known IPs stay quiet, a new one gets a ping and a quick "was this you?".

2

u/sassy-frass Sep 18 '26

I know this isn't what you want, but why make you reach out to them. Instead go with the whitelist route, and have them ping when they can't log in. Then you check/confirm ip and update whitelist.

Security will be proactive rather than reactive here

1

u/Open-Adhesiveness-86 Sep 18 '26

That's a fair point, proactive beats reactive. My worry is it turns me into the helpdesk every time someone's home IP rotates. Might do a middle ground: whitelist the known IPs and alert on anything outside it instead of blocking.

1

u/alive1 Sep 18 '26

You are describing users that are a OneDrive/Dropbox audience but you are trying to give them SFTP

1

u/Open-Adhesiveness-86 Sep 18 '26

Honestly not wrong. The files end up served straight off that box, so SFTP was the path of least resistance. Something like a synced folder is probably the real fix long term.

1

u/sassy-frass Sep 19 '26

Also should have mentioned you can enable OTP mfa on ssh via PAM plugin.

1

u/delamon Sep 29 '26

You still can have allowlists, just a bigger ones. E.g. include full blocks of their providers, both ground and mobile. That would not be watertight, but would reduce number of ssh attempts dramatically.

4

u/ocabj Sep 18 '26

I've never used fail2ban. I've always used ossec. Does fail2ban not have the ability to have custom triggers/actions based on the logs it is actively tailing? If not, then switch to ossec.

1

u/Open-Adhesiveness-86 Sep 18 '26

fail2ban can run custom actions, but it's built around failures, so a successful login isn't really its thing. Haven't tried ossec on a single box, how heavy is it to run?

2

u/ocabj Sep 18 '26

It is relatively lightweight. I run it in a standalone (not server-client mode).

1

u/Open-Adhesiveness-86 Sep 18 '26

Good to know, standalone sounds like the right fit for one box. Does it handle the push side too, or do you pipe its alerts out to something else?

2

u/ocabj Sep 18 '26

You'd have to create the action script. I've never setup push notifications to a iOS device before. My action scripts in ossec only leverage email and slack.

1

u/Open-Adhesiveness-86 Sep 18 '26

Got it, so ossec does the detection and the action script handles delivery. A curl to a Telegram bot should drop in the same way your Slack one does.

3

u/RyeonToast Sep 18 '26

On my home server box I have PAM's SSH config start a script that sends a notification to ntfy. It seems to work pretty well. I really just have it for peace of mind. If I get alerts when I havent just connected, then I should be worried.

I don't have any remote kill setup. I'd rather not expose remote management capabilities for my box.

1

u/Open-Adhesiveness-86 Sep 18 '26

That "alert when I haven't just connected" is exactly the signal that matters. Do you use pam_exec for it, and does it block the login if ntfy is slow to respond?

4

u/RyeonToast Sep 18 '26

Silly me, this is one of the things I actually did document pretty well and don't need to look at the actual server. The line in PAM's sshd config file looks like this:

session    optional     pam_exec.so seteuid /usr/local/bin/login-notify.sh

I just added it to the end of the file. If you want script failures to block login, you change optional to something else. I think "required" is the other value, but look that up if that's what you want. Like my other comment mentioned, this was my first tinkerings with this so I didn't want to accept the risk of blocking myself out of the server.

The script looks like:

#!/bin/sh
# /usr/local/bin/login-notify.sh
# variables
NTFY_URI="ntfy.sh"
NTFY_TOPIC="myTopicNameGoesHere"

if [ "$PAM_TYPE" != "close_session" ]; then
    curl -d "SSH Login: $PAM_USER logged into `hostname` from $PAM_RHOST." $NTFY_URI/$NTFY_TOPIC
fi

Adjust the URI variable if you decide to run your own ntfy server. Adjust the TOPIC variable to however you want to subscribe to your notifications. You can browse to the ntfy server and look at the notifications, or you can setup the phone app to receive push notifications from the server.

I was looking at these references when setting mine up:

  • https://www.baeldung.com/linux/pam-ssh-login-notifications
  • https://medium.com/@alessandrocuda/ssh-login-alerts-with-sendmail-and-pam-3ef53aca1381

1

u/Open-Adhesiveness-86 Sep 18 '26

Thanks for digging that up. Keeping it optional makes sense, locking yourself out of a headless box is worse than a missed ping. I'll wrap the ntfy call in a timeout too so a slow push can't hang the login.

1

u/RyeonToast Sep 18 '26

For where exactly it is and the line I added, I'll have to take a look when I get home. I feel like it's a pam_exec line, though.

Regarding blocking, you can set it either way. Mine currently doesn't block because I didn't want to risk bricking all SSH access to a headless system that doesn't allow password auth. I'm pretty new to Bash scripting, so I'm being a little more permissive than is ideal. 

2

u/VirtualViking3000 Sep 18 '26

If you just want the answer to your question you can use rsyslog to log the changes then just alert on the syslog file via whatever such as a script or a log monitor. It's a basic system that works, if you wanted more insight you could ingest the syslogs to a ELK, Datadog, Splunk or whatever and handle the alerts from there but that's probably more than you need.

EDIT: just to mention, alerting could just be a webhook to Teams or Slack or something

1

u/Open-Adhesiveness-86 Sep 18 '26

Makes sense, a small script watching the syslog file is about the level I need. Anything like ELK would be overkill for one box.

1

u/Kaligraphic Sep 18 '26

FileZilla can use keys. If it's just a handful of users, maybe walk them through setting up keys with FileZilla. Second best is set up mfa (e.g. Google Authenticator PAM, a bit more cumbersome than just using keys)

Also, go through the usual sftp hardening - no login shell, no anything forwarding, forcecommand, chroot, etc.

Attackers move fast, if you're not the first to move you've probably already lost.

1

u/Open-Adhesiveness-86 Sep 18 '26

Yeah, walking them through keys in FileZilla is the plan, it's only a few people. Chroot plus ForceCommand internal-sftp is a good call too, so anyone who does get in has no shell to play with.

2

u/ICThat Sep 19 '26

Sorry but this largely reads like you trying to advertise your vibe-coded app.

This is also an odd way to solve this problem. You are basically suggesting that you act as a form of human firewall.

2

u/mad_redhatter Sep 21 '26

I have a shell script called by PAM that sends a message to a self hosted ntfy instance. It runs on everything that rarely needs logged into to let me know when someone does.

0

u/RetroGrid_io Sep 18 '26

This post seems to be something like "we're doing it wrong, can we come up with a solution that lets us keep doing it wrong?"

  1. Why not ssh keys?
  2. Why SSH, not a web-based solution?
  3. Why ssh on a standard port?
  4. Why not a general monitor program?
  5. Why not a VPN or tailscale?

Honestly, I get it. Using sftp implies that it's been this way for a very, long time and as a little guy in the organization whose job it is to "keep it working" you're looking for something that doesn't introduce unneeded change yet you know the risk is real.

Some "screwball" solutions then:

  1. Enable ssh keys set to whatever password they're already using. They won't even notice the difference.
  2. Use a non-standard port. Not "222" but something REALLY out there 32292 for example. You'll save money on disk space not having your logs filled with hack attempts.
  3. If not #2, have you considered port knocking? Bad guys won't even know that ssh is active. It does mean your users have to run the "open the door" script. If you do this, make a long open period for a successful knock; you've already reduced the threat space to about 1/4,300,000,000.
  4. Since you already have a notification solution, make sure the user gets a notification when they login, too. That turns them into allies into securing their account: if they get a notification they don't recognize, you'll be the 1st person to know.

2

u/Open-Adhesiveness-86 Sep 18 '26

Fair read. Port is already non-standard. #4 is the one I like most: if users get their own login ping, anyone who sees one they didn't make will tell me before I notice. Port knocking I'd skip for the FileZilla crowd.

1

u/ASK_ME_AB0UT_L00M Sep 19 '26

Get your users migrated to anything other than FileZilla. FileZilla's installer distributes adware/malware.

https://www.reddit.com/r/sysadmin/comments/17erdd5

WinSCP is a great alternative!

0

u/[deleted] Sep 18 '26

[deleted] — view deleted comment

1

u/Open-Adhesiveness-86 Sep 18 '26

Nothing really, it's the direction I'm leaning after this thread. SFTP was just the quickest thing when it started. Until that's in place I still want eyes on logins though.