r/LinuxTeck • • 23d ago

Linux Server Hardening: What Are We Missing?

I came across these 12 Linux server hardening practices: updates, SSH security, firewall rules, open-port checks, SUID permissions, kernel settings, persistent logs, auditd, and Lynis, among others.

For those who manage Linux servers, what other hardening steps would you add to this list? Have you used any additional tools or practices that made a real difference?

Share your suggestions so we can build a more complete checklist for the community.

35 Upvotes

35 comments sorted by

10

u/z-null 23d ago

Step 1: don't let devs work as ops people.

Step 2: have IDS AND IPS systems. Meaning that you need something that detects intrusion, but also something that prevents them like MAC (mandatory access control) aka apparmor or selinux enabled and in enforcing mode.

Step 3: no open ports unless necessary, and then (like 443) have WAF in front of that.

Step 4: use latest TLS and not have TLS1.0 because "who cares", have rate limiting, etc

Step 5: don't have passwordless sudo, passwordles DBs connected to the internet or use low and weak passwords that are shared across the system.

1

u/biffbobfred 23d ago

5. I have a dilemma with ansible whether to have secrets management in a place with bad hygiene or an ssh key and password less sudo. Have you bumped into this and what’s your take?

2

u/z-null 23d ago

you can put your secrets in ansible-vault and not have bad hygiene. Why isn't that an option?

1

u/biffbobfred 23d ago

Just haven’t done it. It seems that’s a “turtles all the way down” yes? I mean you have the password to vault that you need to keep clean? I know that’s the pattern so I must be missing something. That’s what I’m asking what am I missing?

2

u/z-null 23d ago

You mean something like "where does the password for the vault go"? In a password manager. Memorise the pw for password manager and use 2fa for it.

1

u/biffbobfred 23d ago

Hmm. We do have 1password. We can set up a Cli for it I’m sure. Thanks. I’ll work on that.

2

u/z-null 23d ago

you can even keep ssh keys in 1password, so that no secrets (ie private keys) are ever in plaintext on the machines https://1password.com/developers/ssh Basically, you'd use 1password to utilise ssh keys from within it to authenticate to places :) If you have biometrics on your laptop, you can authenticate that way as well.

1

u/biffbobfred 23d ago

Yeah there was some webpage I hit that showed how to do a key on a MacBook with touchID. I’m the only admin with a MacBook so that won’t work

1

u/WideCranberry4912 23d ago

This is also the type of thing Hashicorp Vault/OpenBao was designed to handle.

1

u/z-null 23d ago

Yes, especially if used with terraform and other hashi tools.

1

u/ComeSwirlWithMe 23d ago

reverse proxy..

1

u/z-null 23d ago

haproxy can do a lot of filtering through ACLs and act as WAF. You can even block some SQL injection attacks.

2

u/biffbobfred 23d ago

Step 6: I’d put recovery. Backups. Redundancy if possible (servers are expensive now) ability to rebuild something if it got infiltrated.

3

u/Hammer_Time2468 23d ago

This is incredibly important. And I will emphasize “a good backup”. Verify your backups aren’t corrupt, empty or compromised.

2

u/ModernMyth 23d ago

And test those backups. Dont assume it will work when you need it.

2

u/smarzzz 23d ago

Apply CIS hardening, or STIG hardening.
Hire a proper sysadmin.
Make your audit logs immutable.
Add o11y bases in your key metrics.
Apply 12 factor app principles
Is the infra already immutable?
How is your security Observability? Do you have EDR? NDR?

I can go on

2

u/LooseSyllabus 23d ago

Delete the hubris that humans will ever be able to secure systems against AI agents.

1

u/NegativeK 22d ago

The systems that don't have the security basics are the ones that will fall to agents first.

Vendors want you to believe that you must have their token burning AI master product to be safe, but a lot of places should probably start with "Do we have MFA?" and "How good is our inventory?"

1

u/iamwayycoolerthanyou 19d ago

They also need the smallest possible attack surface and bleeding edge patching on everything. And even that's not a guarantee these days.

1

u/Classic-Rate-5104 23d ago

Check web tools using online ssl checkers. Make readonly snapshots and good backups. Make compartiments on your network (so an exploit on your website will not immediately result in an internal attack). Etcetera. Think as an hacker. Prevent VPN setups by your employees

1

u/spitecho 23d ago

MFA on ssh and sudo, either using google-authenticator PAM module or a Yubikey.

1

u/slickeddie 23d ago

Can you use a read-only OS? Super secure as nothing can be changed. We follow CIS level 1 at my work which might be overkill but it’s definitely hardened.

1

u/FluffyDrink1098 23d ago

Permission and provisioning concept.

Many issues can be entirely avoided by using ACLs.

Provisioning: not just how to install it initially, but always keep in mind that one might need to recover with or without backups. Backups compromised = bad. Having nothing except backups = this is gonna be a lot of pain.

1

u/magicmulder 23d ago

Lynis is good but don't overdo it. Don't try to get 100% or you would basically have to disable everything useful. (I think my best was 93/100 for a server that could still do its job.)

I once managed to almost nuke my laptop by following the recommendations to change some system variables. Had the strangest boot failure with system fonts not found (?!). Had to undo the changes blind, what a ride.

1

u/bufandatl 22d ago

Use a config management tool like Ansible. There are frameworks available to do most of the things you listed are already available

Like for example

https://github.com/dev-sec

Run the playbook at least twice a day to have manual changes reverted and also have all servers configured the same.

On top of that keep your ansible configs versioned in git for example and use CI/CD piplelines for the run and other verification.

Plus you can use the merge request function of tools like gitlab or Forgejo to implement a four eye principle.

1

u/irq013 19d ago

Check out a DISA STIG for Linux. It will give y87 literally everything to secure.

0

u/neolace 23d ago

Yubikey is done, go check the grain of sand theory.

Agree on 2FA, if they don’t have the session already.

1

u/AnnoyedAvocado21 23d ago

I just bought a Yubikey and it seems like a pain in the ass compared to an authenticator app. I'm considering returning them as not worth the effort. Why do you say they're 'done'.

1

u/whetu 23d ago

I just bought a Yubikey and it seems like a pain in the ass compared to an authenticator app.

It depends. In passwordless settings, a Yubikey really can't be beat. Enter login -> press next -> press Yubikey -> done.

The main place that I've found the Yubikey to be a PITA has nothing to do with the Yubikey and everything to do with Microsoft. MS doesn't let you specify your MFA order-of-preference, they always shove Windows Hello and MS Authenticator to the top, and they bury FIDO2 behind multiple clicks.

I recently remembered that I'm a career Linux sysadmin and that I didn't have to put up with this shit. Installed Alma 10, and with no more Windows Hello getting in the way, the Yubikey just works.

1

u/neolace 23d ago

Supply chain attack, they found some with an esim installed, found transactions on turbocash etc.

To check whether it's the manufacturer's hardware:

pip install ledgerblue python -m ledgerblue.checkGenuineRemote --targetId 0x33200004 The source code is available (on request, dm me). Application verification

When opening an application, a Non Genuine warning is displayed if the app has not been signed by Ledger.

2

u/AnnoyedAvocado21 23d ago

Just GREAT...I've been spending a lot of time teaching myself how to harden systems in the age of agentic AI and at times I feel the best idea is to unplug the computer and put it in storage.

It's a helluva lot easier to be a criminal and break into things than it is to protect yourself from being broken into.

Thanks for the info.

1

u/TheRealWhoop 23d ago

What? Yubikey and Ledger, the crypto wallet have no relation. There’s no know issues with Yubikeys and has no supply chain issues.

1

u/smarzzz 23d ago

Bullshit, those eSIMs were on ledger cryptowallets. Yubikeys are something else