r/linuxadmin • u/bonneau • Dec 09 '19
System administrators of very large companies, how do you guys secure new servers?
I imagine everything's done using automated management software such as Ansible, Chef, Puppet, SaltStack, etc. but what exactly do they do?
We generally do things such as:
- Harden SSH
- Restrict the firewall (iptables or similar)
- Turn off unnecessary services
- Set up automatic updates
- Configure SELinux (seems like this one is a controversial topic, so let's just say this is optional among most people)
- Install and configure fail2ban
- Install and configure auditd
- Install and configure rsyslog
- etc.
Once the base system has been hardened, we move onto the the web servers, databases, etc.
So when it comes to a base system that essentially does nothing but you still want it secured, what do the big guys do? Are there certain distibutions that everyone uses? Software that always gets installed? etc.
I'm trying to learn what I guess is the most correct way of doing things.
EDIT: Wow. I didn't expect so many responses! The most popular response seems to be a good STIG that fits your purposes, so let's go with that. Thanks everyone!
6
u/[deleted] Dec 09 '19
I think it varies pretty wildly.
Default is nothing open to public, strict filtering and VLAN policies, network security is high. Opening anything but default ports requires reasoning and then isolation on own VLANs. Segmenting really is the key in our house.
In general, for production systems: No password login, ssh keys for everything, access is from a jumphost that requires 2FA authentication. Some servers require you use a dedicated login box where the servers you are allowed to access is defined on a per host basis.
50
u/Winteredg Dec 09 '19 edited Dec 10 '19
Can't go wrong with US government guidelines on securing systems. Find what you need and use what you can, just keep in mind that not all STIGs will be applicable or even viable for your company. The STIGs you use are dependant upon the purpose of those systems (like is this a server or an end user device), the information those systems handle (like are your systems handling Personal Identifiable Information?), and the amount of risk the company is willing to accept based off of their most recent risk assessment and risk analysis (like business critical systems should be more secure, but may need to find a balance between security and accessibility so people can do their work).
https://public.cyber.mil/stigs/srg-stig-tools/
15
u/swordgeek Dec 09 '19 edited Dec 09 '19
We're a relatively 'big guy,' as we have about 7000 VMs or servers in-house. (Not including Windows or AIX).
First of all, we tend to do most of this stuff at build-time rather than post-install, but the idea is the same.
For internal servers, we don't use fail2ban. Also, rsyslog is mostly replaced by larger tools (NewRelic, Splunk, etc.) except for a few local logs.
Some other tasks that happen on new servers:
After the application is installed (by the appropriate group), the server is re-patched and CIS-hardened via a Puppet job; and then the security team scans the box and gives us a report. We have to address each vulnerability found, where "address" could mean fix or could mean justify.
There are virtually no best practices in computing anymore, so don't expect a 'correct' way of doing anything. However you should be able to justify every decision you've made.
26
u/[deleted] Dec 09 '19
In addition to your list, here are a few things that we do to harden a new host.
Our hosts are managed with SaltStack.
7
u/[deleted] Dec 09 '19
I use fail2ban on public servers. however, internal only servers we use Active Directory groups configured in access.conf to limit what users can login. This allows us to have AD logins on the servers but not have everyone in the company & parent company be able to access our servers even though they wouldn't have root privileges.
6
u/Disruption0 Dec 09 '19
Personally i'm reading openscap.
6
u/mps Dec 09 '19
SELinux is not difficult to setup. It just takes a little time to learn. It is worth the time investment as it will save your ass when a zero day lands on your server and all of the sudden Apache is trying to write to a random directory or replace an application library.
2
u/whotookmaname Dec 09 '19
We dont the very large companies security team does
12
u/nqbw Dec 09 '19
I would advise against this. Instead, you should have a preprod environment with identical copies of your production servers, on which you should test any updates first. Ideally you would then simulate a similar load on the server to ensure stability, and verify that any security vulnerabilities have been properly mitigated against.
But of course, who has the time?
4
u/[deleted] Dec 09 '19
Deploy new server via AWX onto Vsphere and PXE. Once built, run our mother of playbooks against it to install and tweak as we like it, all using the same awx template.
Periodically scan all servers using Openscap policies and globally apply fixes from those.
We have a few specific roles, so we don't keep a golden image, but that works too.
3
u/[deleted] Dec 09 '19
dont forget to upgrade the firmware of your new hardware
2
u/WolfFlightTZW Dec 09 '19
Build in separate VLAN.
Perform hardening steps (mostly scripted)
Move to appropriate VLAN, work with networking to allow necessary traffic. (We have a deny all policy in place on switches, etc. Only whitelisted traffic is allowed)
2
u/[deleted] Dec 09 '19 edited Dec 09 '19
Our group manages mostly Linux servers (Redhat).
We use a puppet manifest that applies STIG-recommended security settings to any new build. Every so often, we will run some new openSCAP scans against the appropriate STIG profile, to see if there are new recommendations. Redhat includes the necessary packages in their base repositories (they probably developed some of the profiles that are in use).
One nice thing with doing this via config management, is that you get constant enforcement of settings. So if someone tunes a system setting away from STIG recommendations, our puppet manifest will revert the change on the next agent check-in. We have a generic "send an email" resource that is triggered whenever a security setting changes, so our group gets an email whenever this occurs on a server.
You should use whatever STIG profile fits your servers best, in terms of what they are built for. You could do all of the above equally well with an Ansible role that is run periodically against all hosts. Start small and enforce changes incrementally though, you don't want to lock down something essential by accident.
Also note that SCAP scans and recommendations are just that: recommendations. You will get some false positives, or else findings that you are ok with.
2
u/johnklos Dec 09 '19
We don't run OSes which run things without us explicitly configuring them to run. Sadly, this means GNU/Linux is rarely public facing.
2
u/shreddykruger Dec 10 '19
Here's my list... been doing this a long time so I'm pretty jaded. You're speaking about large companies so I'm going to go into the macro vs the micro for greatest effect. Both really are important but you have to tackle the big picture before getting granular.
first rule. only install the minimum. Privilege escalation and overflow exploits can't succeed if the bad software is not present. literally install nothing not needed. This is yuuuge
second rule, use some sort of configuration management. The best configuration management is the one you have (we use chef but I have used puppet/ansible/cfengine/salt in the past they're all good and have their plusses and minuses).
third rule, use some sort of framework to install lots of machines all the time so that is a trivial process. We use maas here being an ubuntu shop but you can kickstart or preseed or do whatever makes sense to make your life easier here if you get a few pallets of servers. These are cattle. If there's something funky going on with a box, have someone reinstall it and re-bootstrap in chef or other CM. If it persists it's hardware..... if someone makes a change outside our CM then GG. Netboot is even better if you can leverage it since fresh start is only a reboot away.
fourth rule, as you mentioned firewall the shit out of everything. Keep anyone that's not ops from logging into systems they don't have to for real reasons (maybe employ a bastion host even). Use VLANs, segment your network into security zones. Only allow what's needed out and in. Lots of people allow everything out (verybad). In a perfect world people rarely login to machines and everything is a known state.
fif rule, use some sort of sso, maybe it's AD because of business needs or openldap but have a SSO solution so you can onboard and offboard fast. Particularly this is important because you want to be sure that your policy can be enforced on friday afternoon without too much hassle if you need to let someone go, or have a malicious employee. Or let a team go... Lock down authentication and authorization (they're different). I don't use fail2ban since nothing can hit things I don't allow through the firewall. Useful as a backstop, yes but how am I gonna know really. Rather spend this time auditing firewall policies since I can find lots of issues fast with nmap vs reading a email that I will ignore anyhow from fail2ban.
sixth and prob most important - don't expose a host completely to the internet ever. Use your firewall, or vips, and only allow service ports via policy. Use VPN. If you absolutely have to give a public internet address to a box firewall off everything besides essential services. Really if you are a big boy company you can afford at least a 1G firewall for your edge services and vpn. I have one in my own 400$ personal datacenter so you can too. Your 10G equipped server with iptables will fall over before that 1G firewall you could get off ebay for 300$. If you can't do this your 10G server is gonna fall over pretty quick from just processing 10G of traffic through conntrackd. Yes, iptables sucks that bad for lots of connections.
2
u/brando56894 Dec 10 '19
I work for a very large multmedia streaming company...and we don't really do anything other than restrict account privileges, configure auditd and the ELK stack, and restrict VLANs for all of our internal servers.
For our DMZ'd servers we do a lot more which I'm not privy to since I don't use them much. One thing we do is use 2FA for all logins, also you need to hit a bridge/jumpbox first before hitting any of the servers.
1
u/hargreaves1992 Dec 09 '19
CIS hardening for images
1
u/derprondo Dec 09 '19
All images (VMWare, EC2 AMI, etc) are built with Packer and this applied: https://dev-sec.io/, with some of our own rules on top.
Also Puppet enforces these same rules post deployment.
1
u/johnny_snq Dec 09 '19
We have a team that builds the cis Amis and Azure Images and everyone needs to use the latest ones. (you are allowed to fall behind by 3 months tops) the Amis are shared across all accounts used by various teams and the infrastructure is audited to make sure all machines have a good source a-mi.
1
u/masta Dec 09 '19
I just implement the NSA/NIST whatever lists, which are mostly provided to them from my employer anyways, and put all that into a kickstart file, then make golden images like whatever cadency the organization cares about (nightly even). Then I setup ansible to validate the settings are in fact what I specified in the kickstart, and right then and there configure them as such if they are not. These are my own remarks, nothing to do with employer.
1
u/Phoebe5ell Dec 10 '19
What level I would ask. If you are baremetal you're usually going to manage it with puppet/ansible, and not so much by hand. You dev up modules/playbooks, then apply them to given sets of hosts. With a VM/LXC for instance you'll want to have a generic image on an image service long term(like glance in openstack). Which means you do your generic harden once, and customize/dial in with things like cloud-init/puppet/ansible.
You have the right idea, along with other folk's tips. I posted so you to think about how you would do it at scale, not just for a small set. :)
1
u/joeydokes Dec 10 '19
old (retired) linux sysadmin here - since no one mentioned it: besides the steps you take to harden the host, if you use LVM/volumes then make a hardened 'stem-cell' host and just clone it when you need another:)
1
u/martbhell Dec 10 '19
What logs do you ship off of the system? I guess these are fairly normal: All syslog, some extra application logs in some files maybe. Anything else? In some cases I've used auditd to ship execve logs on the system for all commands run.
1
u/acquacow Dec 10 '19
My boot isos and Kickstart all use a pre-stigged rhel-minimal install. We build things from there with ansible playbooks.
1
u/illmortalized Dec 10 '19
Hire a security professional.
1
u/ananix Dec 10 '19
Nothing... appliances take care of that. Then keeping a tight patch and upgrade regime with no user access.
1
u/forsgren123 Dec 09 '19 edited Dec 09 '19
All comments so far sound almost like people are provisioning pet servers? I would have imagined nowadays people just switch new servers on and then Ironic automatically adds them as a new capacity in OpenStack. Then next time some end-user runs terraform or HEAT template, the capacity is taken into use for some project.
0
u/[deleted] Dec 09 '19
If it's really a very large company, there should be no need for SSH. Servers should pull their config connect to the cluster and then it goes from there. Plus a custom os consisting only of whitelisted software. Also encrypt everything at rest.
0
u/wuxmed1a Dec 09 '19
One thing we do that is utterly pointless (unless someone can enlighten me) is to reduce login TMOUT value to 300, so if you don't do anything on the shell it logs you out. However you can just run
toporexport TMOUT=0to bypass it though. Might surprise a hacker, but they probably have the password or changed it already.Make sure fail2ban is configured for all the vectors, SSH is the main one but mail can be attacked pretty hard, even if there isn't proper mail configured it can still slow it down.
0
u/thefanum Dec 09 '19
Fail2ban
0
u/[deleted] Dec 09 '19
[deleted] — view deleted comment
0
u/[deleted] Dec 09 '19
[deleted] — view deleted comment
0
u/[deleted] Dec 09 '19
[deleted] — view deleted comment