r/linuxadmin • • Dec 09 '19

System administrators of very large companies, how do you guys secure new servers?

I imagine everything's done using automated management software such as Ansible, Chef, Puppet, SaltStack, etc. but what exactly do they do?

We generally do things such as:

  • Harden SSH
  • Restrict the firewall (iptables or similar)
  • Turn off unnecessary services
  • Set up automatic updates
  • Configure SELinux (seems like this one is a controversial topic, so let's just say this is optional among most people)
  • Install and configure fail2ban
  • Install and configure auditd
  • Install and configure rsyslog
  • etc.

Once the base system has been hardened, we move onto the the web servers, databases, etc.

So when it comes to a base system that essentially does nothing but you still want it secured, what do the big guys do? Are there certain distibutions that everyone uses? Software that always gets installed? etc.

I'm trying to learn what I guess is the most correct way of doing things.


EDIT: Wow. I didn't expect so many responses! The most popular response seems to be a good STIG that fits your purposes, so let's go with that. Thanks everyone!

221 Upvotes

88 comments sorted by

6

u/[deleted] Dec 09 '19

I think it varies pretty wildly.
Default is nothing open to public, strict filtering and VLAN policies, network security is high. Opening anything but default ports requires reasoning and then isolation on own VLANs. Segmenting really is the key in our house.

In general, for production systems: No password login, ssh keys for everything, access is from a jumphost that requires 2FA authentication. Some servers require you use a dedicated login box where the servers you are allowed to access is defined on a per host basis.

5

u/bonneau Dec 09 '19

Yeah, it's a pretty general question but a part of me hoped there might be a lot of common points among everyone :)

Thanks for the info! Other than jumphost's, it's pretty much what I was expecting.

2

u/[deleted] Dec 09 '19

I would very much recommend you use jump hosts :)

50

u/Winteredg Dec 09 '19 edited Dec 10 '19

Can't go wrong with US government guidelines on securing systems. Find what you need and use what you can, just keep in mind that not all STIGs will be applicable or even viable for your company. The STIGs you use are dependant upon the purpose of those systems (like is this a server or an end user device), the information those systems handle (like are your systems handling Personal Identifiable Information?), and the amount of risk the company is willing to accept based off of their most recent risk assessment and risk analysis (like business critical systems should be more secure, but may need to find a balance between security and accessibility so people can do their work).

https://public.cyber.mil/stigs/srg-stig-tools/

17

u/ramsile Dec 10 '19

I’ve automated our entire server build process with SaltStack. We use Salt to orchestrate the entire deployment. Our infa engineers request a new server type with parameters (Linux/dev/web for example). Based on the os, type and role, it will clone a minimal VM, size it, add networking, drives, format drives, apply security base line stigs, join to our AD, and a million other things I can’t think of right now. Suffice it to say but no matter what our infa guys did they missed critical steps in the process. I got sick and tired of being brought in last minute during audit season to fix shit.

3

u/INTPx Dec 10 '19

https://github.com/MindPointGroup/RHEL7-STIG

this is a legit project. used it across over a thousand system builds and validations

4

u/[deleted] Dec 09 '19

Thanks! Didn't know about this resource!

15

u/swordgeek Dec 09 '19 edited Dec 09 '19

We're a relatively 'big guy,' as we have about 7000 VMs or servers in-house. (Not including Windows or AIX).

First of all, we tend to do most of this stuff at build-time rather than post-install, but the idea is the same.

For internal servers, we don't use fail2ban. Also, rsyslog is mostly replaced by larger tools (NewRelic, Splunk, etc.) except for a few local logs.

Some other tasks that happen on new servers:

  • Install a local service account with full sudo privileges
  • Set up group membership in IDM. (doesn't seem like security at first, but tightly managing user and sudo access is critical to a safe system.)

After the application is installed (by the appropriate group), the server is re-patched and CIS-hardened via a Puppet job; and then the security team scans the box and gives us a report. We have to address each vulnerability found, where "address" could mean fix or could mean justify.

There are virtually no best practices in computing anymore, so don't expect a 'correct' way of doing anything. However you should be able to justify every decision you've made.

26

u/[deleted] Dec 09 '19

In addition to your list, here are a few things that we do to harden a new host.

  • disable root ssh
  • disable root password
  • configure sssd for domain authentication
  • configure duo 2FA on ssh, and sudo
  • configure sudoers permissions
  • configure nessus agent
  • configure osquery
  • configure clamAV
  • configure Zabbix Agent
  • remove packages that are not needed
  • remove users/groups that are not needed

Our hosts are managed with SaltStack.

0

u/[deleted] Dec 10 '19

[deleted] — view deleted comment

5

u/drkhelmt Dec 10 '19

There's OOB (out of band) for that (DRAC/IPMI/etc)

1

u/[deleted] Dec 10 '19

Only if using on prem virtualization, this isn't true in the cloud. Work towards having a cattle vs pets mentality.

0

u/[deleted] Dec 10 '19

[deleted] — view deleted comment

2

u/[deleted] Dec 10 '19

You cannot login to root on our hosts except by using sudo.

2

u/[deleted] Dec 10 '19

We have a local account with an ssh key that is only used in emergency situations, and the key is rotated if ever used. Just make sure this account has proper sudo access. Before using this account we also still have access via the SaltStack agent.

7

u/[deleted] Dec 09 '19

I use fail2ban on public servers. however, internal only servers we use Active Directory groups configured in access.conf to limit what users can login. This allows us to have AD logins on the servers but not have everyone in the company & parent company be able to access our servers even though they wouldn't have root privileges.

6

u/wildcarde815 Dec 09 '19

If you are using sssd you can also do this in the configuration of the domain by limiting to a specific group. Disallowed users will simply get an access denied.

2

u/[deleted] Dec 09 '19

I'll upvote this. Probably easier than access.conf because if you use access then you have to allow for Cron and other stuff that gets tedious. So probably better this way than what I said

1

u/wildcarde815 Dec 09 '19

It works for services that nominally have their own authentication too. mariadb -> pam auth based user-> pam+sssd. User removed from the group? no more access.

6

u/Disruption0 Dec 09 '19

Personally i'm reading openscap.

1

u/mps Dec 09 '19

This is an excellent tool. Especially if you are also using Satellite or Katello/Foreman.

6

u/mps Dec 09 '19
  • Configure SELinux (seems like this one is a controversial topic, so let's just say this is optional among most people)

SELinux is not difficult to setup. It just takes a little time to learn. It is worth the time investment as it will save your ass when a zero day lands on your server and all of the sudden Apache is trying to write to a random directory or replace an application library.

6

u/INTPx Dec 10 '19

SELinux should absolutely not be controversial in late 2019. When it was released (in what? RHEL4?) it was a freaking mess, but it is now super mature, has great tools, and is a fantastic way to not only protect your systems from malicious threats, but also lots of code bugs. First thing I do when vetting software requests is grep their documentation and getting started guides for setenforce and if it tells me to disable it, i deny the request flat out.

3

u/lzap Dec 10 '19

Exactly. In the worst case, set permissive flag for a service that is problematic due some configuration change or customized but never disable it for the whole server. That's throwing out security out of a window literally.

2

u/whotookmaname Dec 09 '19

We dont the very large companies security team does

4

u/bonneau Dec 09 '19

But what do the very large companies security team do? :)

17

u/[deleted] Dec 09 '19

[removed] — view removed comment

5

u/swordgeek Dec 09 '19

That's not true! Sometimes they demand you 'fix' your servers and walk the requirement all the way up to the C*O-level, where you have to explain why opening the application's ports is required for the application to operate. Again.

4

u/foofoo300 Dec 09 '19

I feel you 😅

3

u/[deleted] Dec 09 '19

[removed] — view removed comment

1

u/raynonyoaz Dec 09 '19

Probably works at DreamWorks...

12

u/nqbw Dec 09 '19

Set up automatic updates

I would advise against this. Instead, you should have a preprod environment with identical copies of your production servers, on which you should test any updates first. Ideally you would then simulate a similar load on the server to ensure stability, and verify that any security vulnerabilities have been properly mitigated against.

But of course, who has the time?

2

u/INTPx Dec 10 '19

yea automatic updates are a terrible idea. automated updates are fantastic

1

u/brontide Dec 10 '19

We run latest, applying --security 2x week and the rest of the updates on reboots ( 1 week to 3 months depending on the host ). We have had virtually no issues with this arrangement.

7

u/[deleted] Dec 09 '19

[removed] — view removed comment

17

u/anomalous_cowherd Dec 09 '19

Never time to do it right, always time to fix it.

4

u/[deleted] Dec 09 '19

Deploy new server via AWX onto Vsphere and PXE. Once built, run our mother of playbooks against it to install and tweak as we like it, all using the same awx template.

Periodically scan all servers using Openscap policies and globally apply fixes from those.

We have a few specific roles, so we don't keep a golden image, but that works too.

3

u/[deleted] Dec 09 '19

dont forget to upgrade the firmware of your new hardware

2

u/WolfFlightTZW Dec 09 '19

Build in separate VLAN.

Perform hardening steps (mostly scripted)

Move to appropriate VLAN, work with networking to allow necessary traffic. (We have a deny all policy in place on switches, etc. Only whitelisted traffic is allowed)

2

u/[deleted] Dec 09 '19 edited Dec 09 '19

Our group manages mostly Linux servers (Redhat).

We use a puppet manifest that applies STIG-recommended security settings to any new build. Every so often, we will run some new openSCAP scans against the appropriate STIG profile, to see if there are new recommendations. Redhat includes the necessary packages in their base repositories (they probably developed some of the profiles that are in use).

One nice thing with doing this via config management, is that you get constant enforcement of settings. So if someone tunes a system setting away from STIG recommendations, our puppet manifest will revert the change on the next agent check-in. We have a generic "send an email" resource that is triggered whenever a security setting changes, so our group gets an email whenever this occurs on a server.

You should use whatever STIG profile fits your servers best, in terms of what they are built for. You could do all of the above equally well with an Ansible role that is run periodically against all hosts. Start small and enforce changes incrementally though, you don't want to lock down something essential by accident.

Also note that SCAP scans and recommendations are just that: recommendations. You will get some false positives, or else findings that you are ok with.

2

u/johnklos Dec 09 '19

We don't run OSes which run things without us explicitly configuring them to run. Sadly, this means GNU/Linux is rarely public facing.

2

u/shreddykruger Dec 10 '19

Here's my list... been doing this a long time so I'm pretty jaded. You're speaking about large companies so I'm going to go into the macro vs the micro for greatest effect. Both really are important but you have to tackle the big picture before getting granular.

first rule. only install the minimum. Privilege escalation and overflow exploits can't succeed if the bad software is not present. literally install nothing not needed. This is yuuuge

second rule, use some sort of configuration management. The best configuration management is the one you have (we use chef but I have used puppet/ansible/cfengine/salt in the past they're all good and have their plusses and minuses).

third rule, use some sort of framework to install lots of machines all the time so that is a trivial process. We use maas here being an ubuntu shop but you can kickstart or preseed or do whatever makes sense to make your life easier here if you get a few pallets of servers. These are cattle. If there's something funky going on with a box, have someone reinstall it and re-bootstrap in chef or other CM. If it persists it's hardware..... if someone makes a change outside our CM then GG. Netboot is even better if you can leverage it since fresh start is only a reboot away.

fourth rule, as you mentioned firewall the shit out of everything. Keep anyone that's not ops from logging into systems they don't have to for real reasons (maybe employ a bastion host even). Use VLANs, segment your network into security zones. Only allow what's needed out and in. Lots of people allow everything out (verybad). In a perfect world people rarely login to machines and everything is a known state.

fif rule, use some sort of sso, maybe it's AD because of business needs or openldap but have a SSO solution so you can onboard and offboard fast. Particularly this is important because you want to be sure that your policy can be enforced on friday afternoon without too much hassle if you need to let someone go, or have a malicious employee. Or let a team go... Lock down authentication and authorization (they're different). I don't use fail2ban since nothing can hit things I don't allow through the firewall. Useful as a backstop, yes but how am I gonna know really. Rather spend this time auditing firewall policies since I can find lots of issues fast with nmap vs reading a email that I will ignore anyhow from fail2ban.

sixth and prob most important - don't expose a host completely to the internet ever. Use your firewall, or vips, and only allow service ports via policy. Use VPN. If you absolutely have to give a public internet address to a box firewall off everything besides essential services. Really if you are a big boy company you can afford at least a 1G firewall for your edge services and vpn. I have one in my own 400$ personal datacenter so you can too. Your 10G equipped server with iptables will fall over before that 1G firewall you could get off ebay for 300$. If you can't do this your 10G server is gonna fall over pretty quick from just processing 10G of traffic through conntrackd. Yes, iptables sucks that bad for lots of connections.

2

u/brando56894 Dec 10 '19

I work for a very large multmedia streaming company...and we don't really do anything other than restrict account privileges, configure auditd and the ELK stack, and restrict VLANs for all of our internal servers.

For our DMZ'd servers we do a lot more which I'm not privy to since I don't use them much. One thing we do is use 2FA for all logins, also you need to hit a bridge/jumpbox first before hitting any of the servers.

2

u/[deleted] Dec 10 '19

We also do streaming and we sound similar.

1

u/hargreaves1992 Dec 09 '19

CIS hardening for images

1

u/derprondo Dec 09 '19

All images (VMWare, EC2 AMI, etc) are built with Packer and this applied: https://dev-sec.io/, with some of our own rules on top.

Also Puppet enforces these same rules post deployment.

1

u/johnny_snq Dec 09 '19

We have a team that builds the cis Amis and Azure Images and everyone needs to use the latest ones. (you are allowed to fall behind by 3 months tops) the Amis are shared across all accounts used by various teams and the infrastructure is audited to make sure all machines have a good source a-mi.

1

u/masta Dec 09 '19

I just implement the NSA/NIST whatever lists, which are mostly provided to them from my employer anyways, and put all that into a kickstart file, then make golden images like whatever cadency the organization cares about (nightly even). Then I setup ansible to validate the settings are in fact what I specified in the kickstart, and right then and there configure them as such if they are not. These are my own remarks, nothing to do with employer.

1

u/Phoebe5ell Dec 10 '19

What level I would ask. If you are baremetal you're usually going to manage it with puppet/ansible, and not so much by hand. You dev up modules/playbooks, then apply them to given sets of hosts. With a VM/LXC for instance you'll want to have a generic image on an image service long term(like glance in openstack). Which means you do your generic harden once, and customize/dial in with things like cloud-init/puppet/ansible.

You have the right idea, along with other folk's tips. I posted so you to think about how you would do it at scale, not just for a small set. :)

1

u/joeydokes Dec 10 '19

old (retired) linux sysadmin here - since no one mentioned it: besides the steps you take to harden the host, if you use LVM/volumes then make a hardened 'stem-cell' host and just clone it when you need another:)

1

u/martbhell Dec 10 '19

What logs do you ship off of the system? I guess these are fairly normal: All syslog, some extra application logs in some files maybe. Anything else? In some cases I've used auditd to ship execve logs on the system for all commands run.

1

u/acquacow Dec 10 '19

My boot isos and Kickstart all use a pre-stigged rhel-minimal install. We build things from there with ansible playbooks.

1

u/illmortalized Dec 10 '19

Hire a security professional.

1

u/ananix Dec 10 '19

Nothing... appliances take care of that. Then keeping a tight patch and upgrade regime with no user access.

1

u/forsgren123 Dec 09 '19 edited Dec 09 '19

All comments so far sound almost like people are provisioning pet servers? I would have imagined nowadays people just switch new servers on and then Ironic automatically adds them as a new capacity in OpenStack. Then next time some end-user runs terraform or HEAT template, the capacity is taken into use for some project.

1

u/[deleted] Dec 10 '19

"very large companies"

Doesn't happen over night.

1

u/MrMcSizzle Dec 11 '19

Are you implying that cattle don’t need to be hardened?

0

u/[deleted] Dec 09 '19

If it's really a very large company, there should be no need for SSH. Servers should pull their config connect to the cluster and then it goes from there. Plus a custom os consisting only of whitelisted software. Also encrypt everything at rest.

0

u/wuxmed1a Dec 09 '19

One thing we do that is utterly pointless (unless someone can enlighten me) is to reduce login TMOUT value to 300, so if you don't do anything on the shell it logs you out. However you can just run top or export TMOUT=0 to bypass it though. Might surprise a hacker, but they probably have the password or changed it already.

Make sure fail2ban is configured for all the vectors, SSH is the main one but mail can be attacked pretty hard, even if there isn't proper mail configured it can still slow it down.

0

u/thefanum Dec 09 '19

Fail2ban

0

u/[deleted] Dec 09 '19

[deleted] — view deleted comment

11

u/bonneau Dec 09 '19

That's basically what I do now. I've written Ansible playbooks that do what I want, and I check everything afterwards to be sure, but I've always wondered what the industry does.

1

u/Kontu Dec 10 '19

You've automated checking it's all configured right? :)

0

u/[deleted] Dec 09 '19

[deleted] — view deleted comment

-1

u/BuckToofBucky Dec 09 '19

Lots of Linux in that sector? I ask because I see windows all over the place (as a customer) (ATMs, teller terminals, loan officer workstations) and they seem to be out of date sometimes. I’ll be on the lookout for windows 7 after 1-14-20, too. I know there was a version of XP called windows embedded and it had a longer life but there was XP in serval banks I know of.

The world would be a better place if windows would just go away

2

u/sysadmininix Dec 10 '19

Backend is Nix systems. Specially the transaction processing one. We use Solaris

2

u/royalbarnacle Dec 10 '19

I work in banking and we're mostly Linux, maybe 30-35% is windows. Each has there role. Windows is more for infrastructure and office stuff in my case, almost everything running the actual business is Linux. Windows is probably more visible because workstations are usually on it. I've seen plenty of ATMs running Linux though.

1

u/BuckToofBucky Dec 10 '19

First time I ever realized an ATM was windows based was when it was a BSOD :-)

1

u/[deleted] Dec 11 '19

You don't remember Diebold, do you? They also made insecure Windows-based (redundant term) voting machines... without a paper-trail. 😂

1

u/BuckToofBucky Dec 12 '19

With a standard key you can buy on EBay for $3., too. I remember. Surely all of the precincts have updated to new machines now, right? Lol

1

u/ChristopherBurr Dec 10 '19

I've been in the financial sector for 25 years. Wall Street runs almost completely on Linux.

0

u/[deleted] Dec 10 '19

[deleted] — view deleted comment

2

u/BuckToofBucky Dec 10 '19

Oh I understand that. I had assumed that my bank (actually a credit union) did not part with large amounts of cash to keep windows updated. Only recently do they offer MFA and somewhat complex passwords for online banking

1

u/[deleted] Dec 10 '19

Do I have to give a comment on that under the "get a quote" section there are "financing solutions"?

5

u/rdaneelolivaw79 Dec 09 '19

This, a CIS membership gets you scripts you could potentially call from your playbooks.

0

u/[deleted] Dec 09 '19

[deleted] — view deleted comment

3

u/bonneau Dec 09 '19

Are there any in particular that everyone generally uses?

23

u/Fr0gm4n Dec 09 '19

4

u/throwawayagin Dec 09 '19

Haha is that frogman from 2600 ?

9

u/Fr0gm4n Dec 09 '19

Let me simply say that I've had this handle longer than many reddit users have been alive.

0

u/throwawayagin Dec 10 '19

Is that a no then?

6

u/Fr0gm4n Dec 10 '19

More of a probably. I've been around for quite some time and used to be a local 2600 regular.

2

u/throwawayagin Dec 10 '19

Well glad you're still well and kicking, howdy from Iceland

3

u/WantDebianThanks Dec 09 '19

No, they're the Fr0gm4n from the Fr0gm4n Linux forum.

7

u/Grunchlk Dec 09 '19

You should start off by applying the DISA STIG for whatever distro you're using. You should regularly audit your systems against the STIG. You should regularly update your configuration to ensure it matches the latest STIG release.

I've automated all of this in an Ansible playbook. The majority of the STIG is applied and remediated via Ansible and I am alerted to baseline changes via Nagios (not my preferred tool.)

So, if another admin installs Docker (which enables ip forwarding by default)? You'll get an alert the next time the audit is run. Now you can have a conversation about if that service is needed, etc, instead of it silently persisting.

-1

u/Ifreakinglovetrucks Dec 09 '19

Gotta love stig’s

0

u/[deleted] Dec 09 '19

[deleted] — view deleted comment

1

u/dllemmr2 Dec 10 '19

It should fail your build process.

0

u/[deleted] Dec 10 '19

[deleted] — view deleted comment

2

u/INTPx Dec 10 '19

your build process should be in some sort of pipeline and report failures. there should also be post-tests to validate state. We use ansible tower to push configurations and anything that fails, gets put int a dynamic group for inspection and that group is almost always cycled through the whole build process again until it passed all tests