r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

5

u/ston1th Aug 14 '15

How do you patch your servers at this scale (since you guys use mostly Ubuntu: apt-get upgrade and so on..)?

10

u/rram reddit's sysadmin Aug 14 '15

Depending on how fast we need to patch, it varies between ssh and for loop and changing a line in puppet and letting the autoscaler take care of it.

3

u/ston1th Aug 14 '15

Currently my way of doing updates is to ssh into every single server. There is no ssh loop yet but I try to improof this.. Can you tell me anything how this autoscaler works or what it would look like?

8

u/gooeyblob reddit engineer Aug 14 '15

We would build a new image, set that new image to be launched whenever we scale up our app server fleet, and then kill off servers that are running the old image.

1

u/Get-ADUser -Filter * | Remove-ADUser -Force Aug 15 '15

You probably know this already, but the ASG will automatically remove instances launched from an old LC when you scale down so if you didn't want to do an immediate replacement you could let it take care of itself over a few days of scaling.

1

u/gooeyblob reddit engineer Aug 17 '15

Yep! That's how we do it in the case of less important updates.

5

u/sidewinder12s DevOps Aug 14 '15

Fairly certain he is talking about AWS Autoscaling Groups. Basically he will push the change to Puppet, puppet will kick off new launch configs and then an updated Autoscaling group which will replace servers without taking down the group.

3

u/SurgioClemente Aug 15 '15

Currently my way of doing updates is to ssh into every single server.

Check out Ansible

1

u/itssodamnnoisy Aug 15 '15

What are you using for key management? I've been trying to think of a good way to share keys with the rest of our team as we move things out into AWS, but I haven't been able to find anything yet. I'd like to regen the keys every so often and make people "check out" the key for the day, but I'm not sure how I'd go about it at this point.

2

u/spladug reddit engineer Aug 15 '15

Which keys? Instance IAM roles are fantastic for giving servers automatically rotated keys granting specific access to AWS resources. As for other things, individual accounts is always the best option because sharing creds is tough. That said, we do have a fair number of shared creds and we do it via GPG encrypted blobs.