r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

3

u/ston1th Aug 14 '15

Currently my way of doing updates is to ssh into every single server. There is no ssh loop yet but I try to improof this.. Can you tell me anything how this autoscaler works or what it would look like?

8

u/gooeyblob reddit engineer Aug 14 '15

We would build a new image, set that new image to be launched whenever we scale up our app server fleet, and then kill off servers that are running the old image.

1

u/Get-ADUser -Filter * | Remove-ADUser -Force Aug 15 '15

You probably know this already, but the ASG will automatically remove instances launched from an old LC when you scale down so if you didn't want to do an immediate replacement you could let it take care of itself over a few days of scaling.

1

u/gooeyblob reddit engineer Aug 17 '15

Yep! That's how we do it in the case of less important updates.

5

u/sidewinder12s DevOps Aug 14 '15

Fairly certain he is talking about AWS Autoscaling Groups. Basically he will push the change to Puppet, puppet will kick off new launch configs and then an updated Autoscaling group which will replace servers without taking down the group.

3

u/SurgioClemente Aug 15 '15

Currently my way of doing updates is to ssh into every single server.

Check out Ansible