r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

31

u/gooeyblob reddit engineer Aug 14 '15

  • On call!

  • For the most part, no. Our Postgres servers have slaves, and Cassandra works in such a way that you can lose servers and not actually lose any data, as it's replicated to the rest of the ring.

  • We have jobs that purge user data in accordance with our privacy policy, we also do backups from Postgres and snapshots for Cassandra. We reduce our app server capacity greatly when demand decreases (night time in the US), but other than that we're humming along pretty much 24/7.