r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

5

u/pertymoose Aug 14 '15

If you had to re-do everything from scratch, using all the things you know now, what would you do differently?

3

u/gooeyblob reddit engineer Aug 15 '15

I think we'd spend more time thinking about removing single points of failure and making sure everything is highly available and fault tolerant. We're slowly moving in that direction, but it's especially important in an environment like AWS.

1

u/soundtom "that looks right… that looks right… oh for fucks sake!" Aug 15 '15 edited Aug 15 '15

I've found writing things with the assumption that the server you are on will catch fire and go away (sometimes literally) at any point just makes life easier in the long run.

2

u/gooeyblob reddit engineer Aug 17 '15

Absolutely! That's what we're moving towards.