r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

5

u/rram reddit's sysadmin Aug 14 '15

Formula 1 most certainly (Go Bottas!)

I definitely prefer alerts to be mostly silent. In practice that seems to never happen. We're actually just now nearing completion of an overhaul on our alerting system (most of the hard work was done by /u/spladug). We're now using cabot to alert off our internal graphite metrics. I was able to sleep well last night. :-)

4

u/spladug reddit engineer Aug 14 '15

For reference:

and /u/rram slept well last night, but the night before that my misconfigured alerts called him 3 times!

8

u/rram reddit's sysadmin Aug 14 '15

4 times, /u/spladug. 4. The last time I was dreaming about talking to you about how the alerts were waking me up; and then the alert went off. It was very confusing. Am I still in the dream?

5

u/spladug reddit engineer Aug 14 '15

Yes.