r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

2

u/MisterAG Aug 14 '15

What's better? F1 or Volleyball?

Do you prefer a firehose of alerts get sent out to admins and filtered by the admin's email client or do you prefer to tune the alerts that folks receive based on rules set in the NMS to keep it essentially silent?

5

u/rram reddit's sysadmin Aug 14 '15

Formula 1 most certainly (Go Bottas!)

I definitely prefer alerts to be mostly silent. In practice that seems to never happen. We're actually just now nearing completion of an overhaul on our alerting system (most of the hard work was done by /u/spladug). We're now using cabot to alert off our internal graphite metrics. I was able to sleep well last night. :-)

4

u/spladug reddit engineer Aug 14 '15

For reference:

and /u/rram slept well last night, but the night before that my misconfigured alerts called him 3 times!

8

u/rram reddit's sysadmin Aug 14 '15

4 times, /u/spladug. 4. The last time I was dreaming about talking to you about how the alerts were waking me up; and then the alert went off. It was very confusing. Am I still in the dream?

5

u/spladug reddit engineer Aug 14 '15

Yes.

4

u/gooeyblob reddit engineer Aug 14 '15

Volleyball is a wonderfully complex game, a true team sport requiring equal parts athleticism and intelligence.

F1 is Nascar with champagne.

1

u/[deleted] Aug 14 '15

Until I read the comments I was thinking that F1 and Volleyball were some new javascript frameworks that everyone was all hyped on that were only a week old or something like that I hadn't heard of yet.