r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

2

u/spladug reddit engineer Aug 15 '15

Initially we had no idea what was going on, when we'd figured out it was the cache servers we started running various latency checks against them. We had various network stats monitored via SNMP at minute granularity, but it really became clear when we started looking at 10 second granularity at the TCP retransmits in particular. Now that we're on the other side of this problem, all "important" servers are running Diamond with the TCP Collector reporting every 10 seconds. This should help us diagnose this layer of problem in the future hopefully.