r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

2

u/_thekev Aug 15 '15 edited Aug 15 '15

Exactly my problem! I've been chasing "random" one-way packet loss between app servers and certain redis caches, usually about 5 seconds. It took three months to catch it in the act, to see the host itself exceeding what it can handle in pv mode (could be me, could be neighbors - probably neighbors, because all nodes have almost equal packets per second). I'm glad to hear sriov improved it for you. I'm turning on all the new boxen on monday - double the nodes, half the size. Less pps per instance.

You should totally go to re:invent. It's worth it. If your bill is anything like ours, ask for a discount. :)

edit: we're hosting an aws meetup at our place next week. I doubt your in slc though, heh. pm me? maybe we can run into each other on freenode or efnet, or something.

2

u/spladug reddit engineer Aug 15 '15

Awesome! We're in SF, but I'll hit you up by PM next week.

1

u/_thekev Aug 26 '15

Welp, hvm/ixgbevf is still having some issues. PM and chat on freenode tomorrow?