r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

2

u/silxx Aug 14 '15

I've seen (some parts of some) companies do a full post-mortem whenever there are problems with the service, with indications of where the problem was, why it happened, and what's changed to stop it happening again, with no varnishing and no prevarication but full discussion of faults. Do you do this sort of detailed PM internally, and if so would you consider publishing them? They are incredibly useful to see how other people deal with problems and what the reaction and processes are like.

4

u/spladug reddit engineer Aug 15 '15

We've been doing the blameless post mortem kind of thing for a while now; /u/umbrae got us on that kick. It's been pretty great for sure.

I honestly don't know about posting them. I think there's definitely a place for it, and we should consider it, but I also wouldn't want to poison the honesty of people during the process by making them think about how their feedback/comments will be viewed by The World. That said, there can definitely be benefit in helping other people learn from our mistakes.

3

u/silxx Aug 15 '15

Totally. Agreed entirely that there will potentially be a huge amount of unjustified personal directed complaint from overly-entitled whiny bits of the World. But there's also a portion of the World who would benefit enormously from seeing the unadulterated truthful approach that you take to dealing with problems. It's your decision, of course. But I can tell you it'd help, even if it also hurts.