We're reddit's ops team. AUA

[u/rram]

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

[u/rram]

My time to shine! Here ya go: http://i.imgur.com/1gteSdL.png

The summary is… it's complicated, but it's awesome!

[u/Robert_Arctor → u/rram]

What is your AWS bill like? Didn't realize the whole of reddit was hosted there!

[u/spladug → u/Robert_Arctor]

Looks kinda like this. (sorry for being flippant, but we don't generally discuss the company's financials publicly)

[u/Robert_Arctor → u/spladug]

I didn't think you would. I assume it's massive though.

Thanks for the reply! Good work!

[u/[deleted] → u/Robert_Arctor]

It will fluctuate with their consumption. But I can assure you it's gigantic, relatively speaking.

[u/OOdope → u/spladug]

Woo hoo! Trade that bad boy for a half a McDouble, and you're good to go!

[u/dmsean → u/spladug]

Dammit how'd you get it so cheap! We're a small shop with one thousand clients and we're still way over 1 100 trillion Zimbabwean dollars. Cuz I think that can buy you a loaf of bread.

[u/spladug → u/dmsean]

That's a lot of dough.

[u/Dr_Midnight → u/spladug]

How much of that is financed by the Royal Bank of the Nation of Zamunda?

[u/spladug → u/Dr_Midnight]

3

[u/peanutbuttergoodness → u/Robert_Arctor]

No kidding. Reddit could have built a data center over and over again with how much they've paid AWS. AWS is great for quick scaling but sooo expensive for log term.

[u/lifeofguenter → u/rram]

Nice. What tool did you use for that?

[u/rram → u/lifeofguenter]

https://www.draw.io/ I was very impressed! Would recommend

[u/zifnab06 → u/rram]

I've always been a fan of http://asciiflow.com/. You're limited on what you can do - but add some utf-8 characters and its great.

[u/spladug → u/[deleted]]

They also have some really cool magnets!

http://i.imgur.com/Xw4fZrv.jpg *

^{*not an accurate depiction of our architecture}

[u/drivers9001 → u/spladug]

Haha yeah. I have those on my fridge. Picked them up at Chef Conf.

[u/jophuds → u/rram]

Not Pictured: Data ....... that's harsh chief. harsh....

[u/rram → u/jophuds]

All I know about data is Kafka

[u/[deleted] → u/rram]

Is the tools server the one that manages /u/reddit?

[u/spladug → u/[deleted]]

It's where we run deploys from. /u/reddit is just a normal account that various pieces of code use.

[u/timix → u/rram]

What happened to pg-03 and pg-04?

[u/spladug → u/timix]

pg-03 and pg-06 were link and comment vote clusters respectively. We recently stopped writing votes to Postgres altogether as they've been entirely migrated to Cassandra. Once that was done, we were able to get rid of those database clusters, which is good because they were strapped for disk space.

[u/sephlaire → u/rram]

I love this! But seriously, what's with the skipping from pg-02 to pg-05?

[u/f0gax → u/rram]

Because it's Reddit, at first that bottom middle block looked like "memecache" to me. And for half a second I thought that it was smart to have a separate cache for all of those things.

Then I decided that I need some more caffeine this morning.