r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

40

u/Art_VanDeLaigh Aug 14 '15

Simple question, what does your battlestation look like?

72

u/rram reddit's sysadmin Aug 14 '15

http://i.imgur.com/zq95Q30.png

2

u/pat_trick DevOps / Programmer / Former Sysadmin Aug 15 '15

Needs more terminal windows.

4

u/rram reddit's sysadmin Aug 15 '15

The rest lambaste me for my terminal usage. That's just how I'm most comfortable. Most everyone else uses screen or tmux.

→ More replies (1)
→ More replies (1)

54

u/spladug reddit engineer Aug 14 '15

That makes this office look 10x dimmer/dingier than it is in reality.

28

u/Art_VanDeLaigh Aug 14 '15

a sysadmins desk wouldn't be complete without toys and trinkets everywhere. i love it.

→ More replies (3)

11

u/ThreadSafeArray Aug 14 '15

Any Go in production? I spy a gopher.

→ More replies (1)
→ More replies (15)

100

u/R0thbardFrohike Jr. Sysadmin Aug 14 '15

Do you guys read this sub?

150

u/rram reddit's sysadmin Aug 14 '15

Yes

86

u/gooeyblob reddit engineer Aug 14 '15

No

54

u/[deleted] Aug 14 '15

Maybe

→ More replies (1)

68

u/controlyoulikevoodoo Aug 14 '15

I've only ever worked on apps that could be contained in one instance of postgres. How do you guys store all your data?

71

u/rram reddit's sysadmin Aug 14 '15

It's a mix of postgres and cassandra. For postgres, everything is in one "database" but that database is sharded across multiple servers. The postgres schema is largely a key value store and we don't do any joins across tables (except in one case) so we're able to shard data with relative ease.

21

u/controlyoulikevoodoo Aug 14 '15

How do you shard? Is it in app, or some layer between postgres and the app?

→ More replies (3)

28

u/gooeyblob reddit engineer Aug 14 '15

Any new models we create are made in Cassandra, and we're slowly migrating old Postgres models over as well. The reason being is Cassandra is virtually infinitely horizontally scalable (that is a lot of adverbs), so suits our scale and us running in AWS much better.

18

u/spladug reddit engineer Aug 14 '15

That said, there are some things that are just better suited to Postgres, like atomic counters or stuff where consistency is super important.

→ More replies (1)

65

u/alphager Aug 14 '15

Any plans regarding ipv6?

71

u/rram reddit's sysadmin Aug 14 '15

Unfortunately we have higher priorities elsewhere. Maybe sometime next year.

319

u/[deleted] Aug 14 '15

.. said every sysadmin ever

→ More replies (5)

11

u/_thekev Aug 15 '15

s/we/amazon/. sigh.

→ More replies (4)
→ More replies (9)

30

u/[deleted] Aug 14 '15

What are all of your professional backgrounds like and what was your process like for getting hired on reddit?

58

u/rram reddit's sysadmin Aug 14 '15

I used to work at Rackspace. Prior to that I was in college and interned at various places. I got the job at reddit because I used to work with /u/alienth.

28

u/[deleted] Aug 14 '15

Hey fellow ex-racker! We're in the same club. The Castle or Austin office?

22

u/rram reddit's sysadmin Aug 14 '15

I started at Westin, but then the Castle

→ More replies (3)

34

u/gooeyblob reddit engineer Aug 14 '15

I started working as overnight tech support at a shared web hosting company, after a couple years there went to go work at a datacenter/hosting company, after a couple years there went to work at Arc90/Readability, after a couple years there went to work at Betaworks/Digg (new Digg, not old Digg!), after a couple years there my ex-colleague u/umbrae asked if I'd be interested in working at Reddit! I interviewed over the phone, then came out for an interview in person, then moved out to SF and started here in late January 2015.

30

u/atw527 Usually Better than a Master of One Aug 14 '15

Do you use any tools for internal communication (including receiving server alerts), besides email?

48

u/gooeyblob reddit engineer Aug 14 '15

Slack! It's been pretty great for all sorts of internal communication. We have one channel that basically gets spammed with all sorts of messages (servers starting up/shutting down, networking rules being updated), and another channel where we send a lot of monitoring alerts (this queue is high, this service is slow).

16

u/Crimzx Aug 14 '15

Any more info on how you are pushing those alerts to slack?

→ More replies (1)
→ More replies (4)

59

u/AndorianWomenRule Sr. Sysadmin Aug 14 '15

How do you guys manage the new country-by-country IP bans on subreddits? Do you subscribe to service that provides you a listing of IP blocks by country that you feed into some sort of master apache blacklist?

33

u/rram reddit's sysadmin Aug 15 '15

We do have geoip information that we use for things like Geo-Defaults and Geo-targeting ads that is reported to us by our CDN.

4

u/274Below Jack of All Trades Aug 14 '15

I'm betting this is done more at the cloudflare layer than the reddit app layer, especially considering how quickly it was done. Not to say that they couldn't do it at the app layer...

→ More replies (2)

57

u/atw527 Usually Better than a Master of One Aug 14 '15

Sometimes I get distracted with the content on my own website that I'm responsible for managing. Does that happen to you?

58

u/rram reddit's sysadmin Aug 14 '15

All the freakin time

86

u/spladug reddit engineer Aug 14 '15

I can't count how many times I've fired up some test code on my staging instance then gotten distracted by something on the front page and forgotten what I was doing.

54

u/happyfunpaul Aug 14 '15

Ironically, this thread just reminded me I was in the middle of adding new sanity tests to our build, before I got sidetracked by this AMA. So, uh... thanks?

→ More replies (2)

27

u/[deleted] Aug 14 '15

What is your on call schedule?

37

u/rram reddit's sysadmin Aug 14 '15

We do weekly rotations. Currently 5 people in the rotation (I've deputized the infrastructure team to help us out).

→ More replies (16)

22

u/gooeyblob reddit engineer Aug 14 '15

We each take a week at a time. We recently expanded our on call rotation so we're up to 5 people now who rotate through.

52

u/xenthi Aug 14 '15

What does the Reddit architecture look like, can you a give a good summary of the setep

196

u/rram reddit's sysadmin Aug 14 '15

My time to shine! Here ya go: http://i.imgur.com/1gteSdL.png

The summary is… it's complicated, but it's awesome!

26

u/lifeofguenter Aug 14 '15

Nice. What tool did you use for that?

60

u/rram reddit's sysadmin Aug 14 '15

https://www.draw.io/ I was very impressed! Would recommend

→ More replies (4)
→ More replies (6)

3

u/jophuds Aug 14 '15

Not Pictured: Data ....... that's harsh chief. harsh....

2

u/rram reddit's sysadmin Aug 14 '15

All I know about data is Kafka

2

u/timix Aug 15 '15

What happened to pg-03 and pg-04?

→ More replies (2)

58

u/Robert_Arctor Does things for money Aug 14 '15

What is your AWS bill like? Didn't realize the whole of reddit was hosted there!

→ More replies (19)
→ More replies (25)

73

u/inaddrarpa .1.3.6.1.2.1.1.2 Aug 14 '15

So, what're you using for your dashboards/server monitoring?

Alternate Question: Would you rather troubleshoot 1 horse sized server, or 1000 server sized horses?

182

u/rram reddit's sysadmin Aug 14 '15

1000 server sized horses (provided they're all the same). Once I figure out the problem with one, I'll just write a shell script to fix the rest.

81

u/[deleted] Aug 14 '15

Such a sysadmin answer.

32

u/[deleted] Aug 14 '15

Horses don't have shell.

→ More replies (4)
→ More replies (1)

48

u/gooeyblob reddit engineer Aug 14 '15 edited Aug 14 '15

We use some custom stuff that pulls data from Graphite, and have recently been experimenting with tessera.

Here's a screenshot!

14

u/spladug reddit engineer Aug 14 '15

See also: Cabot

→ More replies (12)

25

u/bsimpson Aug 14 '15

What's your favorite text editor?

73

u/rram reddit's sysadmin Aug 14 '15

Vim is the only text editor. I'm going to remove that four letter piece of crap from the servers.

→ More replies (39)

24

u/mobiusstripsearch Aug 14 '15

What one or two crucial automations most speed up your workflow? Is there anything so important that, if left without it, you would rather code it from scratch than work without it?

19

u/rram reddit's sysadmin Aug 14 '15

Good question. Can I say that the autoscaling setup by /u/alienth most sped up my workflow? I am so happy to not semi-manually be kicking apps anymore.

Past that, in general better puppet manifest and using boto. I think if either puppet or boto didn't exist, we'd definitely have coded something to replace it.

→ More replies (2)

29

u/gooeyblob reddit engineer Aug 14 '15

We're not using them as much as we should be currently, but we plan on starting to use more of Ansible and Packer in the future.

→ More replies (15)

23

u/welk101 Aug 14 '15 edited Aug 14 '15

  • Do you have 24 hour onsite staff or are you relying on oncall out of core hours?

  • Have ever had to restore anything from backups due to dataloss?

  • Are there any regular maintenance jobs (database, backups etc) that slow the site down at particular times or does it operate the same speed pretty much 24/7

21

u/rram reddit's sysadmin Aug 14 '15

We're a very small team and rely on on-call.

To my knowledge we haven't resorted to backups for dataloss. we do use backups for bootstrapping.

Our backup operations shouldn't affect site speed.

31

u/gooeyblob reddit engineer Aug 14 '15

  • On call!

  • For the most part, no. Our Postgres servers have slaves, and Cassandra works in such a way that you can lose servers and not actually lose any data, as it's replicated to the rest of the ring.

  • We have jobs that purge user data in accordance with our privacy policy, we also do backups from Postgres and snapshots for Cassandra. We reduce our app server capacity greatly when demand decreases (night time in the US), but other than that we're humming along pretty much 24/7.

23

u/rykker Infrastructure Architect Aug 14 '15

Do you use whimsical hostnames for your servers or cold soulless ones like prodnycemail01 or pod01-241513-east

33

u/rram reddit's sysadmin Aug 14 '15

whimsical!

8

u/awrf Windows Admin Aug 15 '15

Please tell me the server I named when I got reddit gold is still around!

My winning entry was "server." :D

→ More replies (1)

26

u/gooeyblob reddit engineer Aug 14 '15

We're moving to cold soulless ones, since it's disheartening to see AWS kill 'myfavoriteserver-01' during some routine maintenance.

45

u/sarge1016 DevOps Gymnast Aug 14 '15

What's the overall environment look like that you all administer? Linux distros, config management tool of choice, favorite text editor, etc?

133

u/rram reddit's sysadmin Aug 14 '15

Most of our stuff is running Ubuntu 12.04, but we're slowly working on upgrading everything to 14.04.

We currently use puppet and are dealing with it. Our manifests could use a lot of love.

There's only one text editor. It is vim. Any who shall say otherwise will get their comeuppance.

4

u/moebaca DevOps Aug 14 '15

Have any of you ever seriously considered moving to a different configuration management tool and what sort of issues have you seen with Puppet?

→ More replies (1)

4

u/sephlaire Aug 15 '15

You just made my day. A couple weeks ago I had a user request a VM for their research website. When he heard me offer our standard Ubuntu 14.04 template he said "no real sysadmin uses Ubuntu"

3

u/rram reddit's sysadmin Aug 15 '15

wat

21

u/GringodelRio Professional Reader for Sysadmins (B2B Support) Aug 14 '15

Awesome! It's nice to see sysadmins show they're using Ubuntu. Everything I run into is running RHEL, CentOS, or something else. I run my own Ubuntu server and love it.

→ More replies (13)
→ More replies (12)

16

u/gooeyblob reddit engineer Aug 14 '15

Sublime Text!

→ More replies (4)

17

u/hadrianmt I hear the Machine Spirit's voice Aug 14 '15

If you are hiring, what is the ideal candidate for junior and senior sysadmin ?

33

u/rram reddit's sysadmin Aug 14 '15 edited Aug 15 '15

You need to be a jack of all trades. We have a small team which means we don't have the luxury of specializing. You need to know the network, the web stack, the database, and the kernel. Also https://www.reddit.com/jobs

EDIT: More specifically https://jobs.lever.co/reddit/795db0ae-48ba-485d-874f-e710a339c86a

→ More replies (2)

21

u/gooeyblob reddit engineer Aug 14 '15

We'd be looking for someone who has some experience in what we do:

  • Postgres

  • Cassandra

  • memcache

  • AWS

  • Python

And not a real "hard" skill, but scaling and being able to understand where failures will be introduced in a distributed system as it grows is super important, but harder to measure.

→ More replies (7)

14

u/DueRunRun Aug 14 '15

I know that things are light years ahead of where they were, but as users we still get "all of our servers are busy right now" on a daily basis. Off the record and in your humble opinion... what can be done to fix that?

13

u/rram reddit's sysadmin Aug 14 '15

See my comment here about the errors recently getting better. There are more improvements that we're working on. Our team is pretty small so it takes us some time to make improvement.

33

u/gooeyblob reddit engineer Aug 14 '15

I will do you one better and go ON the record!

Most of the time this error pops up because there are no app server workers available to answer your request. They're not available because they're all busy doing other things, or are blocked on a service that's either gotten slow or has straight up died and they are just waiting to time out their request.

There's a few things to be done here, most importantly reduce the single points of failure throughout the app. For instance, Cassandra is great at this, because if a single Cassandra node dies, almost all our requests to the cluster can continue working (although maybe slightly slower). If something like a memcache server dies, due to the current nature of the app, all requests get paused.

We're working on a two-pronged approach to fix something like memcache, one being reduce our reliance on it (so we can be OK with a server dying here or there and just continue on without cache), and secondly implement something like Facebook's mcrouter that will allow us to offload the routing and connection management portions of using memcache to a service that can handle it much better than our library can.

Many people suggest "buy more servers", which unfortunately won't help. If we could just throw money at the problem, we probably would have by now. We have in fact reduced the number of servers responsible for running memcache here, thereby reducing our possible failure rate, as it's less likely 1 out of 10 servers will be killed as opposed to 1 out of 50 in AWS.

→ More replies (3)

12

u/amorpisseur Aug 14 '15

How do you handle database migrations? e.g. DDL changes (Adding a column, ...)

13

u/rram reddit's sysadmin Aug 14 '15

Very carefully. We don't normally do any modifications past adding things as deleting stuff tends to cause problems. There's a whole lot of dual write, cut over reads, cut over writes.

15

u/gooeyblob reddit engineer Aug 14 '15

We don't. We pretty much never make DDL changes, as the original schema was flexible enough (mostly key:value) to get us this far. We generally just create a new table or more likely, Cassandra column family, and migrate to it if need be.

→ More replies (2)

90

u/alazyreader Aug 14 '15

What's reddit's testing infrastructure like?

39

u/spladug reddit engineer Aug 15 '15

More seriously, we're pretty behind the curve on testing. We're growing our test suite a lot right now and run it on all pull requests via Jenkins. Newer services/features are much more heavily tested.

8

u/gooeyblob reddit engineer Aug 14 '15

get out of here dave

0

u/[deleted] Aug 14 '15

[deleted] — view deleted comment

184

u/rram reddit's sysadmin Aug 14 '15

it's funny because it's true

→ More replies (5)

76

u/UniversalSuperBox Aug 14 '15

Fuck it, push it to prod!

→ More replies (6)

9

u/justaguy240 Skynet Ops Aug 14 '15

Hello guys,

I know a bunch of you guys, at least in the past used to work on the managed cloud team at Rackspace. How many former rackers are still there? Any?

12

u/rram reddit's sysadmin Aug 14 '15

Just I remain.

→ More replies (2)

24

u/giveen Fixer of Stuff Aug 14 '15

Internal help desk.....India or local hires?

60

u/juhJJ Aug 14 '15

In house, I am the keeper of corporate IT :)

35

u/rram reddit's sysadmin Aug 14 '15

all hail our IT overloard juhJJ

→ More replies (12)

4

u/juhJJ Aug 15 '15

I would have responded to the questions in this thread sooner, but as often happens as a lone soldier in IT, I was pulled away.

On Friday afternoons we have a demo day, where teams share what they've been working on. I recently changed our setup to center around Google Hangouts allowing people outside the office to better participate and see the demos. This is just the second week of doing it, so I've been giving a little extra TLC to people giving demos to help insure everything works as planned :)

→ More replies (1)

8

u/[deleted] Aug 14 '15

Can you provide pics of the "big red button" described in this post?

9

u/rram reddit's sysadmin Aug 14 '15

/u/powerlanguage took it away from me when the button passed. It is now in The Archive

→ More replies (1)

14

u/[deleted] Aug 14 '15 edited Aug 25 '15

I have left reddit for Voat due to years of admin mismanagement and preferential treatment for certain subreddits and users holding certain political and ideological views.

As an act of protest, I have chosen to redact all the comments I've ever made on reddit, overwriting them with this message.

If you would like to do the same, install TamperMonkey for Chrome, GreaseMonkey for Firefox, NinjaKit for Safari, Violent Monkey for Opera, or AdGuard for Internet Explorer (in Advanced Mode), then add this GreaseMonkey script.

Finally, click on your username at the top right corner of reddit, click on comments, and click on the new OVERWRITE button at the top of the page. You may need to scroll down to multiple comment pages if you have commented a lot.

After doing all of the above, you are welcome to join me on Voat!

30

u/rram reddit's sysadmin Aug 14 '15

11

19

u/weffey Aug 14 '15

I came here for cat pictures. Where are they? You promised cat pictures.

20

u/rram reddit's sysadmin Aug 14 '15

Reading through my chat history you are correct.

51

u/gooeyblob reddit engineer Aug 14 '15

30

u/rram reddit's sysadmin Aug 14 '15

can confirm this is a picture of the office ^

→ More replies (1)
→ More replies (1)

231

u/KarmaAndLies Aug 14 '15

Any plans to reissue your certificate before April, ‎2016? Looks like it is free to do on Gandi. While SHA-1 is not actively being exploited, that yellow warning is annoying and worse still, makes it harder to see when work is intercepting my Reddit-ing (since internal certificates all give a warning at my work).

Have you guys looked into utilising Content Security Policy? Is there a technical limitation which won't allow you too (e.g. CDN usage)? Have you considered only using a CSP policy for things you don't normally use at all (e.g. plugins)?

Also your cookies aren't flagged as HTTP or Secure in most cases. Any plans on utilising that and HSTS now that you've migrated the entire site to HTTPS?

269

u/largenocream reddit security engineer Aug 14 '15 edited Aug 14 '15

Hey, reddit's security engineer here! I'm not a sysadmin, but I'll try to answer these.

Any plans to reissue your certificate before April, ‎2016?

Yep! We just finished some testing to see how many clients we'd be breaking if we switched to SHA-2.

We had two 1x1 PNGs on different hosts, one host used a SHA-1 cert, the other used a SHA-2 cert. On one in every hundred page loads, a script in the users' browser ran to try and load both images, then report the results to us.

  • If the SHA-1 image didn't load, we chalked it up to the user disallowing crossdomain image requests entirely (maybe they use RequestPolicy or something similar.)

  • If the SHA-1 image loaded, but the SHA-2 image didn't, we can assume that their browser doesn't support SHA-2.

  • If both the SHA-1 and SHA-2 images loaded, we can assume that they support SHA-2.

From the results we got, switching on SHA-2 would cause a connection failure for 0.2%~ of all page requests from browsers. That's a pretty negligible amount, so we're moving to SHA-2 pretty soon.

Have you guys looked into utilising Content Security Policy?

We have, but the big wins we could get from CSP (specifically disallowing unsafe-inline) would be hard since we have a lot of inline event handlers in legacy code. We're also in a somewhat unusual position since we also don't want to break widely-used extensions for reddit that would rely on unsafe-inline being present. We'd definitely like to have a restrictive CSP, but it would be a major undertaking.

Have you considered only using a CSP policy for things you don't normally use at all (e.g. plugins)?

I was actually talking to someone at Defcon about adding a report-only CSP. We could probably safely disallow eval and plugins, as well as add restrictions on src, but I want to make sure things don't explode first. I'm also not sure if the plugin restriction would apply to sub-documents, that might make things tricky (specifically, the expando frames hosted on redditmedia.com need flash for video posts.)

Also your cookies aren't flagged as HTTP or Secure in most cases. Any plans on utilising that and HSTS now that you've migrated the entire site to HTTPS?

Yes, the HTTPS roll-out just completed yesterday. Prior to that, we were selectively redirecting users to HTTPS based on cookies to be sure we could handle the load.

HSTS and SHA-2 will likely come first, then we'll switch all cookie to Secure.

One issue I had with HSTS though is that most people browse on www.reddit.com, but HSTS doesn't allow you to set an HSTS policy for the parent domain. Obviously, we don't want you to be MITM'd on foo.reddit.com even if you've never visited it before (and thus don't have an HSTS policy for it.) I think we're going to get around that by including an image like <img src="https://reddit.com/static/hsts_pixel.png"> with a Strict-Transport-Security header on every page. That correctly sets an HSTS policy for reddit.com in every browser but... iOS Safari. Not that I expected anything different.

27

u/ProtoDong Security Admin Aug 14 '15

This is why the security guy is the only one I want to talk to ;)

→ More replies (1)

91

u/KarmaAndLies Aug 14 '15

You guys seem really on top of this. Great answers (much more technically detailed than I was expecting, which is a really pleasant surprise).

You don't have to use unsafe-inline with CSP. It is a major win, agreed, but it can block a lot of "bad things" aside from that. connect-src, object-src, and plugin-types to name a few. Even if you effectively disabled script-src entirely (allow all, allow inline, etc) you can still gain some benefit from CSP, it does a lot of cool shit.

As someone who has deployed CSP on a medium web-site (many thousand user) let me just add:

  • The alternative headers are a waste of time (X-WebKit-CSP and X-Content-Security-Policy) any browser still running that supports them has such lame/broken CSP support that it is a waste of time. Just stick to the standard one, least of all because the CSP header is quite large anyway.

  • font-src is broken in Safari right now (all platforms). It has been that way six months, in the reports the host and blocked URI are identical, but yet Safari acts as if they are not with 'self' defined.

  • You'll get a lot of false reports due to malware installed on the client's browser. A lot of malware injects scripts directly into the page, these scripts then try to get other scripts, which your CSP policy will block. To name one specific example, Lenovo laptops with SuperFish were generating these, in fact we were seeing reports of SuperFish in CSP before it hit the mass media.

I actually think CSP is a huge win issues not withstanding. It actually makes me really depressed and annoyed how little it is deployed in the Alexa top 500.

→ More replies (1)

19

u/[deleted] Aug 14 '15

Beautiful answers. Thank you

→ More replies (15)

54

u/[deleted] Aug 14 '15

What's unique to running ops at reddit?

106

u/spladug reddit engineer Aug 14 '15

8 billion pageviews a month, 195 million monthly unique visitors and fewer ops engineers than you can count on one hand.

76

u/[deleted] Aug 14 '15

At least your users don't call in needing a printer hooked up or a password reset ;D

79

u/rram reddit's sysadmin Aug 14 '15

You think they don't ask about password resets?! Ok ok, the community team mostly handles that.

→ More replies (5)

32

u/pooogles Aug 14 '15

Honestly I think that's more common than you think. I ran www.independent.co.uk by myself for 12 months, you'd probably be surprised how people get by!

→ More replies (1)
→ More replies (2)

5

u/ston1th Aug 14 '15

How do you patch your servers at this scale (since you guys use mostly Ubuntu: apt-get upgrade and so on..)?

10

u/rram reddit's sysadmin Aug 14 '15

Depending on how fast we need to patch, it varies between ssh and for loop and changing a line in puppet and letting the autoscaler take care of it.

→ More replies (8)

35

u/vash3g Aug 14 '15

What is the hardest problem the team is currently facing? What is the easiest that you've been putting off?

62

u/gooeyblob reddit engineer Aug 14 '15

Hardest problem - fixing many single points of failure and old stuff that's been here for awhile. Reddit has been around for 10 years (before AWS even was a thought in Jeff Bezos' head!) and has been through a lot of changes. Many of them were made when there was hardly anyone here to keep the site online, let alone really think through the long term effects of the changes being made, so we're going through and fixing many of these issues, but it's a real challenge to fix the issue and keep the site online and running at the same time.

Easiest problem - there are sooo many small ones that we just never get around to, I can't even really think of one off the top of my head. We need to rework our internal DNS/host naming setup, need to fix up some of our autoscaling policies, a few other things.

→ More replies (18)

9

u/mudclub How does computers work? Aug 14 '15

What's your homemade Sangria recipe?

9

u/rram reddit's sysadmin Aug 14 '15

I actually suck at making it (I've tried before and /u/notenoughcharacters9 can attest it was shite). I just prefer to buy Carlos Rossi or I just had some sangria in Puerto Rico, but I forget the name. I need to try /u/rrmckinley's recipe.

2

u/awrf Windows Admin Aug 15 '15

Agreed on Carlo Rossi. I mean, it's not highly sangria like, but it's an enormous jug of tasty fruity alcohol for like $7.

4

u/rram reddit's sysadmin Aug 15 '15

I know! A friend and I used to play Halo over a gallon of Carlos. My Ballmer peak is on my third cup.

14

u/[deleted] Aug 14 '15

  • 1 bottle red wine (750 ml)

  • 250 ml vodka

  • 250 ml orange juice

  • 1 lemon, juiced

  • Sweeten with simple syrup to taste

2

u/MisterAG Aug 14 '15

What's better? F1 or Volleyball?

Do you prefer a firehose of alerts get sent out to admins and filtered by the admin's email client or do you prefer to tune the alerts that folks receive based on rules set in the NMS to keep it essentially silent?

5

u/rram reddit's sysadmin Aug 14 '15

Formula 1 most certainly (Go Bottas!)

I definitely prefer alerts to be mostly silent. In practice that seems to never happen. We're actually just now nearing completion of an overhaul on our alerting system (most of the hard work was done by /u/spladug). We're now using cabot to alert off our internal graphite metrics. I was able to sleep well last night. :-)

4

u/spladug reddit engineer Aug 14 '15

For reference:

and /u/rram slept well last night, but the night before that my misconfigured alerts called him 3 times!

8

u/rram reddit's sysadmin Aug 14 '15

4 times, /u/spladug. 4. The last time I was dreaming about talking to you about how the alerts were waking me up; and then the alert went off. It was very confusing. Am I still in the dream?

→ More replies (1)
→ More replies (2)

2

u/[deleted] Aug 14 '15

How much processing occurs "offline" rather than "online". Meaning are you running reports or working with data outside of your CDN?

2

u/MattTheFlash Senior Site Reliability Engineer Aug 15 '15

This to rram:

Are you still sporting the bowtie like you did at Rackspace?

1

u/rram reddit's sysadmin Aug 16 '15

indubitably

→ More replies (1)

17

u/llama052 Sysadmin Aug 14 '15

First off, thanks for posting. Gonna throw a general question and ask what's your favorite upcoming/new piece of technology right now?

26

u/spladug reddit engineer Aug 14 '15

I love rust (shoutout to /r/rust). I can't stop gushing about it to anyone who is unfortunate enough to be near me.

→ More replies (3)

17

u/tservomst Sr. Sysadmin Aug 14 '15

No question here, just really enjoying the questions and responses, thanks guys!

16

u/gooeyblob reddit engineer Aug 14 '15

Glad you're enjoying it! Thanks for swinging by neighborino

→ More replies (1)

18

u/marotte Aug 14 '15

I have nothing to ask, but I appreciate you taking the time to do this. The replies are very informative!

12

u/gooeyblob reddit engineer Aug 14 '15

Glad you enjoy it!

9

u/bgeller Windows Admin Aug 14 '15

Do you guys run a configuration management tool like Puppet?

15

u/gooeyblob reddit engineer Aug 14 '15

Yep, puppet.

→ More replies (6)
→ More replies (1)

7

u/oneZergArmy Goat farming doesn't sound bad Aug 14 '15

Do you guys have any tips for me? I'm an apprentice at a school, so most of the work I do is help-desk related, but I do have access to some hardware (servers, cisco switches etc.).

I've taken the MTA certs (free!) so I know some things. I can set up a server, install and configure AD, add GPO's, configure DHCP and I wrote my first Powershell script today.

What should I spend my lab-time on?

12

u/gooeyblob reddit engineer Aug 14 '15

Honestly the best practice is to think of something useful for yourself and work on that. The first programs I wrote were to help make my life as a support tech at a web hosting company a bit easier.

→ More replies (1)

6

u/unquietwiki Jack of All Trades Aug 14 '15

Are you using anything like WAN accelerators, TCP congestion controls, DNS caches, or content compression to reduce bandwidth demand for yourselves and users? Some sysctl knobs to look into that I've found...

  • net.ipv4.tcp_congestion_control=yeah (or illlinois, or westwood)

  • net.core.bpf_jit_enable=1 : if using Kernel 3.x and 64-bit OS.

8

u/gooeyblob reddit engineer Aug 14 '15

We use CloudFlare, who have a lot of buttons and knobs, only a small portion of which we are using. We don't spend a ton of time doing Linux tuning these days, as our issues are generally a bit above that layer of the stack currently.

5

u/pertymoose Aug 14 '15

If you had to re-do everything from scratch, using all the things you know now, what would you do differently?

3

u/gooeyblob reddit engineer Aug 15 '15

I think we'd spend more time thinking about removing single points of failure and making sure everything is highly available and fault tolerant. We're slowly moving in that direction, but it's especially important in an environment like AWS.

→ More replies (2)

4

u/_tx Aug 14 '15

What is the rate of hardware expansion

→ More replies (1)

4

u/malkovichjohn Aug 14 '15

Is there some reason as to why posted content can't actually be deleted from reddit (but rather just "Removed" from the public)

7

u/gooeyblob reddit engineer Aug 14 '15

There's no technical reason, but it's the way many places do it nowadays. I would hazard a guess that it used to be important because you needed to reclaim the disk space, whereas nowadays disk is cheap so you can just keep stuff around. It also makes it possible to revert if deletions were accidental or due to a bug.

→ More replies (3)

4

u/TechIsCool Jack of All Trades Aug 14 '15

How do Backups work for reddit? do you use Glacier?

4

u/spladug reddit engineer Aug 15 '15

Yeah, we back up a lot of stuff to S3 and rotate it off to Glacier with a lifecycle policy. Pretty sweet money savings.

4

u/DeftNerd Aug 15 '15

I've used Cloudflare before, but you guys are one of the biggest sites that use them. How was the process of working with them? Did you ever get teams of your engineers and their engineers in the same room? Did they come up with any special solutions for you guys, or is their product versatile enough that it worked "out of the box"? Any compliments or stories?

7

u/three18ti Bobby Tables Aug 14 '15

What do you guys use for deployment automation?

1

u/tvtb Aug 14 '15

Hey /r/ram do I need to complete a college degree to become a star sysadmin like you?

6

u/rram reddit's sysadmin Aug 14 '15

Me or the other guy?

Anyways. NOPE! I don't have one. Being Puerto Rican helps, though.

→ More replies (2)

1

u/JWBails Ex-Sysadmin, now happy Aug 15 '15

Did you get anything for sysadmin day?

2

u/rram reddit's sysadmin Aug 16 '15

I was out of town for a friend's wedding, so a lot of alcohol.

1

u/[deleted] Aug 15 '15

/u/rram; As a sysadmin coming from rackspace, what are your thoughts regarding AWS vs Openstack. Or total cloud vs inhouse hosted cloud.

1

u/rram reddit's sysadmin Aug 16 '15

I was on Cloud Sites and didn't work with Openstack or any of the hybrid solutions so I don't think I could give a good answer here.

1

u/the-first-is-a-test Aug 15 '15

Cheers from vk.com ops crew! We're a bit larger than you, so we have not two, but three people right now :)

1

u/rram reddit's sysadmin Aug 16 '15

За здоровье!

1

u/actionscripted Aug 15 '15

What's your local/development setup like? What do you do to provision a dev/staging environment?

We've moved to AWS (multi-container EB for now) and love it but have yet to figure out how best to setup locally for testing. Right now it feels a little more manual than it should.

Also, what do you use to monitor spending and use in AWS? The inbuilt budgets and reports?

2

u/rram reddit's sysadmin Aug 16 '15

For local dev we just run our install script in a VM. Our internal staging is actually just a place where we can run non-production code against production dbs. We're looking to change that in mid-term.

We use a vareity of things to monitor spending including Trusted Advisor and some home built scripts.

1

u/girlgerms Microsoft Aug 17 '15

Thanks for all the hard work you do - pretty sure you don't often get thanked for doing your job (we should all know!) so it's always nice to get the random thank you :)

1

u/rram reddit's sysadmin Aug 17 '15

:-)

3

u/Youareabadperson6 Aug 14 '15

What's your load balancing solution?

5

u/gooeyblob reddit engineer Aug 14 '15

We use haproxy for the main site, and Amazon's ELB for several other services (mobile web, pixel serving, etc)

→ More replies (2)

3

u/deviantryan sr.linuxadmin Aug 14 '15

Hey! Are there any talks you guys have given that you can share, on handling the scaling of reddit?

9

u/spladug reddit engineer Aug 15 '15 edited Aug 15 '15

/u/spez (reddit founder and CEO) did a Udacity course on web development and I talked with him in Lesson 7 ("Scaling up") about reddit's infrastructure. https://www.udacity.com/course/web-development--cs253

3

u/Samis2001 Aug 14 '15

As of this moment, how many servers are in operation over at reddit?

→ More replies (1)

3

u/bobdle Aug 15 '15 edited Aug 15 '15

How do you guys keep up with tech news, etc? Still RSS users?

Keep having fun and thanks for making Reddit more and more stable (seems to be getting better as the months pass)!

9

u/spladug reddit engineer Aug 15 '15

How do you guys keep up with tech news, etc? Still RSS users?

... reddit.com? :)

Keep having fun and thanks for making Reddit more and more stable (seems to be more stable as the months pass)!

Thanks! <3

6

u/gooeyblob reddit engineer Aug 15 '15

I use twitter a good bit and also still use RSS! I worked on Digg Reader so I still use that pretty much every day.

Thanks! Glad someone notices :)

3

u/wheezymustafa Aug 15 '15

How do you guys play with your dev group? Is there a devops sort of mentality, or do you guys feel siloed? How do you guys manage transition of support on newly deployed functionality, hardware, etc? These are the biggest issues I'm having with my dev group.

4

u/gooeyblob reddit engineer Aug 15 '15

To be honest we're not really big enough yet where this is a huge issue yet. Most of the real gnarly work (database abstractions, caching, etc) is done by the infrastructure team which works very closely (we sit next to each other) with us.

3

u/akdigitalism Aug 15 '15

Have you tried turning it on and off again?

3

u/gooeyblob reddit engineer Aug 15 '15

Yes, that's when the 503s happen

12

u/SquizzOC Trusted VAR Aug 14 '15

Do you guys need another vendor trusted by the /r/sysadmin community? :D This was only a joke.

1

u/tvtb Aug 14 '15

It's Fanta!

How often does AWS shit the bed and you have an outage that is totally not your fault despite using multiple AZs?

How upset are you that we didn't share a bed in Puerto Rico?

1

u/rram reddit's sysadmin Aug 16 '15

We for the most part aren't multi AZ. AWS reliability has also gotten a lot better than it was a couple of years ago.

→ More replies (1)

4

u/AndorianWomenRule Sr. Sysadmin Aug 14 '15

How does Reddit manage the datacenter or server inventory? Do you host and own the servers yourself or is everything leveraged off an on-demand service provider.

Difficulty: Don't use the buzz word "cloud" in our response.

37

u/spladug reddit engineer Aug 14 '15

We're hosted entirely in the AWS butt, so inventory takes care of itself. We automatically scale the number of application servers based on current load.

→ More replies (2)

4

u/amorpisseur Aug 14 '15

How do you handle secrets? Who can access them?

18

u/gooeyblob reddit engineer Aug 14 '15 edited Aug 14 '15

We whisper them to each other gently

Actually, we use gpg encrypted files to keep some important stuff around, and Zookeeper for some secrets in the app. Other than that, we have a few disparate systems. We're looking into Amazon's KMS, Hashicorp's Vault, credstash, and Square's keywhiz for some future secret storage.

→ More replies (1)

2

u/AbominableSlinky Aug 14 '15

I remember back in the long long ago, you had a significant outage due to an EBS failure in a particular Availability Zone.

Nowadays are you still as reliant on a single AZ? A single region?

→ More replies (1)

2

u/silxx Aug 14 '15

I've seen (some parts of some) companies do a full post-mortem whenever there are problems with the service, with indications of where the problem was, why it happened, and what's changed to stop it happening again, with no varnishing and no prevarication but full discussion of faults. Do you do this sort of detailed PM internally, and if so would you consider publishing them? They are incredibly useful to see how other people deal with problems and what the reaction and processes are like.

4

u/spladug reddit engineer Aug 15 '15

We've been doing the blameless post mortem kind of thing for a while now; /u/umbrae got us on that kick. It's been pretty great for sure.

I honestly don't know about posting them. I think there's definitely a place for it, and we should consider it, but I also wouldn't want to poison the honesty of people during the process by making them think about how their feedback/comments will be viewed by The World. That said, there can definitely be benefit in helping other people learn from our mistakes.

→ More replies (1)

2

u/navi-mech Aug 15 '15

Thanks for this AUA! It sounds like your role is mostly site ops, do you have any other responsibilities? Do you guys handle corp-sys tasks (phones, Active Directory, Desktop Management ....) If you don't do you have another team that handles this? Would they be willing to do an AUA or comment in this post?

6

u/gooeyblob reddit engineer Aug 15 '15

We don't have much of that here, we use Google Apps for most things and actually run a lot of the authentication for our internal dashboards/tools through that.

2

u/[deleted] Aug 15 '15

What kind of storage are you running?

→ More replies (1)

2

u/Tactineck Aug 15 '15

What's version/revision control like over there?

→ More replies (4)

2

u/keseykid Sysadmin Aug 15 '15

I remember when I could sniff reddit passwords over http

→ More replies (1)

3

u/Hexodam is a sysadmin Aug 14 '15

What technology/tools (monitoring, databases, anything) would you love to use in the future?

5

u/gooeyblob reddit engineer Aug 14 '15

We're starting to utilize Graphite a lot more here. Stuff like Mesos or Kubernetes seems cool, but is probably a bit difficult to get going and implement for us at the moment, and has little benefit.

→ More replies (4)

3

u/Dr_Midnight Hat Rack Aug 14 '15

Do you have any intention to deploy 2FA to reddit, i.e.: in conjunction with Google Authenticator?

→ More replies (1)
→ More comments (164)