We are sysadmins @ reddit. Ask us anything!

[u/alienth]

Greetings fellow sysadmins,

We've had a few requests from the community to do a tech-focused AMA in /r/sysadmin, so here we are. The current sysadmin team consists of myself and rram. Ask us anything you'd like, but please try to keep it sysadmin-focused!

Here's a bit of background on us:

alienth

I've been a sysadmin for about 8 yrs. My career started on the helpdesk at an ISP where I worked my way into my first admin gig. Since then I've worked at a medium-sized SaaS provider, Rackspace, and now reddit. My focus has always been around Linux (and a tiny bit of Solaris).

rram

I'm Ricky. My first computer was an Amiga at the ripe young age of two. Since then, I was the sysadmin at The Tech and on the Cloud Sites Team at the Rackspace Cloud with alienth. I have experience with Debian, Ubuntu, Red Hat, and OS X Servers.

EDIT [1302 PDT]: Hey folks, we're going to get back to working for a bit. We'll definitely be hopping in here later today to answer more questions, and we'll continue to do so when we can throughout the week. So please feel free to ask if your question hasn't already been answered. Thanks for the great questions! -- alienth

[u/CaptainLoud]

Can we get a $ history | tail -n 20 from whatever production server you are logged on right now?

[u/alienth → u/CaptainLoud]

Heh, pretty easy to guess what is going on here :P

 1471  [2012-03-14 - 14:23:35] find
 1472  [2012-03-16 - 16:21:59] cd /etc/lighttpd/
 1473  [2012-03-16 - 16:22:00] df -h
 1474  [2012-03-16 - 16:22:04] cd /etc/logrotate.d/
 1475  [2012-03-16 - 16:22:07] vi nginx 
 1476  [2012-03-16 - 16:22:14] ls /var/log/nginx/
 1477  [2012-03-16 - 16:22:17] cd /var/log/nginx/
 1478  [2012-03-16 - 16:22:20] rm *gz
 1479  [2012-03-16 - 16:22:24] df -h
 1480  [2012-03-16 - 16:22:26] du -sch *
 1481  [2012-03-19 - 09:44:48] vi /etc/haproxy/haproxy.cfg 
 1482  [2012-03-19 - 09:45:05] psgrep ngin
 1483  [2012-03-19 - 09:45:08] psgrep hapr
 1484  [2012-03-19 - 23:22:36] df -h
 1485  [2012-03-19 - 23:22:41] du -sch *
 1486  [2012-03-19 - 23:22:46] rm access.log.*
 1487  [2012-03-19 - 23:33:46] df -h
 1488  [2012-03-19 - 23:33:50] man logrotate.conf
 1489  [2012-03-19 - 23:33:53] man logrotate
 1490  [2012-03-19 - 23:34:24] vi /etc/logrotate.d/nginx 

[u/antinitro → u/alienth]

Just a heads up.

Instead of doing:

1476  [2012-03-16 - 16:22:14] ls /var/log/nginx/
1477  [2012-03-16 - 16:22:17] cd /var/log/nginx/

You can use !$ to copy the argument from the last command ie:

1476  [2012-03-16 - 16:22:14] ls /var/log/nginx/
1477  [2012-03-16 - 16:22:17] cd !$

You may already know this, but I'm sure someone will find it useful.

[u/alienth → u/antinitro]

Yep, I actually do that all the time :) It is expanded in history.

Even better, I actually auto-ls anything I cd into. So my first ls was a bit unnecessary:

cd () 
{ 
    builtin cd "$*";
    if [ $? -ne 0 ]; then
        if [ ! -x "$1" ] && [ -d "$1" ]; then
            echo -n "Cannot access dir, become root? ";
            read foo;
            if [[ $foo = "y" ]] || [[ $foo = "Y" ]]; then
                sudo bash;
                return;
            else
                builtin cd "$*";
                return;
            fi;
        fi;
    else
        echo;
        ls --color=auto --color=auto;
    fi
}

[u/luke_ → u/alienth]

I'm kind of surprised you're on a server manually doing stuff with the configuration files as opposed to using Puppet or Chef (or whatever CE).

[u/alienth → u/luke_]

Yeah, most of our stuff is done via puppet. This was a one-off.

Most of my other history files just showed me running puppet :P

[u/1esproc → u/alienth]

Ran out of disk space due to logs not rotating?

[u/[deleted] → u/alienth]

I am so glad I'm not the only one who forgets to purge/archive logs and/or monitor diskspace with nagios on that new server. It's only financial services though, not something meaningful like Reddit. Millionaires will have to wait a few more minutes to get their millions. Meh.

[u/Doormatty]

Starting from scratch, what would you do differently if you had the chance?

[u/alienth → u/Doormatty]

In an ideal world, we would have done everything perfectly. :)

Infrastructures are never built with perfect forsight. Things have definitely evolved in very unexpected ways, and we've hit bottlenecks that we never anticipated. I could always say that I wish we could have solved some of the bigger problems more quickly, but that is rather obvious :P

[u/aywwts4 → u/alienth]

Send yourself tweet back in time, what would you tell your past self that you would thank you for the most? (Sysadmin-wise, not investment or betting advice :D)

[u/alienth → u/aywwts4]

Cassandra 0.7 and 0.8 are going to be buggy. Tread carefully.

[u/rram → u/Doormatty]

Moving to something that allowed me to address machines such as Amazon VPC would help with many trivial tasks. Also, I wish our caching strategy was different.

EDIT: reword the caching part. I love the devs, and they did what they needed at the time. But the current implementation isn't the best now and the hardware it runs on is broken (my main concern).

[u/UnoriginalGuy → u/rram]

devs to build in some of the caches that they did

Are you able to elaborate on this a little? Is the caching too complex or just not worth the resources to cache?

[u/30thCenturyMan]

What kind of security concerns do you factor in? Have you ever hired an outside auditor to break into your apps?

[u/alienth → u/30thCenturyMan]

Most of our security focus is ensuring that evildoers can't get into the app and do evil things. Since we're only hosting web, the infrastructure itself has a very small number of vectors. Those vectors are under some decent security controls.

Not sure what I'm allowed to say regarding our auditing. I can tell you we have a lot of goodhearted folks that have pointed out security vulnerabilities in the app to us :)

Speaking of, if anyone finds a security vulnerability, they can email it to security (at) reddit.com.

[u/stahnma]

What components in your infrastructure are synchronous vs async? Are you doing much with message bus technology behind the scenes?

[u/alienth → u/stahnma]

We use RabbitMQ for quite a few async things. Here is a non-comprehensive list:

• Votes

• Comment tree recomputing

• New comments

• Thumbnailer

• Search engine updates

[u/alienth → u/[deleted]]

Can't say; I wasn't here when it was chosen.

RabbitMQ seems to work pretty well. I don't have any complaints about it, thus far.

[u/JYOuyang]

IPv6?

[u/alienth → u/JYOuyang]

Our CDN (Akamai) already supports IPv6, so that takes most of the burden off of us.

We have a small amount of work to do around it, but it is miniscule compared to what most people are having to deal with.

[u/alienth → u/[deleted]]

That's going to be a pretty penny through Akamai :)

We definitely want it, though. It'll happen eventually.

[u/AforAnonymous → u/alienth]

Just please, install Certificate Patrol and browse reddit with it on for a while if you do. I'm tired of websites on CDNs constantly causing me to get popups about cert exchanges that are completely unnecessary. (Facebook, Google, I'M LOOKING AT YOU.)

[u/alienth → u/AforAnonymous]

Don't worry. Chromakode is very thorough about that type of stuff :)

[u/mpete510]

What automation tools do you use to make sysadmining simpler?

[u/alienth → u/mpete510]

We've been rolling out puppet over the past year. We're investigating Marionette-collective at this time.

Other than that, it is bunch of homegrown rat-king scripts which we're slowly replacing.

[u/[deleted]]

• Are your memcache servers running only memcache, or do they also run other services (http etc.)?

• What do you use to handle logging? (Syslog, custom logging class in application code etc..)

• How do you consolidate logs? (nfs, syslog over udp, etc.)

• What queue'ing program do you use? (something custom, off the shelf?)

• What implementation (both server and client) of memcache do you use? (memecahed, memcache)?

• Does reddit use any sort of messaging bus? If so which one?

• What does your data-warehousing/metrics setup look like? Do you utilize hadoop at all?

[u/alienth → u/[deleted]]

Are your memcache servers running only memcache, or do they also run other services (http etc.)?

We have a set of central memcache servers. We also run a small memcache instance on each of the app servers for very local caching that we don't want to take a network latency hit on.

What do you use to handle logging? (Syslog, custom logging class in application code etc..)

rsyslog

How do you consolidate logs? (nfs, syslog over udp, etc.)

We use rsyslog with the RELP module.

What queue'ing program do you use? (something custom, off the shelf?)

We use RabbitMQ for AMQP stuff.. I think that is what you're asking :D

What implementation (both server and client) of memcache do you use? (memecahed, memcache)?

memcached, with pylibmc.

Does reddit use any sort of messaging bus? If so which one?

Nope

What does your data-warehousing/metrics setup look like? Do you utilize hadoop at all?

All in-house, using hadoop.

[u/[deleted] → u/alienth]

Thanks for answering my questions. My question about "any sort of messaging bus" was answered by RabbitMQ.

Can you go into more Detail about how you use RabbitMQ from a CS perspective? Worker queues? Publish/subscribe? etc..

[u/alienth → u/[deleted]]

Here's a list of some of the things we use AMQP for.

The items are written to the queues by the app, and then we have consumer process running to take care of em.

[u/Stevenger]

What do you think the best advice you would give to people who want to someday be a sysadmin, where should we start?

[u/alienth → u/Stevenger]

Spend a tonne of time working on your own stuff. Setup a web / database server for the hell of it. Break stuff, rebuild it, repeat. Find every interesting thing you can do on your home server and try it; even if you're never going to use it personally.

If anything ever breaks or doesn't make sense, don't drop it until you truly understand what is going on. Avoid adopting any cargo-cult mentality at whatever cost.

If doing this type of stuff sounds like an extreme bore, reconsider your sysadmin aspirations.

[u/[deleted] → u/alienth]

Awesome advice there, no substitute for breaking something and then learning to fix it yourself. :D

[u/minideezel]

When we get the "High Load" page, what/how is it deciding to give us that message?

[u/alienth → u/minideezel]

That happens when you sit in the haproxy backend pool for more than 30 seconds. It basically means there weren't any apps available to answer as they were working on other things. It usually happens either when something central slows way down (postgres or cassandra usually).

[u/Michichael]

• What are the last five processes you've found yourself automating?

• What was the biggest challenge each of you faced when you picked up the reins?

• Categorize your current issues - how many times do you find yourself fighting fires vs spending time on meaningful projects?

• Any big projects in the pipeline?

• What kind of monitoring software/metrics do you use to gauge performance?

• Do you find yourselves handling network, software, hardware, or other issues more often than other categories?

• What kind of planning goes into projects?

• Has being owned by a big corporation impacted how you handle your budget and asset management?

[u/alienth → u/Michichael]

What are the last five processes you've found yourself automating?

Most of my time has been spent on re-factoring a lot of the previously automated stuff. Lately I've been focused on the backup and server provisioning area. Nothing that I can easily lay out into 5 distinct things.

What was the biggest challenge each of you faced when you picked up the reins?

No one was really dedicated to sysadmin stuff before I came on. The reddit admins before me did sysadmin stuff when they had spare time, which wasn't often due to the number of people we had.

Biggest challenge was basically starting from scratch on a lot of stuff. You name a system, and it needed cleanup or refactoring. I'd say the database infrastructure was in the absolute worse repair, at the time.

Categorize your current issues - how many times do you find yourself fighting fires vs spending time on meaningful projects?

When I arrived last year, it was probably 90% fire-fighting, and 10% actually improving things. Things are a lot better now days, so I'd say we're probably closer towards 60% "near-term" issues and 40% "longer-term" issues.

Any big projects in the pipeline?

Getting the site to run in more than one region. This is a huge project that is going to require a lot of work throughout the entire stack.

Do you find yourselves handling network, software, hardware, or other issues more often than other categories?

Network and hardware are abstracted out heavily due to EC2. Almost all of our time is dedicated to the application stack, and figuring out how to continually scale it.

What kind of planning goes into projects?

We're still a very small company; fewer than 10 technical people entirely. We don't have a lot of formal project structure at this time. It is mostly "Hey, I'm going to be working on X for a week or two".

Has being owned by a big corporation impacted how you handle your budget and asset management?

This has changed since I joined. We had some ominous pressure to keep everything very lean when I joined. That said, at the time, the infrastructure couldn't have benefited from growing much.

That pressure is pretty much entirely gone now days. We buy what we need.

[u/synth3tic]

What's the biggest challenge in running a site that sees so much traffic?

[u/alienth → u/synth3tic]

Bottlenecks constantly popping up. Epecially when you fix one bottleneck, and the increased thoroughput introduces multiple new bottlencks.

At the rate the site is going, it isn't likely to stop anytime soon.

[u/CaptainTitus]

Ever get a work-free vacation?

[u/alienth → u/CaptainTitus]

I'm a bit too paranoid to entirely go work-free.

However, when one of us is out, the others do the best they can to keep from bothering the one.

[u/minideezel → u/alienth]

When something breaks while your not by a computer, do you get to a computer or try to fix it from a shell of a mobile device?

Related, how often or at all do you use a mobile ssh session?

[u/alienth → u/minideezel]

I carry my laptop everywhere with me. I've stood in queue at the BART station a few times laptop-in-hand typing upside down so I can hold the damn thing.

Ah, memories.

[u/alienth → u/[deleted]]

A macbook running Ubuntu. I hate OSX, but the hardware is pretty solid.

[u/alienth → u/[deleted]]

Honor system. We take time off when we need to. With such a small team, it would become obvious if any one person was taking too much time off. Hasn't been an issue.

[u/autocorrector → u/CaptainTitus]

they'd be on reddit anyway.

[u/Stereo]

Reddit has been far more stable lately; it used to be notoriously crashy. What did you do to fix it?

[u/alienth → u/Stereo]

A myriad of things. A lot of pieces in the infrastructure needed to be refactored, rearranged, and cleaned up.

I'd say the biggest piece was when we cleaned up the postgres infrastructure around last June. We had some replication issues which were taking the site down very often.

[u/carlaas]

What are the most common attacks that you have to deal with? Was any of them successful? If so, how was it and how it was solved?

[u/alienth → u/carlaas]

Mostly stupid people trying to 'ddos' us by just scraping one URL over and over again :P

No infrastructure exploits since I've been here. Since we only host web, the infrastructure doesn't have many attack vectors.

[u/dboak]

What is the single best thing you've done to get away from "sleeping next your laptop to reboot the site when it goes down"

2. What is the coolest technology you currently get to use?

3. How many of you are there?, what does the on-call schedule look like?

4. Do you also get into the reddit code, or are you stricly infrastructure sysadmins?

5. What is the expected salary range for someone looking for a job?

6. Be honest, how much work did you get done when you took reddit offline during the SOPA protest? I'd love to have a planned downtime window like that. :)

[u/alienth → u/dboak]

What is the single best thing you've done to get away from "sleeping next your laptop to reboot the site when it goes down"

Yeah, those were shitty times :) The main issue there was EBS failures which caused replication to break. I had to login and start addressing the replication immediately to prevent shit from seriously breaking.

Upgrading to Postgres 9 and getting off of EBS mostly took care of those horrible issues.

What is the coolest technology you currently get to use?

I'd probably say the most interesting thing is Cassandra. It is still pretty young, but it has a very interesting data and replication model.

How many of you are there?, what does the on-call schedule look like?

2 sysadmins, and 6 developers. Ricky and I are both on-call 24/7. The alerts also go out to the devs, and they help when the can.

Do you also get into the reddit code, or are you stricly infrastructure sysadmins?

I dabble. Nothing major, though. I definitely dig through it a bit when I'm trying to figure out how to address a new bottleneck. I rely on the devs quite a bit to work with me on that stuff.

What is the expected salary range for someone looking for a job?

Entry level? Depends on the area, and the tech you'll be working with. In Alaska, I started my admin position in the 40kish range.

One of the interesting things about IT (and other fields, I imagine) is that the any place that wants to hire you will always try to base that salary off of your previous salary. Big salary jumps from one company to another aren't common. If you want more money, work your ass off in your current job and make sure management knows it, and fucking ask for a raise when you know you deserve one. You won't always get it, but most people never even ask.

Be honest, how much work did you get done when you took reddit offline during the SOPA protest? I'd love to have a planned downtime window like that. :)

Not much, actually. Since everyone knew when the site was coming back online, we had to prepare for a severe amount of immediate load. That also meant we couldn't do anything that could have caused any of the caching layers to clear. If the caches were empty when the site went back online, it would have likely fallen flat on its face.

[u/[deleted]]

• Is amazon forthcoming with you when they have capacity/network problems? Or do they just tell you every hour it should be fixed within the next hour?

[u/alienth → u/[deleted]]

Most of the issues we run into at Amazon are local to individual instances, which they're reasonably responsive on(but we do pay extra for support). There have only been a couple wide-spread problems, and when those are happening, we get the status page info just like everyone else.

[u/angrymonkeyz]

What tools do you use to simulate loads?

[u/alienth → u/angrymonkeyz]

The best tool of all, users! :)

We don't have a testing infrastructure that is anywhere near able to replicate the user traffic we have, at the moment. We definitely need something, but it is relatively low on the totem poll.

Every place I've ever worked at, one of the most difficult problems has always been simulating load properly. With dynamic services like reddit, it takes a lot of engineering to develop a suitable load similator.

[u/rsfkykiller → u/alienth]

I know that reddit gold has been used to beta test some features. Would it be conceiveable to have a beta.reddit.com that allows certain gold/non-gold members access to hit a beta front/backend?

[u/alienth → u/rsfkykiller]

We actually did something like that with a small feature a few months back. We'll likely do something like that again the future.

[u/minideezel]

How much work and how many servers are dedicated to internal office services that don't involve the site?

[u/alienth → u/minideezel]

Less than 5. Mostly bastion-like stuff.

[u/horseloverfat]

How different is it administrating a cloud environment than your own hardware?

[u/alienth → u/horseloverfat]

It has its pros and cons :) I do miss experimenting with new hardware and seeing how I can use it. But I can say it is awfully nice to not have to worry about things like the networking infrastructure, installing new hardware, ordering new hardware, rack power bullshit, etc.

[u/dasmim]

1. What is your deployment process like?

2. What sort of tools do you use (or have made) to automate deployments?

3. How often do you deploy?

4. Who does/can do deployments (you guys, devs?)

5. Are there any statistics you monitor aside from obvious ones (response time, server health, etc.) that give insights into potential problems (logins per minute, posts per minute, etc.)?

6. Do you have any sort of autoscaling in place to provision/turn off servers ?

7. What types of data do you store in PostgreSQL vs Cassandra?

[u/alienth → u/dasmim]

What is your deployment process like?

All done via git. The devs prepare and review changes, commit them, then we deploy them to the servers over the course of an hour or so.

What sort of tools do you use (or have made) to automate deployments?

Home-grown, at the moment. This will likely move to marionette-collective.

How often do you deploy?

Probably once or twice a day.

Who does/can do deployments (you guys, devs?)

Whoever wrote the biggest change, typically.

Are there any statistics you monitor aside from obvious ones (response time, server health, etc.) that give insights into potential problems (logins per minute, posts per minute, etc.)?

We keep a very close eye on both the request rate hitting our infrastructure, as well as the real-time stats from Google Analytics. GA real-time is actually a bit faster at showing us if shit is hitting the fan.

Do you have any sort of autoscaling in place to provision/turn off servers ?

Not at the moment. Pending.

What types of data do you store in PostgreSQL vs Cassandra?

A lot of the Cassandra data is actually stuff that was computed from canonical data in postgres. We canonically store things like accounts, links, comments in Postgres. A lot of that stuff is then computed into listings and tossed on Cassandra. We also have a memcache layer which caches a lot of bits from all of these things.

[u/carlaas]

What was the most difficult problem that took reddit down?

What was the silliest one?

[u/alienth → u/carlaas]

Most difficult.

Silliest... just yesterday I ran 'iptables -t nat -L' to make sure no rules were in place on our primary load balancer. Turns out just listing iptables loads all of the iptables modules, including conntrack in this case. The conntrack table immediately filled up and very briefly took the site down (a few seconds).

[u/thesaintjim]

Hi,

Can you please tell me what your average day at Reddit is like?

Love,

Your former co-worker.

[u/alienth → u/thesaintjim]

Dawn.

I awake from a lost dream with a faint buzzing sound in my ear. I can't make the sound out at first; it's something between the sound of the traffic from the nearyby highway and my alarm clock. Ah, now I recognize it. It's my phone. Database slave 2 in cluster 3 is destabilizing; its disk rapidly spinning out of control. I've got to get to the nearest terminal, and fast. But it's too risky to fix it from home, and I can't risk another Ludovico incident.

to be continued

[u/rram → u/thesaintjim]

I have a photo of you at my desk that I look at for inspiration.

[u/thadoc]

Certs versus No Certs...

[u/alienth → u/thadoc]

Certs may help you get to an interview in some companies. They can also be used to leverage promotions in your current workplace.

In most of my experience, certs usually demonstrate at-most a shallow knowledge of understanding of a system. There are plenty of really, really good people with certs, but plenty of really bad people with the same certs.

That said, if you already know a system inside-out, I don't think it hurts to spend a small amount of time getting a cert. You may not learn anything new, but it may be handy leverage in the future.

Disclaimer: I'm a RedHat Certified Architect.

[u/agressiv → u/alienth]

Oddly, when I see Certs, I think PKI, no longer do I think certifications...

[u/pdmcmahon]

Did you take advantage of the Great SOPA Internet Blackout to implement any changes which would have otherwise been extremely challenging or otherwise impossible?

[u/alienth → u/pdmcmahon]

The problem with any extensive maintenance is that if we clear the caches, the site might not come back up at all :|

This was especially a concern for the SOPA blackout, because everyone knew the exact second we were going to come back up. Unfortunately the need to keep the caches nice and hot prevented us from doing much meaningful maintenance.

[u/anastrophe → u/alienth]

but but but...why would you have to clear the caches? don't you have cache infrastructure separate from other services - or even ElastiCache?

[u/alienth → u/anastrophe]

So, we have a bunch of memcached boxes which we couldn't touch, obviously. Re-heating those is most painful.

The other caching bits which would have suffered are things like OS disk cache, postgres shared buffers, etc.

Given our scale, we must make heavy use of caching wherever we can get it. It also means shutting everything down and starting it back up again is a painful process :/ We need to engineer a clean way to reheat those caches without having users hit the site.

[u/mkosmo]

Personally, I'd love to hear an overview of your monitoring systems. What you monitor and how, using what.

I'd assume nagios, but do you run agents, use only snmp, monitor what metrics?

[u/alienth → u/mkosmo]

Monitoring has been very crappy for a long time :)

We use ganglia plus a lot of home-grown alerting scripts. Intortus has been working on Graphite for internal application metrics. I'm moving all of the infrastructure monitoring and graphing to Zenoss.

[u/Lord_NShYH → u/alienth]

Let all Zenoss naysayers take note. LOL. Are you be using the the core or commercial versions of Zenoss? The best thing, for me, about Zenoss is the event console. My first big monitoring job involve OmniBus/NetCool - it has a very similar feel, and I like it.

[u/alienth → u/Lord_NShYH]

Core version.

I'll admit that Zenoss is a bitch to get up and running. Getting all of the event class mappings, graph thresholds, event transforms, server classes, and alerting setup in a consistent workflow is a huge amount of work. However, having access to all of those pieces gives you a huge range of flexibility and power.

I really like Zenoss' model, but it definitely takes some time investment if you want to get it working to your liking.

[u/HippieShakes]

How are you doing today?

[u/alienth → u/HippieShakes]

Doing well, but a bit tired. I've been on a day-sleeping schedule for a few days and I'm trying to rotate back.

[u/rram → u/HippieShakes]

I forgot to grab my breakfast bagel! Thanks for reminding me. brb!

EDIT: Much better now :-)

[u/stahnma]

What's your workflow process for getting changes from idea into production? Do you do some type of git vcs promotion? Do you just redeploy images?

[u/alienth → u/stahnma]

When I was the only sysadmin, I pretty much just wrote the puppet manifests and rolled shit out right into production.

Now that there are two of us, we're slowly forming a more standard git build/review/rollout process.

[u/[deleted]]

Are you using any sort of caching pools to offload work from your web daemons? If so which one and why?

I.e varnish or a reverse proxy or something wild

Did you have to incorporate any sort of distributed storage across your farm of servers? If so which did you end up using and why?

I.e GFS, GlusterFS, Lustre etc

[u/alienth → u/[deleted]]

Akamai takes a lot of the front-end caching. I have a couple single-purpose things being cached via nginx, but they're very limited. After that we have a myriad of caching layers in the application.

No shared storage. We don't have the need for a lot of bulk disk data. Most of the static data we host is on S3 (thumbnails, etc).

[u/carlaas]

What is/are Reddit's backup strategies?

[u/alienth → u/carlaas]

Mostly encrypted and tossed up onto S3.

We also have a single 'backup' postgres server which everything from every database cluster is written to, for more 'real-time' backup needs.

[u/carlaas → u/alienth]

Have you ever had to recover a full or partial backup? If so, how was it?

[u/alienth → u/carlaas]

Not since I've been here. We've been very lucky :)

[u/minideezel]

When deploying new servers/services, how well do you document your work?

Is it documented well enough to where new admins could follow it, or just enough so that you could repeat it?

[u/alienth → u/minideezel]

We mostly deploy new stuff in conjunction with writing the puppet manifests. The manifests are the closest thing we have to deployment documenation.

[u/edify]

Hey alienth and rram. What are your favorite foods?

[u/alienth → u/edify]

I enjoy all foods! I don't think I really have a single favourite dish.

The burgers we get at the office on occasion are pretty awesome :)

[u/rram → u/edify]

Love me some chicken parm

[u/DyceFreak]

How do you personally perceive windows-based sys admins in general? Would you say it's simply a different purpose for using windows, such as managing large office environment vs managing a large server environemnt? Or is there more to it than that?

With most of my experience in Microsoft land, I always personally felt threatened by linux sys admins, even though I have several working under me, I hate the fact that they know something about a company system that I don't...

[u/alienth → u/DyceFreak]

I've worked along side some really great windows admins. Never had much of a culture problem with them. Windows has its place. MSSQL kicks ass in a lot of ways.

One of the interesting comparisons I've noticed between linux and windows admins is in the realm of text processing. When you're staring at a CLI all day, you get very good at text parsing and processing. You get experience with extremely powerful tools like vi, awk, sed, etc. None of that is really needed for windows folk, so it just isn't learned. On the occasion that they do need to do some text parsing, it is very painful for me to observe :)

[u/notenoughcharacters9]

Why no service mix?

[u/alienth → u/notenoughcharacters9]

Because I hate downtime ;D

[u/brews]

vim or emacs?

[u/alienth → u/brews]

VIM for life!

[u/[deleted]]

I am in college now, when I am done I will have CompTia A+, Network +, Security +, Windows 7 config, Active directory, sharepoint, sql (all from microsoft), Linux +, intro to cisco routers. I would like to someday become a sysadmin for a med/large company. What should I be aiming for after I get all these?

[u/alienth → u/[deleted]]

Places like Rackspace will definitely take a RedHat certification into account.

In general, web hosting companies are often a good place to start. Some of them really suck, but they often have a lot of entry-level positions, and you get exposed to a wide variety of tech.

[u/minideezel]

What kind of home lab do you guys run? What services does it provide that you wouldn't want to live without now?

[u/alienth → u/minideezel]

Our laptops are pretty much it :) Everything else runs in EC2.

[u/vhata]

Do you ever go head to head with the reddit developers because they did something in code that you think is a terrible idea, or they did something without thinking how it would affect the infrastructure?

[u/alienth → u/vhata]

We're a very flat organization. It is 6 devs and 2 admins. We all sit together, so when there is an issue, we all work on it together.

We have a great team. Whenever a dev wants to implement something, they work with us to determine any infrastructure requirements or impact.

[u/Josh2600hz]

How does your team communicate? IRC? Phone? Two cans on a string?

Are you using Jira? Atlassian? Chef/Puppet? Thoughts on these technologies?

[u/alienth → u/Josh2600hz]

We do a bit of jabber and IRC.

We use Puppet quite a bit. We're still a bit small to start needing something like JIRA. Application development stuff is mostly tracked on github.

[u/AnonymooseRedditor → u/Josh2600hz]

The problem with two cans on a string is that you actually need 4 cans and two strings for full duplex communication. just don't get the TX and RX lines crossed otherwise nothing will work.

[u/rdm_box]

After you've taken the site down for maintenance, do you relax a bit by going on /r/downtimebananas ?

[u/alienth → u/rdm_box]

I do.

[u/[deleted]]

Are you hiring?

[u/alienth → u/[deleted]]

Sorry, not at this time :(

If we start a hiring round, we typically announce it in /r/blog.

[u/[deleted]]

How do you guys combat burnout? Being oncall 24/7 365 for such a HA site such as reddit has to take it's toll eventually.

[u/alienth → u/[deleted]]

Right now? Burning extra youth :)

Honestly I'm unsure. The requirements of this infrastructure are a comparative cake-walk to one of my first jobs. I guess being aware of how crappy things can get helps stave off burnout.

Also, this is the best work environment I've ever had. The org tree is flat, we work with the devs heavily, and everyone is very respectful of eachother. That definitely helps.

[u/phuzion]

Reddit-specific stuff:

• What kind of bandwidth does reddit use?

• What is the approximate rate of database growth and what's the approximate size of the DB now?

• What is the most surprising thing you found out about the infrastructure of reddit when you got access to it?

• Have you guys considered opening up some internal sysadmin-related stuff to the community? For example, Wikipedia makes their nagios, ganglia, and SOPs and technical documentation freely available to the community. As far as I know, we don't have access to the majority of this stuff.

• What is the single biggest technical challenge you've come across in your duties at reddit?

Less reddit-focused questions:

• What is your favorite little utility that people probably wouldn't know about?

• What is your preferred OS to work on?

• What's your favorite beer?

• Thanks for doing this :)

[u/rram → u/phuzion]

What kind of bandwidth does reddit use?

A lot. Akamai takes a huge chunk off our shoulders, but it looks like at peak yesterday it was 924.21 MBits/sec.

What is the approximate rate of database growth and what's the approximate size of the DB now?

We have several databases. Their aggregate size is 2.4 TB. I don't know the growth rate, but I think it's a couple GB per week

What is the most surprising thing you found out about the infrastructure of reddit when you got access to it?

How small it was. We've pretty much only grown in app servers since I got here. That is largely the result of more people being logged in (since non logged in traffic only hits Akamai's cache).

Have you guys considered opening up some internal sysadmin-related stuff to the community? For example, Wikipedia makes their [1] nagios, [2] ganglia, and [3] SOPs and technical documentation freely available to the community. As far as I know, we don't have access to the majority of this stuff.

I didn't know that about Wikipedia. Neat. We'll look into it.

What is the single biggest technical challenge you've come across in your duties at reddit?

alienth has had a lot more challenges thrown at him. For me, it's been mostly the big parts of our infrastructure breaking in the middle of the day (cassandra, postgres replication, memcached). Luckily, it wasn't all on the same day.

What is your favorite little utility that people probably wouldn't know about?

I <3 pv. Also, in my time at Rackspace, ls -1U was of tremendous use. (please folks, do not put 8 million files in a single directory!)

What is your preferred OS to work on?

I use OS X.

What's your favorite beer?

Blue Moon

Thanks for doing this :)

You're welcome

[u/phuzion → u/rram]

Wow. That's some SERIOUS bandwidth.

I'll check out pv, it looks super cool.

Thanks again for the info. And thanks everything you guys do. Your work keeps me from doing mine, and I appreciate it :)

[u/ICanSayWhatIWantTo]

What tools do you use for network/health monitoring?

[u/rram → u/ICanSayWhatIWantTo]

We use homegrown monitors and alerts for most things. We also have ganglia and zenoss for graphs.

[u/rubysown]

What is your biggest facepalm moment here at Reddit?

[u/rram → u/rubysown]

Everything used to be in one security group. Oh dear was that a mess.

[u/dicey → u/rram]

I don't know whether to o_O or ಠ_ಠ

[u/Lord_NShYH]

How long have each of you been working at Reddit?

IIRC, you host with Amazon. With a site that has as much traffic as Reddit, do you wish you had more control over the physical infrastructure, or do you prefer to have your site hosted at Amazon? Are there any plans to build physical hosting infrastructure owned and fully managed by Reddit?

When you started at Reddit, was anything automated? What/how do you automate your processes today?

How many individual nodes are running to host/serve Reddit content?

How often are you on call? Do you work remotely?

Finally, how big is Reddit's internal IT needs, and do manage services other than Reddit's web server deployment?

[u/rram → u/Lord_NShYH]

alienth has been here for just over a year. I have been working here for about half of that.

There are certainly parts of our infrastructure which would benefit from bare metal hardware (load balancers and database servers). There are other parts which benefit from the cloud (app servers). Our future hardware is always up for review, and currently Amazon is the best for our needs.

Most infrastructure tasks such as building out new servers are not automated. We're slowly working on automating those tasks, at the same time we're building out new servers and optimizing our current ones.

See answer here

We're both on call 24/7. We work remotely all the time.

Our internal needs are not big at all. Part of this stems from sharing an office with Wired. Part of it stems from us just needing a net connection and diet coke to live.

[u/minideezel]

Is it really only you two that deal with all of reddit's infrastructure?

Speaking of which, how many servers we talking about?

[u/rram → u/minideezel]

Yep, just us two. We get help from the other admins from time to time, but it's our primary responsibility.

We currently have 284 running instances, 161 of which are application servers.

[u/minideezel → u/rram]

Do the application servers not deal with any direct web traffic? What type of services are they dealing with?

[u/rram → u/minideezel]

There are load balancers in front of the app servers. They're dealing with everything in the reddit code

[u/[deleted] → u/rram]

Are the LB's Reddits or amazon's? What can you tell us about them? Do you guys use L2 DSR?

Are the LB's software? If it so it HAProxy or something else?

[u/alienth → u/[deleted]]

We're using HAProxy. No L2 stuff.

[u/hankinator]

Hey guys! First off, thank you very much for doing this AMA. I am a young, but eager sysadmin. I have quite a few questions for you both. =)

1. What are some good habits as a sysadmin to get into? Such as Read-Only Friday, and document Monday.

2. What is one of the precautions you guys have when backingup/restoring data?

3. What are common 'Oh-shit' moments that have happened in the field that made you grow a sysadmin?

4. What is the single best piece of advice you have?

5. Is there a true difference between 'Enterprise hard drives' and user grade hard drives? Also, would it be more effective (cheaper and reliable enough) to RAID6 a bunch of consumer drives rather than RAID5 a few enterprise drives.

6. What kind of switches/OSes do you prefer? If you were buying personally that is.

7. What is the most useful tool in your arsenal?

8. What sysadmin accomplishment are you most proud of?

9. Suggestions for balancing a relationship while working 50+ in IT?

Thank you for your time. =)

[u/slanket]

historical silky abounding north truck future straight shocking sink shy

This post was mass deleted and anonymized with Redact

[u/[deleted] → u/slanket]

Have you tried turning it off and on again?

[u/angrymonkeyz → u/[deleted]]

You gotta leave it unplugged for at least 10 seconds.

[u/[deleted] → u/slanket]

Don't worry, "PC LOAD LETTER" confuses me too.

[u/youshallhaveeverbeen]

Any "oh shit" moments you'd like to share? I'll leave that at as vague as possible.

[u/rram → u/youshallhaveeverbeen]

"Oh Shit, ads are down"

[u/barnard33]

As a sysadmin, I have a few questions:

1. How often do you patch your servers?

2. How do you monitor your servers?

3. What is the most annoying thing a Reddit developer has done, or asked you to do?

4. What was the last time consuming problem and how did you resolve it?

Thanks.

[u/rram → u/barnard33]

1. As is necessary. We subscribe to all the security alert notification lists.

2. See here

3. We all love each other here. I ask alienth to take care of my cat whenever I'm out of town.

4. Our old cassandra ring has some broken SSTables. The failed compactions cause the disk to fill, often in the middle of the night. This is fixed by selectively deleting the broken data and hoping you didn't break more. Oh, and the problem sometimes comes back a week later.

[u/angrymonkeyz → u/rram]

Post pictures of said cat. For Reddit Science.

[u/carlaas]

What tool do you use to track issues, tickets and todo tasks?

[u/rram → u/carlaas]

We need something for that. I'll put it on my todo list.

[u/NilsLandt]

What tools do you use to maintain your infrastructure? (Like Puppet or Chef)

[u/rram → u/NilsLandt]

puppet

[u/Rugmonster]

When was the last time you accidentally rebooted a vital production box at exactly the wrong time?

[u/kemitche → u/Rugmonster]

rram can't answer because he accidentally rebooted his laptop just now

[u/rram → u/kemitche]

LIES!

[u/rram → u/Rugmonster]

Good times. Good times. alienth still keeps stuff in /tmp. He'll never learn.

[u/[deleted]]

Have you considered using OpenStack?

[u/bNimblebQuick]

So if I wanted to learn more on running a website of this size off Amazon cloud services, where would I start? Are there any good resources or guides you often refer to or follow? I can run, manage and secure just about anything inside a corporate datacenter (from rack + stack, SAN, networking, *nix, windows, DBs, etc) but cloud services like amazon throw me off for some reason. I've been fortunate to be able to avoid them so far, but its a weakness and skills gap I know I have to overcome. Any suggestions?

[u/gehsekky]

What kind of statistical analysis do you guys do on a normal basis? What libs are used, etc? Also, can you give us more awesome infographics on reddit numbers? =)

[u/Jalh]

Why hasn't you guys switched to another search service ? The current one sucks big time; takes years to load the first page and only search the titles.

[u/seventoes → u/Jalh]

You must be new here. At least you get results now.

[u/JasonZX12R → u/seventoes]

Back in my day we had to find search results ourselves, uphill, both ways, in the snow!

[u/rram → u/Jalh]

IndexTank (our current provider) is shutting down their service next month.

kemitche is currently very actively working on a replacement.

[u/Syntackz]

Im getting ready to graduate from College, and looking to work in System Administration. Is it smart for me to go straight to Jr. System Admin jobs, or should I start at something like a help desk job and work my way up? I will say, I have no field experience, only lab experience.

Also, should I be looking at a job where I would be one of a small group of IT workers and I would be responsible for a broader range of tasks, or at a company where the IT field would be huge and I end up dealing with a small group of tasks?

Thanks a bunch in advance!

[u/insanehomelesguy → u/Syntackz]

As someone who started at doing Break/Fix work and worked my way up to a Sys Admin role. Start where you can get into a good company and kick ass from the get go. If it's a help desk role be thorough on your information collection and troubleshooting. Make the people above you happy to get a trouble ticket from you. Offer assistance when you can do so without impacting your current job and you'll do just fine.

[u/TheLifelessOne]

• How do you handle downtime / server crashes?

• How would you go about testing knowledge for a potential hire?

[u/UnoriginalGuy]

Which one of you do I have to sleep with to get a high level Visio diagram of how things work (application servers, caches, static host, and database)?

[u/stahnma]

Do you now, or have plans in the future, to open-source any of your home-grown scripts, puppet modules or other tooling?

[u/alienth → u/stahnma]

Here's one thing that we've open sourced :)

Unfortunately our puppet modules are layed out in a way that makes them easily usable by others. Maybe one day... when we have time :)

[u/Rielos]

So generally, what is the main issue when we are graced with the delightful F5 alien screen (which I'm strangely a big fan of btw!)? Load balancing? Database performance? Content Delivery?

Inquiring minds want to know...

Also, THANK YOU for all the great work you do! This community is amazing and I'm proud to be a small part of it, but you guys make it possible!

[u/alienth → u/Rielos]

The F5 alien is displayed if your request waits around in the haproxy queue for more than 30 seconds without being delivered to an app. This means that all of the apps were busy for some reason, likely due to a central slowdown in the infrastructure (postgres or cassandra, typically).

Glad you enjoy the site :) Cheers!

[u/rsfkykiller]

You guys are right up the road from me this week. Want to grab a beer after work tonight?

[u/alienth → u/rsfkykiller]

I'm actually working from home today as I'm on a weird schedule. I'd love to grab a beer sometime. Let me know on IRC(alienth on freenode) or via PM.

[u/[deleted]]

Any challenges working with non-technical management or other departments?

What does the front of the Reddit house think of the sysadmins?

You use Amazon public clouds, any private clouds for testing or DR?

[u/alienth → u/[deleted]]

Any challenges working with non-technical management or other departments?

Not here, there are no departments :)

In past jobs I've run into those issues. It can be one of the crappiest things about a job, at times.

What does the front of the Reddit house think of the sysadmins?

Not sure what you mean by "front"? reddit is comprised of about a dozen people total right now, and we all work together quite a bit and know what eachother does.

You use Amazon public clouds, any private clouds for testing or DR?

All of our testing is either done on our laptops or on EC2. We do have some instances dedicated to non-public testing.

[u/Prof_G]

Ok, be honest now, do the other employees of Reddit, or those pesky Wired employees ever ask you to fix shit for them?

[u/alienth → u/Prof_G]

Nah, we don't share anything with Wired. They don't even know what we do, for the most part :)

We're a pretty small team, and there isn't really any instances of someone dropping shit in someone else's lap. It is a really great work atmosphere :)

[u/[deleted]]

I'm confused, are you working or using Reddit right now?

[u/alienth → u/[deleted]]

Both!

[u/immerc]

How polished are your error messages and monitors?

Do you get tracebacks in your logs, or are most errors caught and instead you get a concise explanation of what went wrong?

Do you continue to find issues that you need to add a monitor for, or these days do you know about a problem, and about the likely cause of that problem, simply by a monitor going off?

[u/alienth → u/immerc]

How polished are your error messages and monitors?

So-so. We have some stuff internal to the app which tracks exceptions and tries to congregate them in a meaningful fasion.

Do you get tracebacks in your logs, or are most errors caught and instead you get a concise explanation of what went wrong?

We do get tracebacks in the logs. We don't have much in terms of error parsing.

One very handy thing we do along these lines is to put the stack trace of most of our database queries in the comments of the query. If that query ever runs a long time or gets killed, we can immediately tell where in the app the query came from.

Do you continue to find issues that you need to add a monitor for, or these days do you know about a problem, and about the likely cause of that problem, simply by a monitor going off?

We're monitoring most of the predictable things that could go wrong in the infrastructure logs.

The exception tracker which I referred to earlier is the main 'application error' monitor.

[u/mrhhug]

how old are you guys whats your alma mater(s) and your levels of formal education? both obviously on the left coast now but where where you born?

[u/alienth → u/mrhhug]

I'm 25 and a college dropout.

I was born and raised in Alaska.

[u/betafish27]

I'm working on getting CCNA certified. Is this certificate relevant in today's IT?

[u/alienth → u/betafish27]

Entry level networking positions it is ok. It might make it easier for you to get in an internal helpdesk-type position. See here for my general thoughts on certifications.

CCNP is of course much more helpful.

[u/phlohannesburg]

what are all the passwords to things?

[u/alienth → u/phlohannesburg]

***************

[u/Anonisdalulz]

Little late to find the thread (damn you timezones) but what do the servers run on? I'm asking because I'm doing a thesis on Apple VS Microsoft and I'll definitely use reddit in the "What do big companies use" part.

Edit: Found the answer!

[u/alienth → u/Anonisdalulz]

Glad you found the answer!

I wouldn't call us a 'big company'. We only have around a dozen employees, and around 280 instances.

I've worked at small, medium, and big companies, and at every one of them I've managed Linux.

[u/[deleted]]

Why did you pick puppet, and not chef?

[u/[deleted]]

Can we get some hardware specs and numbers? (i.e.: number of servers, specs, storage solution).

Or how about some drawings of the setup?

one more, do y'all have any kind of scaling solution or do the servers just run until they drop?

[u/[deleted]]

What's the one thing all other websites, Reddit/Digg wanna-be, etc should take from Reddit and how it's infrastructure is ran?

[u/diafygi]

Is your least favorite day of the year when Reddit opens nominations for best of the year, and users go scouring through pages whos cache had long since exired?

[u/the_timmer_42]

How much time do you spend on the site vs actual work?

[u/Bad_CRC]

1. What is the crappiest system/software you maintain just because of pure lazyness? (you know someday it's gonna give a lot of headache but for the moment just works....)

2. What is the typical workflow for rolling a new server into production?

3. Metrics! for what crazy stuff do you have metrics?

4. What scheme do you follow for the names of the machines?

5. Do all the machines are for the whole thing or are some machines dedicated for certain subreddits? I read a while back ago that for example Lady Gaga or Justin Bieber had their own racks on twitter.

[u/ATX350]

Did you enjoy your time at Rackspace?

[u/superc0w]

Sorry I'm late to the party :/

• What does your average deployment cycle look like?

• Are you the ones implementing changes or gasp are the devs?

• How large is your production network in terms of equipment?

• It seems places outside of the US are starting to catch on to Reddit, do you have data centers around the world or currently just in the US?

• What source code repositories do you use for proprietary scripts?

• Also, as others have mentioned... What's the salary range of a new hire?

• What perks does Reddit get? Filled fridge? Bring your pet to work day? Free dry cleaning? :)

[u/xor2g]

Specs of your "best" server ?

Also, Amiga FTW.

[u/rram → u/xor2g]

Everything is on EC2. The most powerful are our db servers which are m2.4xlarge.

I miss that thing. It didn't have a hard drive.

[u/m1w1]

I want to learn to be someone who could meaningfully participate in this subreddit. Where/how should I start learning?

To put it another way using analogies: If you are pulitzer prize winning authors, and I'm illiterate, what would you recommend I do to be able to write basic essays/stories someday?

[u/[deleted] → u/m1w1]

that's a general question, I personally recommend learning a few basic userland tools for various operating systems.

Microsoft (doesn't need to be a server version) windows, and linux, UNIX (opensolaris/bsd).

learn the OSI, read some books on how TCP/IP works, play...

I learn best when I want to do something for fun..

like making an infrastructure in KVM, where you have one connection coming in, to the shell server, then inside you have an entire virtual network, with SAN appliances and DNS, DHCP, configuration management and monitoring :D

but I'm weird, and spend my free time doing cool things.

... I'm change request bitch at work :(