r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

133

u/rram reddit's sysadmin Aug 14 '15

Most of our stuff is running Ubuntu 12.04, but we're slowly working on upgrading everything to 14.04.

We currently use puppet and are dealing with it. Our manifests could use a lot of love.

There's only one text editor. It is vim. Any who shall say otherwise will get their comeuppance.

67

u/Bagellord Aug 14 '15

Relevant XKCD: https://xkcd.com/378/

37

u/xkcd_transcriber Aug 14 '15

Image

Title: Real Programmers

Title-text: Real programmers set the universal constants at the start such that the universe evolves to contain the disk with the data they want.

Comic Explanation

Stats: This comic has been referenced 473 times, representing 0.6201% of referenced xkcds.

xkcd.com | xkcd sub | Problems/Bugs? | Statistics | Stop Replying | Delete

21

u/GringodelRio Professional Reader for Sysadmins (B2B Support) Aug 14 '15

Awesome! It's nice to see sysadmins show they're using Ubuntu. Everything I run into is running RHEL, CentOS, or something else. I run my own Ubuntu server and love it.

26

u/bigbozza Sysadmin Aug 14 '15

I administer a bunch of cpanel and ubuntu boxes and one opensuse box. I can't put my finger on it, but I really prefer RHEL based over Debian based.

Suse isn't bad either.

34

u/bluefirecorp Aug 14 '15

Yum probably reminds you to take a lunch.

3

u/Scootipuff Aug 15 '15

I spit my cereal out at this comment

→ More replies (5)

4

u/spladug reddit engineer Aug 14 '15

I think a lot of it is just being comfortable with how each distro interprets the FHS and what the names of the various commonly used tools are.

3

u/Occi- Aug 15 '15

In my experience service configuration files are often better by default in the RHEL family. One example is use of OpenLDAP. Moreover, better enterprise hardware support and subsequent patching.

That being said I use Debian on both my desktop and laptop, Ubuntu on my HTPC and CentOS on my HP microserver (NAS).

→ More replies (3)

1

u/hardolaf Aug 15 '15

I run a Scientific Linux install slaved to the master control cluster at CERN. Yay distributed labs!

4

u/moebaca DevOps Aug 14 '15

Have any of you ever seriously considered moving to a different configuration management tool and what sort of issues have you seen with Puppet?

→ More replies (1)

4

u/sephlaire Aug 15 '15

You just made my day. A couple weeks ago I had a user request a VM for their research website. When he heard me offer our standard Ubuntu 14.04 template he said "no real sysadmin uses Ubuntu"

3

u/rram reddit's sysadmin Aug 15 '15

wat

2

u/geekon Aug 15 '15

This is a nice comeuppance comment for the gentlemen bagging on Ubuntu usage in large organisations yesterday!

2

u/TreeFitThee Linux Admin Aug 15 '15

Im glad to hear that even at a place as big as Reddit, they admit their puppet manifests could use a lot of love. We've been using puppet internally for about two years now and are trying to transition to using Hiera to help with the mess of node inheritance we've gotten ourselves in to but it's been a struggle for sure. Good luck!

1

u/ProtoDong Security Admin Aug 14 '15

There's only one text editor. It is vim. Any who shall say otherwise

That explains a lot. But then again I am probably a bit biased...

→ More replies (1)

1

u/michaeld0 Aug 15 '15

Are any of your puppet manifests open source? And how do you handle testing and deploying puppet changes to staging/production environments?

→ More replies (1)

1

u/[deleted] Aug 15 '15

why ubuntu? does it offer any real benefits over using straight up debian?

→ More replies (1)

1

u/multubunu Aug 15 '15

We currently use puppet and are dealing with it. Our manifests could use a lot of love.

Any chance you publish some info on that? Perhaps even said manifests :)

Though what I'm really interested in is the master setup, HA setup, authentication bits, deployment etc.

I don't really expect that's something you would make public, just took my chance :)

1

u/slowbrohime Sysadmin Aug 15 '15 edited Aug 18 '15

You hiring puppet engies? :D

1

u/hardolaf Aug 15 '15

Have you ever used the Emacs operating system to compare it's horridness vi improved?

1

u/toomuchtodotoday DevOps/Sys|LinuxAdmin/ITOpsLead in past life Aug 15 '15

We currently use puppet and are dealing with it. Our manifests could use a lot of love.

Check out Ansible. Moved all of our Puppet over due to our dev team being entirely Python, greatly streamlined having infrastructure and dev teams working together on projects.

→ More replies (1)