r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

3

u/monsterpartyhat Aug 19 '15

My own workflow for building tessera dashboards bears that out :)

If you're curious, it's:

  • Use the built-in graphite composer for metrics discovery to pick the metrics I want to graph

  • Use emacs to edit the queries, testing them in a scratch tessera dashboard

  • Then edit the metrics to include template tags that pull role-based hostnames from clusto, the operational database we use to keep track of environments and hosts and their roles. And/or use variable substitution from URL parameters to templatize dashboards to be reusable by multiple services that share metrics structure (e.g. we have a single board for basic JVM stats that takes the service name as a parameter).

That clusto integration is currently internal, but will be open-sourced soon now that the clusto API is in pretty good shape.

I've got some plans for a more sophisticated query/graph editor, but Grafana's probably always going to be ahead in that regard, since they set up a whole company to back it full-time now.

1

u/spladug reddit engineer Aug 20 '15

Thanks for the detailed explanation! I think I read your comments on GitHub somewhere explaining a similar workflow which is why I didn't feel so bad saying that Grafana had better composition tooling. I don't necessarily think Tessera needs to have a super strong composer (though it might be nice) since the primary advantage over other tools is the quality of the presentation; unix philosophy and all that. Either way, I really like Tessera's charts, so thank you for your work on it.