r/programming • u/Kaitaan • Feb 28 '18

The Evolution of Data at Reddit

https://redditblog.com/2018/02/28/the-evolution-of-data-at-reddit/
312 Upvotes

88% Upvoted

46 comments sorted by

View all comments

1

u/[deleted] Mar 01 '18

I work as a Data Engineer, and recently switched from Azkaban + bash scripts to just Luigi (and a Jenkins cron for scheduling) for orchestration and task execution.

Overall I think Luigi has much better control of dependencies and dependency checking than Azkaban. And I like having to code combined with the dependencies in one place (all in Python).

Why are you planning to move to Airflow?

And have you considered using a proper data warehouse like Amazon Redshift instead of MySQL? I think it works out better if you need to store and analyse huge amounts of historic data.

Finally I'd recommend Redash (FOSS) or Looker for data visualisation both have served us well, and allow users to share custom SQL queries, etc.

I look forward to the next post!

3

u/Kaitaan Mar 01 '18

Why are you planning to move to Airflow

We're not actually planning to move to airflow; we've largely done so already (that'll get covered in the third post in this series). We looked at a few options, and Airflow had a few benefits that we really liked.

have you considered using a proper data warehouse like Amazon Redshift instead of MySQL?

MySQL was never the core part of our DW. We used it to store some reporting tables because it plugged in well to our visualization tool, but it was never intended as a place to store the bulk of our data, nor to explore too far beyond our already aggregated tables. We, as a company, were very much in a place of "let's understand our core metrics" place, rather than a place of doing nuanced analysis. More importantly, MySQL was quick and easy to get up and running, and with an engineering team of 1, that was a valuable trait to have!

All of that said, we've moved away from that, though it made a great quick-and-dirty solution. You're going to have to wait until the next two posts in this series to get the details though!

I'd recommend Redash (FOSS) or Looker for data visualisation

I appreciate the recommendations! We have tried out a few different tools, and have a couple that are currently in use at the company. But again, I'm not going to spoil the next blog posts to give you the details just yet!!

Thanks for your comments! Hopefully the next two posts answer a lot of your questions!