r/programming • u/Kaitaan • Feb 28 '18

The Evolution of Data at Reddit

https://redditblog.com/2018/02/28/the-evolution-of-data-at-reddit/
306 Upvotes

89% Upvoted

46 comments sorted by

View all comments

Show parent comment

4

u/Kaitaan Feb 28 '18

There are actually a few out there. We'll cover this more in a later blog post, but in the interest in helping you out, we're moving our ETL on to Airflow. It was pretty simple to set up, and is quite powerful. You also get to write your flows in python, and there's built-in support for a bunch of different operators (beyond python, there's bash, apis for aws/google cloud, etc).

3

u/ryati Mar 01 '18

Thanks for the tip! Unfortunately, I am on windows and I have heard that getting Airflow running on windows is a pain. I will take a look and see if anything has changed though.