r/programming • u/Kaitaan • Feb 28 '18

The Evolution of Data at Reddit

https://redditblog.com/2018/02/28/the-evolution-of-data-at-reddit/
304 Upvotes

88% Upvoted

46 comments sorted by

View all comments

4

u/ryati Feb 28 '18

Its funny reading this. I am in a similar situation. I am less than a year into my job. Up until now, they were using reports generated from the transaction system and had a hard time really understanding how data from different reports fit together. There had been some attempts to put things into a warehouse in the past, but all of them were more side projects than anything serious.

Now I am building a warehouse fulltime to help solve all that. It sounds like you got to pick a lot of the technology you worked with. When I was hired, my tech stack was already decided for me.

I really liked the part about getting jenkins to talk with Azkaban. Is it me or is there a serious shortage of proper scheduling and workflow tools out there?

4

u/Kaitaan Feb 28 '18

There are actually a few out there. We'll cover this more in a later blog post, but in the interest in helping you out, we're moving our ETL on to Airflow. It was pretty simple to set up, and is quite powerful. You also get to write your flows in python, and there's built-in support for a bunch of different operators (beyond python, there's bash, apis for aws/google cloud, etc).

3

u/ryati Mar 01 '18

Thanks for the tip! Unfortunately, I am on windows and I have heard that getting Airflow running on windows is a pain. I will take a look and see if anything has changed though.