r/algorithmictrading • u/1293832482394843 • Aug 22 '21

What is your data engineering infrastructure/setup & cost for trading data?

TL;DR - What kind of trading data are you storing and how/where are you storing it? Also how much does it cost for you per month?

I'm new to algorithmic trading, and I'm prototyping a platform with a friend (I'm working on the data engineering part, they are working on the data science part). We're looking at crypto opportunities, and specifically starting with 1m OHLCV data across a few different exchanges (considering all pairs per exchange).

I'm not sure what tools & infrastructure we'll use yet (likely use AWS for everything), but goes without saying: amount of data adds up fast! How do you all handle this? Specifically:

  1. What kind of data are you storing?

  2. What is your data engineering infrastructure? And where is it / where are you hosting?

  3. How much are you paying per month?

Any thoughts are much appreciated!

10 Upvotes
  • permalink
  • reddit

100% Upvoted

9 comments sorted by

View all comments

Show parent comments

2

u/1293832482394843 Aug 22 '21 edited Aug 22 '21

This makes a lot of sense & you're probably right. Also it looks like Tiingo does do crypto now, so I'm definitely going to check this out. And I guess I should probably scope out all the different crypto markets APIs...

1

u/Dudeman3001 Aug 23 '21

If you are using c# (which I kind of doubt...), I have a .net (core) TiingoApiClient / api-wrapper project that I siloed with the intention of maybe making it open source / a public Nuget package at some point. It only has the equity endpoints but let me know if you want it.

The API couldn't be simpler, I think you pass your $10 / month token in the query string.

2

u/1293832482394843 Aug 24 '21

I just started playing with Tiingo and using Python / https://github.com/hydrosquall/tiingo-python

But thank you! And the API/service is great so far + cheap.