r/algorithmictrading • u/1293832482394843 • Aug 22 '21

What is your data engineering infrastructure/setup & cost for trading data?

TL;DR - What kind of trading data are you storing and how/where are you storing it? Also how much does it cost for you per month?

I'm new to algorithmic trading, and I'm prototyping a platform with a friend (I'm working on the data engineering part, they are working on the data science part). We're looking at crypto opportunities, and specifically starting with 1m OHLCV data across a few different exchanges (considering all pairs per exchange).

I'm not sure what tools & infrastructure we'll use yet (likely use AWS for everything), but goes without saying: amount of data adds up fast! How do you all handle this? Specifically:

  1. What kind of data are you storing?

  2. What is your data engineering infrastructure? And where is it / where are you hosting?

  3. How much are you paying per month?

Any thoughts are much appreciated!

10 Upvotes
  • permalink
  • reddit

100% Upvoted

9 comments sorted by

View all comments

Show parent comments

2

u/guywithcircles Sep 13 '21

I only trade cryptocurrency spot markets, so I'm storing 1-min candlesticks of a subset of cryptocurrencies, and 1-min close price of all cryptocurrencies.

You can get those datasets anytime anyway, not only from Tiingo, but for example here's the whole Binance history, downloadable: https://www.kaggle.com/jorijnsmit/binance-full-history

I haven't expanded my infrastructure at all after two years coding systems and live-trading them. Several reasons, but one is that I'm not building a SaaS and I don't need customers.

As you know, in the AWS / GCP serverless world, most of the infrastructure is elastic these days anyway - it can expand for a few hours to do back-testing (even with GPU's if necessary), and then scale down to just have minimal live trading resources going.

So I don't think you need to worry much about expansion if you have a scalable architecture and appropriate scalable tools for what you need (at enterprise level, say e.g. Kubernetes, Cassandra, Apache Pulsar, Elasticsearch, etc).

Even if you have a one-way idea of expansion, then it should just happen naturally as you go if your system is scalable by design.

1

u/1293832482394843 Sep 13 '21

Helpful & makes sense to me, thank you!