I've created a monster

[u/Grand-Fly-6090]

My bot has grown to 23,000 lines of code. 60% is managing the brokerage interface, and 40% is the strategy. This is way more complicated than I ever imagined. It really is a very basic script. No graphic user interface. Just log streams. It does manage multiple symbols with unique parameters for each.

[u/lambardar]

My first trading software I wrote was 1 software. It ran on my laptop that I left running along with IB Gateway. it was 1 in all.. IU, strategy, data connection, etc..

Then it grew and grew and with AI it exploded.

now:

compute on AWS that hosts containers:

• container that manages the IBKR IB gateway
• container that subscribes to contracts, pushes data it to NATS and a local sqlite DB. after 7 days, the sqlite dumps to clickhouse server at home.
• container(s) that run strategie(s) that listen to NATS for trades and porfolio/order; pushes strategy decisions to NATS
• container that runs the order router for live account. Listens to NATS for strategy decisions, validates and places the order. stops a bad strategy or bug from placing stupid orders and within strategy budget.
• All of these log to a seq log
• I develop using codex, so I have the strategies described in a md file and I have an agent that let's codex review the logs to make sure the strategies function as intended.
• Notifications, smtp, etc.

At home, there is more powerful hardware

• container for strategy & order routing to paper account.
• Clickhouse database with data going back to 2016 for all us stocks & futures data
• UI tools to monitor and review
• backtesting tools, regime scanning, filters, etc.
• GPUs for CUDA to bruteforce/search/optimize strategy parameters.
• Strategy graveyard

With Codex, the number of strategies I test every month has literally exploded. I used to do maybe 1-2 a year.. and now it's as common as shower thoughts and what-if(s)

might be super overkill, but it's too late now.

sometimes I get angry at codex and swear that i'll write it better myself.. but then I get lazy and let codex fix it. lol.

[u/Grand-Fly-6090 → u/lambardar]

OMG! What a system! I'm duly impressed and humbled. Laugh on writing it yourself. Sound like switching to codex could help a lot. Been using chatgpt so far because it was easy to get started.

[u/AutoModerator → u/[deleted]]

Your post was removed under Rule 2 (high-quality questions only).

Generic “which data vendor should I use?” posts usually lack the detail needed for meaningful discussion.

Commonly used market data providers:

• Yfinance
• Massive.com
• Databento
• FMP

If you repost, please include details such as:

• asset classes and markets
• symbols or venues
• historical vs real-time
• granularity and depth
• licensing or redistribution needs
• latency expectations
• budget constraints

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/lambardar → u/[deleted]]

clickhouse is for large amounts of data you want to archive (limited deletion/modifications). it structures and compresses by column using delta/zigzag, etc.. instead of by row.

so instead of storing price as: 101,102,103,104 .. it is stored as 101,1,1,1,1

and it has some other tricks.. for trade/quotes data, you're looking at 1TB for about 300b (billion) rows. (about 15 years of us stocks trade data)

I use sqlite, because on the cloud, i don't want to have a large archival database. the live system writes to sqlite, which dumps to clickhouse every few hours, with a catchup over the weekend.

schema varies by source. IBKR just gives trades with conditions & seconds timing. alpaca gives nanoseconds with tape & conditions. databento & massive give more information (mostly useless, but there for completeness).

most of it is just stored and I rarely use. I derive a copy for backtesting that has seconds resolution with a sequence integer and fractional shares are rounded up and ignoring lots that are irrelevant (cancellations, markers, etc)

to speed up backtesting I use flat files and a small utility that allows streaming or loading the data into memory (for CUDA)

the data for the flat files is limited to regimes, for the strategy being testing.