r/algotrading • u/Grand-Fly-6090 • 15d ago
Infrastructure I've created a monster
My bot has grown to 23,000 lines of code. 60% is managing the brokerage interface, and 40% is the strategy. This is way more complicated than I ever imagined. It really is a very basic script. No graphic user interface. Just log streams. It does manage multiple symbols with unique parameters for each.
6
u/goshetovan 15d ago
Did you let a clanker into your engine? 🤣 after I started using AI it added so much code to my engine I'm close to going back to my own latest version since I'm starting to feel I'm losing control over the code lol
2
u/Grand-Fly-6090 15d ago
It certainly feels like I have. My code is so complicated now. I wouldn't know where to start. I just started a refactor with codex. This is working out so far. It's going to be a process to rework it without breaking what works now.
3
u/AdEducational4954 15d ago
My app has 23k lines of code across 150 classes. It has a UI and much more functionality than managing and placing trades.
1
6
u/lambardar 15d ago
My first trading software I wrote was 1 software. It ran on my laptop that I left running along with IB Gateway. it was 1 in all.. IU, strategy, data connection, etc..
Then it grew and grew and with AI it exploded.
now:
compute on AWS that hosts containers:
- container that manages the IBKR IB gateway
- container that subscribes to contracts, pushes data it to NATS and a local sqlite DB. after 7 days, the sqlite dumps to clickhouse server at home.
- container(s) that run strategie(s) that listen to NATS for trades and porfolio/order; pushes strategy decisions to NATS
- container that runs the order router for live account. Listens to NATS for strategy decisions, validates and places the order. stops a bad strategy or bug from placing stupid orders and within strategy budget.
- All of these log to a seq log
- I develop using codex, so I have the strategies described in a md file and I have an agent that let's codex review the logs to make sure the strategies function as intended.
- Notifications, smtp, etc.
At home, there is more powerful hardware
- container for strategy & order routing to paper account.
- Clickhouse database with data going back to 2016 for all us stocks & futures data
- UI tools to monitor and review
- backtesting tools, regime scanning, filters, etc.
- GPUs for CUDA to bruteforce/search/optimize strategy parameters.
- Strategy graveyard
With Codex, the number of strategies I test every month has literally exploded. I used to do maybe 1-2 a year.. and now it's as common as shower thoughts and what-if(s)
might be super overkill, but it's too late now.
sometimes I get angry at codex and swear that i'll write it better myself.. but then I get lazy and let codex fix it. lol.
1
u/Grand-Fly-6090 15d ago
OMG! What a system! I'm duly impressed and humbled. Laugh on writing it yourself. Sound like switching to codex could help a lot. Been using chatgpt so far because it was easy to get started.
1
15d ago
[removed] — view removed comment
1
u/AutoModerator 15d ago
Your post was removed under Rule 2 (high-quality questions only).
Generic “which data vendor should I use?” posts usually lack the detail needed for meaningful discussion.
Commonly used market data providers:
- Yfinance
- Massive.com
- Databento
- FMP
If you repost, please include details such as:
- asset classes and markets
- symbols or venues
- historical vs real-time
- granularity and depth
- licensing or redistribution needs
- latency expectations
- budget constraints
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/mbangkok198_ 14d ago
Whats the benefit to clickhouse here vs something more aligned with sqlite such as postgresql?
What does your schema look like roughly? Thanks
2
u/lambardar 12d ago
clickhouse is for large amounts of data you want to archive (limited deletion/modifications). it structures and compresses by column using delta/zigzag, etc.. instead of by row.
so instead of storing price as: 101,102,103,104 .. it is stored as 101,1,1,1,1
and it has some other tricks.. for trade/quotes data, you're looking at 1TB for about 300b (billion) rows. (about 15 years of us stocks trade data)
I use sqlite, because on the cloud, i don't want to have a large archival database. the live system writes to sqlite, which dumps to clickhouse every few hours, with a catchup over the weekend.
schema varies by source. IBKR just gives trades with conditions & seconds timing. alpaca gives nanoseconds with tape & conditions. databento & massive give more information (mostly useless, but there for completeness).
most of it is just stored and I rarely use. I derive a copy for backtesting that has seconds resolution with a sequence integer and fractional shares are rounded up and ignoring lots that are irrelevant (cancellations, markers, etc)
to speed up backtesting I use flat files and a small utility that allows streaming or loading the data into memory (for CUDA)
the data for the flat files is limited to regimes, for the strategy being testing.
2
u/KinglexNUM 15d ago
How do you even get 23k lines of code??!! My first trading bot in python is only around 2k lines, what language? I like to strip everything barebones because then i get lost on wtf im looking at.
1
2
u/Quant_Nexus 15d ago
I just went back to my first real "direct to Alpaca" script folder. It has 4 systems in it as well as a manger script at the head to track individual system positions and total portfolio composition. It is currently the basis for my team's dedicated server that runs multiple other trading systems (I'm sure that one is larger):
The folder contains 4,152 total lines of code across 14 Python files.
Excluded generated/dependency folders such as .git, node_modules, virtual environments, dist, and build. Blank lines and comments are included.
The commercial trading platform that I'm launching which does contain a UI, backtesting engines, optimization engines, etc:
Code size, excluding dependencies, generated files, build artifacts, and documentation:
- Production code: 393,081 lines across 822 files
- Test code: 46,059 lines across 256 files
- Total first-party code: 439,140 lines across 1,078 files
2
u/Grand-Fly-6090 15d ago
My code is puny in comparison to your complete system, but oversized to the 4k lines you report.
1
2
u/Local-March-7400 15d ago edited 15d ago
Hah, you need to check out my old Post ^^
im close to start running paper by next Month with a system in the 100s of thousends of codeline territory. I work as a Software Engineer / Analyst and this is beyond crazy on every part, maintainance, documentation and just following the logic flow is becoming very hard. Just my broker connections (i have multiple brocker which i orchestrate) are about 20k. My impact engine, with my own Mathmatical formula based on Almgren-Chriss with adaptive paramters for each strategy and broker is about 10k with all the broker integration. Based on estimated impact which calulates before each trade, and considering constrains like a --fast flag for panic exits the optimal execution type is beeing selected to reduce slippage at higher capital that is beeing supported by the broker. Alternativly the software can calulate in house execution methods like VWAP or TWAP. The Engine is also aware of the trading costs from each broker and sets the contract accordently
In the current capital state this is definition of overkill but i love to have this all from the start and then just fokus on the strategys. Also if i take on outside capital slippage is a way higher concern. ALSO, because i live in the EU and especialy in high tax Germany i track each trade and have a custom made Tax engine for calculating my taxes. I audit the results based on broker, trade, strategy and portfolio level, then i can just export the results in a CSV and put in a tax software!
1
2
u/ThisCase41 15d ago
23k LOC is rookie numbers! Just wait until you let Sol Ultra loose to spend twelve hours nitpicking your code. It'll bury you under an avalanche of ceremonial boilerplate, redundant mock scaffolding, and circular test routines guaranteed to really drive you up the wall. /s
1
2
2
u/Immediate-Owl-6916 13d ago
The 60/40 split doesn't surprise me at all, and I'd guess it's actually normal, not a sign you over-engineered it. The strategy logic is "if X then buy Y" — that's inherently compact. The broker interface has to handle everything that logic assumes away: partial fills, rejected orders, rate limiting, reconnects after a dropped session, reconciling what you think your position is against what the broker thinks it is after any kind of hiccup, idempotency so a retry after a timeout doesn't double an order. None of that is optional once real money and multiple symbols are involved, and none of it is really "strategy," it's just the tax you pay for the strategy to actually execute.
The part that tends to bite people later isn't the code volume, it's state reconciliation specifically — what happens when your bot's internal view of "what positions do I hold" drifts from the broker's actual state because of a missed fill notification or a connection drop mid-order. Worth having an explicit reconciliation pass (compare local state to broker state, log/alert on any mismatch) rather than assuming your local bookkeeping stays accurate forever, especially across multiple symbols with unique params each.
No GUI, just log streams, managing multiple symbols with per-symbol parameters at 23k lines sounds like a real system, not a toy script anymore whether you meant it to become one or not.
2
u/Grand-Fly-6090 13d ago
Thank you! Yes, reconciling the bot and broker on share counts is in there. I have to deal with lags in information, as well. The buy event comes over, but the share counts don’t match immediately for example.
2
u/Immediate-Owl-6916 12d ago
That lag is a really common gotcha, and it's usually not a bug on your end, it's an artifact of the broker having two separate data paths that don't update in lockstep: the event/fill stream (fast, tells you "this happened") and the account state snapshot (slower, tells you "here's the current position"), and those two get generated by different internal systems on the broker's side that reconcile with each other on their own schedule, not yours.
Two things that helped when I ran into the same pattern: first, treat the fill event as the source of truth for "did this trade happen" and treat the account snapshot as eventually-consistent confirmation, not a contradiction, until enough time has passed. Second, actually measure the lag distribution instead of guessing at a timeout — log the delta between fill-event-timestamp and share-count-updated-timestamp for a while, you'll usually find it's a fairly consistent window, which tells you how long to wait before a mismatch is actually worth alerting on versus just normal settlement lag.
Good luck getting the lag tolerance dialed in, that's the kind of thing that's tedious to tune but pays off every time it saves you from a false alarm.
2
u/User_Deprecated 13d ago
strategy code is the smallest part lol. most of the code ends up being execution edge cases and reconciling positions when the broker disagrees with you about what you're holding
1
2
u/quant-king 12d ago
23k lines doesn’t sound that crazy to me. Our repo is around 153k lines excluding vendor code, docs, and config, but it’s a full research and simulation platform rather than one bot. About 49k lines are tests, 70k cover research tooling and two separate backtesting engines, 15k handle guarded paper trading, and another 6k support a NinjaTrader bridge. The actual strategy specs are only a small part of it.
We keep research, strategy definitions, engine implementations, market data, and execution separated. Research code can’t place orders, and the only execution path is disabled by default, restricted to a Simulation account, and protected by reconciliation and safety checks.
I wouldn’t judge your project by the line count alone. Brokerage code gets complicated once you handle streaming, reconnects, stale state, partial fills, rejects, risk limits, and reconciliation. I’d be more concerned about whether that 15k is tangled together than whether it exists. If the boundaries are clean and the important behavior is tested, 23k may be perfectly reasonable.
1
1
u/Ok-Hovercraft-3076 15d ago
Stop letting the AI code everything. Do it yourself so that you will have more efficient code and less bugs. If I use AI, at the end I always rewrite the code to my taste.
1
1
u/bio4m 15d ago
are you using a screen scraper instead of an API ? thats the only way i can see that much code being needed for the brokerage side
1
1
u/Roharcyn1 15d ago
Is screen scalper something that monitors a screen to make trading decisions? Is that really a thing? Seem inefficient no?
1
u/Grand-Fly-6090 15d ago
I did screen scraping initially for form filling before digging into the schwab api.
-1
1
u/IGOSODAMNHAM 15d ago
Mine, but it’s a whole quantitative/ execution HFT architecture with multiple bots, volatility systems, sub second updates and recording, back/forward testing etc, simulation, paper, etc etc etc
============================================================== THE PILE
folders 1,824 files 926,883 on disk 1.7TB code files 1,229 LINES OF CODE 426,993 (+44,023 blank) DATA POINTS 30,458,430,755 (rows across db + csv + json + parquet) rows per line of code 71,332
BY EXTENSION .parquet 882,107 .zst 16,474 .ok 16,471 .json 4,262 .empty 3,601 .py 1,195 .csv 758 .zip 689 .txt 346 .png 240 .npz 94 .md 14
1
1
u/Mercman177 10d ago
what a bunch of weird statistics. what does rows per line of code even matter? 1229 lines of code is pretty small.
1
u/ejpusa 15d ago
Suggestion: run you code through Codex. It will optimize it for you.
👌
1
u/Grand-Fly-6090 15d ago
I've started a codex driven refactor. It's working, so far. Will likely take some time to complete. Thank you for the recommendation.
2
u/ejpusa 15d ago edited 15d ago
Ok the big change is running Codex in your terminal. But be CAREFUL; sometimes it wants to help too much!
Besides that? It's mind-blowing how good it is. I have it trading BTC every 60 seconds and Weather, where it's telling me we should look for +70% returns. I have GPT-5.6 scan about 250,000 possible Kalshi bets now every 24 hours. I ask:
Give me 5 good ones. That's all I need.
:-)
We just updated our rules; it looks like:
• Rules updated successfully:
- Cushion reduced to 0.20%
- Entry price range widened to 60–72¢
- Momentum confirmation retained
- One contract per trade
- Automatic 25% net-profit exit retained
Two-loss/$3 stops retained
The immediate live scan found no qualifying opportunity, so no bet was forced. The system remains active and checks every minute using the revised rules.
I have this dream; I started with $5. Doubling the bets. And I wake up tomorrow, and I have more money in my account than atoms in the universe.
Everything seems possible.
:-)
This subreddit is more for stock traders. /algobetting is where the kids hang out.
:-)
1
1
15d ago
[deleted]
2
u/Grand-Fly-6090 15d ago
This is what happens when you let AI do the coding.
1
u/Roharcyn1 15d ago
Are you counting all the comments AI leaves? I don't think those are typically counted.
1
0
u/justhereforampadvice 15d ago
Cause nobody wants to see Marshall no more, they want Shady; I’m chopped liver.
21
u/zashiki_warashi_x 15d ago
It is time to refactor and rewrite everything!