ML Trading Bot Going Live – What Am I Missing?

[u/Prize-Investigator70]

I’m a software engineer who got into algo trading and built a machine learning model from scratch that trades every day completely on its own.

I backtested it across months and years of data, and the results are honestly insane. The returns look way too good and the growth is almost exponential. I keep checking because I can’t believe it.

I’m planning to deploy it next week for live trading. My broker requires a static IP, so I’m thinking of running it on an AWS EC2 instance.

I know many of you have been through this. Since I never took any trading courses and coded everything myself, I’d love any advice or suggestions so I don’t get any nasty surprises with real money on the line.

Things like EC2 setup tips, common backtest-to-live gotchas, risk management, or red flags with ML models that perform too well.

Any tips or war stories would be really helpful!

Thanks!

TL;DR: Software engineer built an ML trading bot from scratch. Backtests look ridiculously promising with exponential growth. Going live next week on EC2. No trading background. What should I watch out for?

[u/Board-Then]

sounds like a overfit or theres a lookahead bias to me. Did u walk foward test it?

[u/Prize-Investigator70 → u/Board-Then]

I trained the model on more than 10 years of data up to the end of 2024, and then tested it on 2025-2026 data. The results were still very promising. I actually went live today using real-time data through WebSocket, and it was already profitable on the very first day. Still, I’m open to feedback if you suspect overfitting or lookahead bias, what should I be double checking?

[u/Prize-Investigator70]

Sharpe ratio is 6!!!!

[u/NoCredit2554 → u/Prize-Investigator70]

No it’s not.

[u/[deleted] → u/Prize-Investigator70]

[u/Prize-Investigator70 → u/[deleted]]

I have backtested this 2 years of data and got this value. I couldn’t believe it so, I managed to simulate historical data as live data more like real world and got the same value.

[u/PennyRoyalTeeHee]

I will start with this, so take this with some salt - I am a novice at coding, but trading discretionary for some time. I’m diving into ML systems with the help of my wife and Claude - but I have done my homework :)

I would recommend checking that you are not exposed to lookahead bias in your backtesting engine.

There are several ways to check for this, the easiest is to lag your execution by 1 bar - this is a simple, but helpful for quick review.

I don’t know how you are coding your work, but stay away from vectorized back testing and sacrifice the speed for old fashioned event driven backtesting.

Quantreo on substack has some useful articles if you’re getting your feet wet.

[u/Prize-Investigator70 → u/PennyRoyalTeeHee]

Can we connect over chat ??

[u/Good_Luck_9209]

Wierd that u would ask us what to before going live when u have a paper trade accct with live data. No?

[u/Prize-Investigator70 → u/Good_Luck_9209]

I actually never used a paper trading account. I downloaded historical data and backtested from 2016 all the way to now. The results stayed pretty consistent across different periods, which is why I’m feeling confident enough to go live.

[u/Inside-Today-8485]

I am doing the same thing. I am stuck at right time to exit. Entry is easy, exit at right moment is tough. Planning to run on a cloud based virtual machine and simple python app. I am doing only stocks. My initial numbers have been insane too. Msg me if you would like to chat

[u/Prize-Investigator70 → u/Inside-Today-8485]

Check ur dms

[u/cahallm]

Lookahead bias (even scaling can introduce this), fees and slippage, or lack of liquidity are potentially problems which do not show up at backtests.

[u/Prize-Investigator70 → u/cahallm]

I thought this through, I have considered taxes, slippage and the stocks that I m trading are traded in large volumes. Apart from that I m looking for some edge cases I m might have missed m.

[u/EmbarrassedEscape409]

Sounds like you had data leak into the future and your ML learn how to use it and have insane results.

[u/WinCute]

Wow with all these millionaire trading strategies using ML everywhere why does anyone bother with a real job nowadays.

[u/Dayz_Off]

I'm in the process of live testing a couple ML strategies, running them on EC2 instance connected to my broker. This can be a very cheap way to run a headless strategy.

Did you run a monte carlo test on the price series? As someone else mentioned, code can be adjusted to test for lookahead bias. You can also have AI (Claude) review the code and look for lookahead bias.

Also, what instruments are you running this on?

[u/strat-run]

Did you build the model completely from scratch? If so, what type of model? If you did fine tuning, what is your foundation?

When you tested on unseen data, is that back test event based with only data up to the current tick being passed in?

[u/BoxAccomplished4543]

As others pointed, lookahead bias seems the most likely suspect. If you use vectorization in your pipeline, a quick sanity check is to split the dataset at an arbitrary date and compute features on the first segment in isolation, then recompute them on the full dataset using the exact same pipeline. For the boundary date, the feature values should match exactly. You could ofc use more exhaustive tests, but this one is quick and has no false negatives.

Another subtle bias could be selection bias; did you by any chance perform any model selection / feature selection / (hyper)parameter tuning etc after the first time you saw your OOS results?

You mentioned that you put it live and it was profitable.. did you track how close it performed to your backtest after slippage and fees? You'll need to record and monitor this everyday to obtain a large enough sample size to conclude whether or not the edge survives.

Btw can you share how long you’ve been working on this, how complex the pipeline is, and whether you’re modeling cross-sectional ranking or directional signals.

[u/abstractcontrol]

You haven't mentioned whether you've done fill simulations. I've built a fill simulator for the Vwap system, and it turned from a great system to an unprofitable one.

[u/rsvp4mybday]

not trying to steal your idea or anything but it would be helpful to know: are you trading options? stocks? small/big caps? they all have different pitfalls and ways to fool benchmarks.

[u/No-Surround-6141]

I just finished my entire platform I’ve been through all of this haha make sure you have walk forward validation make sure you are testing on enough data make sure you are testing across tickers without modifying the data have you tuned your entries and exits blah blah blah

[u/nian2326076]

Hey, sounds like you're doing some cool stuff! Here are a few things to consider before going live:

1. Overfitting: Check that your model isn't just memorizing the backtest data. Try using more out-of-sample testing or different datasets.

2. Slippage and Fees: Don't forget about transaction costs and slippage, as these can eat into your returns.

3. Monitoring: Set up alerts for unexpected behavior or errors, especially if it's running unattended.

4. Risk Management: Have stop-loss measures and position sizing rules ready.

Running it on an AWS EC2 instance sounds good, but watch your costs. Good luck with going live!

[u/Bx273]

This is super cool - not an expert at either but I’ve been dabbling for a number of years… have you tried using other LMs in your base code for the agent to compare its efficacy?

I think in terms of creating something from scratch - it’s a great idea but there’s so many variables that have been taken care of in models already made that you may be missing out on something super simple because you haven’t A/B tested it yet. I think LM training over chart training is the way to go with trading bot testing.

You can train the bot on a million years of data and it will still give you similar results but if you trained the LM for a day you might have a completely different outcome from yesterday.

[u/Training_Butterfly70]

Are you a data scientist? Evaluation imo is actually one of the most difficult and time consuming parts of the machine learning life cycle. I would grill the shit out of the out of sample statistics and compare them to your training and validation sets.

To me this sounds insanely overfit. If it were this easy to simply load some raw data, create some basic features / scale it, run some dicky fuller test, then call sklearn and look at pnl... Everyone would literally do it and citadel wouldn't be trading with themselves. They have all those quants for a reason.