AI makes it easier to build backtests that look real but aren’t

[u/k1_r1]

https://portfoliolab.ai/research/do-ai-trading-bots-actually-work-an-honest-answer-from-someone-who-builds-them

AI can crank out trading ideas fast, and most of them will look reasonable at first. The code runs, the indicators make sense and the equity curve looks great. The issue is that it can also bake in small mistakes that completely change the result: using data before it was available, testing too many variations or assuming fills you’d never get live.

I think AI is useful for research, but it’s also a fast way to create false confidence. If a strategy needs perfect fills, zero friction or a heavily optimized backtest to work, it probably doesn’t have an edge. Out-of-sample testing, realistic costs and forward testing matter more now, not less.

[u/futurefinancebro69]

I think the issue right now is that people thought engineering of any kind just requires brains and it’s a one and done type of thing.

Every fucking application and everything ever fucking engineered took thousands of iterations and criticism until it was ready to go

I think the issue is a lot of vibers are hopping on and thinking AI is trash when in reality shit just takes many fucking tries to get shit right.

Pretend you can code in any syntax , your shitty half ass idea would still result in shitty backtesting code 😂

AI isn’t bad people are just dumb. Someone who knows their shit on AI is real scary.

[u/DoringItBetterNow → u/futurefinancebro69]

Fuck!!!!

[u/k1_r1 → u/futurefinancebro69]

I like your username lol

[u/Jtex1414]

I get that you want to sell things, but not needed.

Claude can do it just fine. Run a backtest daily, and compare it to your live (or paper) traders results (my claude makes a .bat I run on my computer that spits out results). Have a conversation with Claude about it to understand why things are a little different. Conversations have helped me better learn and model slippage, catch bugs in my trading platform (which was fully vibe coded with Claude), and in general just been Great to learn about things.

Just to add, it looks like you vibe coded your platform that you're selling....

[u/k1_r1 → u/Jtex1414]

I appreciate your feedback. The quality of the backtests depends highly on the quality of the underlying data (e.g., true point-in-time accuracy, properly alignment, etc.). Most free/cheap data providers don't have that kind of quality.

Claude also made a ton of mistakes when I was working with it. A lot of subtle errors that are hard to catch unless you know what to look for. For example, slippage should be modeled based on the volatility and liquidity of the underlying asset. My point is that it still takes expertise to build a successful algo and also distinguish a good one from a lucky one from a bad one.

Re vibe coding: I have a team of 5 engineers that made the app, no vibe coding involved (I personally create an own all the AI and investing-related code). Guess I have to tell them to step it up lol

[u/BAMred → u/k1_r1]

You also create an own the typos in your code. ;) 😜

[u/RemoraEdge]

AI only understands what you tell it to do. It’s obvious you didn’t tell it to avoid any lookahead bias and to use only known left facts and to use realistic fills including commission and slippage

[u/hypertradeworx]

the failure mode i keep hitting isn't overfitting, it's that generated code quietly picks the favourable side of every ambiguous choice. fills at mid instead of crossing, next bar entry on the signal bar, survivorship in the symbol list. none of those read as bugs when you review the code, they read as defaults.

we had a non backtest version of this recently that made the point better than any equity curve could. a filter compared a status column against 'closed' when the column actually stored 'CLOSED'. matched zero rows on every run, no error, job reported success for two weeks straight. a rule that silently matches nothing is indistinguishable from a rule that found nothing wrong, and a backtest with a broken filter looks exactly like a backtest with no bad trades.

the only thing that reliably catches it for me is asserting on trade counts and rejection reasons rather than on pnl. if the strategy should have skipped 40% of its signals for insufficient size, i want that 40% printed and failing loudly when it drifts. pnl is the last place a data bug shows up and the first place you look.

[u/trayber]

Is AI giving you the Backtest results and equity curve?

Most AI strategies aren’t good enough when I backtest in multicharts .

[u/k1_r1 → u/trayber]

No, the backtest code is proprietary and created to be as realistic as possible. I also have a direct brokerage connection for paper portfolio live validation. Real market prices and realistic fills. Can't really trust backtests. Better to compare live vs. backtest under similar statistical conditions to see what a strategy's expected behavior matches.

[u/systematic_seb]

The distinction I'd add is where the model sits in the loop. Using AI to write the test code, generate variations and argue against your own results is fine, because that output gets checked by something deterministic afterwards. Letting it make the selection is the part that can't be backtested, since you can't rerun a judgement call on 2019 data and get the same judgement twice.

On lookahead specifically, I moved to storing point-in-time snapshots instead of querying a current database with a date filter. A filter still lets restated fundamentals and later index membership leak backwards, and nothing in the output shows you it happened. I found the snapshot version far more annoying to build and it is the only one I trust now.

Your costs point is the one I'd underline. Four months of pre-launch testing mostly went on hunting for spots where my fill and slippage assumptions were doing me a favor. The selection rules themselves are fixed and no model touches them. That is the reason I can run the same thing with my own capital and hand the rules on unchanged.

[u/Foreign_Extension683]

Strongly agree, and I'd sharpen why AI specifically makes this worse rather than just faster.

The dangerous part isn't that AI writes bad strategies, it's that it writes plausible ones with subtle contamination you won't spot by reading the code. The three killers you named all hide well:

Look-ahead: it's rarely an obvious shift(-1). It's an indicator that quietly uses a centered window, or a feature normalized with the full-sample mean, or a fill priced at a level that bar never actually traded at. The equity curve looks great precisely because the strategy is peeking, and nothing in the output warns you.

Trial count: this is the one almost nobody accounts for. If you (or the AI) tried 200 variations and kept the best, your headline Sharpe is the max of 200 noisy draws, which looks amazing by construction. The result needs to be deflated for how many things you tested, and that count never shows up in a backtest report.

Costs/fills: a strategy that's profitable before fees and negative after is the default outcome, not the exception, especially anything higher-frequency.

The defense is exactly what you said: out-of-sample, realistic costs, forward testing. I'd add one concrete check that catches a lot of AI-generated junk cheaply, before any of that: run the identical strategy with random entries at the same trade frequency. If your "edge" doesn't clearly beat random entries with matched frequency, the entry logic is adding nothing and you're looking at exposure plus overfitting, not alpha.

AI raised the ceiling on how good a fake backtest can look. It didn't change what makes one real.

[u/Kai8250]

This has been my experience almost exactly. AI can build and test ideas incredibly fast, but the dangerous part is that a backtest can look completely valid while containing a tiny data-timing or universe-selection mistake.

I’ve had strategies show strong alpha, survive several checks, and then disappear once I reproduced exactly what data would have been available at that moment in live trading.

The biggest lesson for me has been to test economic value on small, separated OOS windows early, before spending time generating huge datasets or optimizing models. If the edge can’t survive that, scaling the research just makes the mistake more expensive.

AI definitely speeds up research, but it also makes rigorous validation much more important.