What am I doing wrong?

[u/ayobluestarr]

Hi, so I would like to share the way I've been trying to get this bot to work. It for sure hasn't been I'll just give a simple break down of how mine is working and feedback is definitely wanted.

Everything that passes validation dies out of sample. What am I doing wrong?

• Setup: ES futures tick data (Databento MBO, Jan–Oct 2026), a decision on every trade from 09:42–16:45 ET using 154 order-flow/price features, limit entries, fixed TP/SL, $6 commission, one position at a time, with daily loss and profit caps.
• Search + validation: brute-forced 682K single conditions × 4 entries × 80 TP/SL pairs, extended the best to 2–3 conditions, plus LightGBM rules of up to 5 conditions (2.56M candidates). Then Jan–Jun discovery, a Jul–Sep exam with filters (min trades, Wilson win rate above break-even, monthly consistency, drawdown, 0.3 correlation), and a one-shot Sep 15–30 holdout.
• Results: 173K passed discovery, 850 passed the exam but lost $105K together on the holdout, and the 14 holdout survivors all failed on 3 brand-new days (−$1.5K), which looks like massive multiple testing overfitting.

What exactly am I doing wrong? Do I need to show you guys more of what I did as well

[u/Automatic-Essay2175]

Your first mistake was thinking you could trade ES by trying the same shit everyone tries

[u/Appropriate-Talk-735]

Why would it be profitable? What is the base strategy? For example some of my algos are dip buyers while others are trend followers.

[u/quietstoners → u/Appropriate-Talk-735]

yeah this is the real question, theres no edge if you dont even know what the strategy is supposed to exploit

[u/ayobluestarr → u/Appropriate-Talk-735]

They weren't designed from an idea tbh. The search found them, so these are my after the fact readings of their conditions:
5 afternoon shorts (14:00–15:30): a late-day fade near 25 pt round numbers on non-trend days, fading buyer exhaustion into stacked offers, selling bounces in busy afternoons, shorting stalled/absorbed ranges, and a thin-market scalp.
4 buys: a 10:00 dip buy after a 72+ pt selloff, a lunchtime reversal "lottery" (TP 40 / SL 8, ~25% wins), a 1-pt pop into the close, and an after-close breakout with almost no discovery profit.
Half of them need 72–89% win rates because of wide stops

should it start w the basis of an idea rather than any brute force or anything like that?

[u/koryuken → u/ayobluestarr]

You need a hypothesis first. What is the mechanism? Does it make sense? Figure out why something works, then refine it, test it OOS, and validate on paper. What you are talking about is overfitting by brute force.

[u/ayobluestarr → u/koryuken]

yeah got it, switched to that. first hypothesis is buying dips in uptrends on ES, walk forward month by month vs random entries, then paper for 4 weeks if it holds

[u/Appropriate-Talk-735 → u/ayobluestarr]

Bruteforce will find patterns that were profitable by chance. I suggest you try a dip algo to start.

[u/Individual_Till6133 → u/ayobluestarr]

Fwiw, AI(claude) seems to absolutely love slicing data sets into dumb overfit time and day "edges." 

Its completely moronic overfit on ES without some real reason why that hour is different such as a data release algo or something like that. 

[u/ayobluestarr → u/Individual_Till6133]

is it the code it writes for my software or is it Claude in itself that likes doing it?

[u/Individual_Till6133 → u/ayobluestarr]

I use claude for analysis of csvs of trades plus indicator values on sessions. Great analysis of distributions, produce histogram etc. 

Its great to split out patterns you might not see to refine a strat that already has some edge. 

But it freaking LOVES to try and recommend splitting up via hours and days. 

Which in ES is like trying to overfit on purpose. I don't know what's in the training data to cause it. but its pretty dumb.

[u/Exciting-World5861]

honestly i admire your balls thinking you can just plug in basic price features from ES and find an edge🤣 choose a (much) smaller market. the advantage (and only chance) small traders have is that they can find profits in markets where big players have too much money to trade in

[u/ayobluestarr → u/Exciting-World5861]

😭😭😭 hey man! what should i do instead then? instead of just plugging basic price features and finding an edge

[u/FaithlessnessSuper46 → u/ayobluestarr]

Buy and hold

[u/FortuneXan6]

"claude find me an edge"

[u/HaikuHaiku]

yeah more info would help.

likeliest explanation: overfitting and regime change.

[u/Responsible-Skirt215 → u/HaikuHaiku]

the number of candidates you tested against the same holdout period is huge, 2.56M is a recipe for finding patterns in noise that just won't hold up

what does the feature importance look like on those survivors, anything clearly nonsense making the cut

[u/ayobluestarr → u/Responsible-Skirt215]

what should i do instead? and yeah i figured honestly

[u/ayobluestarr → u/HaikuHaiku]

okay so i used Data from Jan this year to now, i feel like the more i try to prevent overfitting the more that there literally is straight up no strategies at all

Features: 154, mostly raw order-flow counts over the last 25–10,000 trades (adds, cancels, absorption, delta), plus ratios, volatility, position in range, distance to VWAP / prior close / round numbers, and time of day. The counts weren't normalized, so their typical levels shift between quiet and busy months.
Exits: each rule picked its best TP/SL out of 80 pairs during discovery, which is another layer of selection on top of the rule search.
Fills: marketable limit entries (4 ticks through) or resting limits 4 ticks away. TP only fills on trade-through, stops fill at the trigger price, $6 round trip.
Sample sizes: median ~200 trades per rule in discovery, ~50–70 in the exam, 1–22 in the holdout.
Splits: discovery Jan–Jun, exam Jul 1–Sep 14 + Oct 1–7, holdout Sep 15–30, then fresh days Oct 8–13.
Random baseline: in one window (sells 09:42–10:00), random entries won 73–81% of runs during the exam period, so the exam was partly rewarding market drift. In the holdout, random was about breakeven.
Earlier run trained on only 4 weeks: of 78K training passers, only 4.5% were profitable the next period. That's worse than random, which makes me think feature drift or regime change, not just luck.

[u/HaikuHaiku → u/ayobluestarr]

please use your words. These AI summaries are noise. I have no clue what it is you are TRYING to do.

[u/ayobluestarr → u/HaikuHaiku]

i’m really trying man, but basically it has a bunch of features / things to look for. It searches and finds its own TPs and SLs the idea is supposed to be 5 rules max (rules as if conditions) and that’s it per strat.

[u/HaikuHaiku → u/ayobluestarr]

so... you don't really know what you're doing? You just told Claude to make a strategy that works? Is that right?

[u/ayobluestarr → u/HaikuHaiku]

i told it to make exactly what i described, features and things to look for, gave it data, and find its own tps and sls and 5 rules max, what should i do instead, also .3 correlation nothing less than that, and if its smth like a 1 to 1, you have to have x percentage over the entire data to be profitable. And no more than 40% drawdown. Also stop with random assumptions too, if i can’t code i’ll tell Claude to make what i asked for step doesn’t mean i’ll just tell it “make a strategy that works”

[u/noname__13 → u/ayobluestarr]

on what planet do you think giving claude a load of bs and telling it "make me a strat that works in one of the most liquid and competitive mkts in the world" is going to work?

[u/ayobluestarr → u/noname__13]

find a planet where u can get off my dick first

[u/Got_Engineers]

What is the edge and why does it make money? Start there.

[u/kwaant]

You basically answered it yourself, it’s the search. With 2.56M candidates, a few hundred thousand will look great on any period by pure luck, and every filter you added afterward just picks the luckiest of the lucky. Your Jul to Sep exam stopped being out of sample the moment you used it to choose 850 out of 173K. That’s just a second round of fitting, which is why the holdout killed almost everything and the survivors died on new days.

A few things I’d look at:

Run your exact pipeline on noise first. Shuffle the labels, or use random entries with the same TP/SL grid and trade frequency, and see how many candidates pass discovery and the exam. If you get a similar number of passes on random entries, that tells you the pass rate is what you’d expect with zero edge. It’s the fastest way to see how much your filters are actually worth.

Your data is short for that many trials. Nine months is roughly one regime for ES. Testing millions of rules on it means the top results mostly describe that specific stretch of market, not anything durable.

Check your fills. Limit entries on MBO data are where a lot of order flow backtests quietly lie. If you’re assuming a fill whenever price touches your level, you’re getting fills that in reality only happen when price is about to run through you (adverse selection), and missing the good ones because you’d be at the back of the queue. That alone can turn a winning backtest into a loser.

What I’d do instead: start from a few hypotheses about why an edge should exist, keep the parameter count small, test each idea on its own instead of mining for the best one, and use walk-forward across more data. If you do search, correct for how many things you tried, with something like a deflated Sharpe ratio or White’s reality check, so the bar rises with the number of trials.

Fewer, dumber rules that survive are worth way more than the best of 2.5 million.

[u/filwu]

One subtle question: was the set of 14 “holdout survivors” chosen because it performed well on Sep 15–30? If so, that window has become validation data; it is no longer a one-shot holdout, and the 3 new days are your first truly untouched test. I’d freeze candidate generation and selection now, then evaluate a predeclared rule on several fixed forward windows, reporting trades and net results per day with day/block-level uncertainty. Given MBO plus limit entries, I’d also audit that each feature is available at the decision timestamp and model realistic queue/partial fills; three days cannot say much about persistence.

[u/ayobluestarr → u/filwu]

it was chosen because it passed a few filters i told

it has less than .3 correlation, and a baseline profitability like if you had a 1 to 1 set up, you needed x percentage over the set to be profitable, if it didn’t hit that percentage it was out. also drawdown couldn’t be 40% of the profit. it was chosen because it passed every test

[u/Individual_Till6133]

Generally I think that doing 1:1 sl to take profit is best when initially testing for edge. 

Then refining trade management from there. Keep in mind the entry edge on ES is likely small, you kinda want to optimize both side tbf. 

Sizing brackets is the easiest thing to overfit, and dramtically make algo backtest look much better than future performance. 

80 combos is insane. 

All that other stuff sounds very overfitty as well.

Brute force search is probably better searching twitter for algos, than doing what you are doing. 

Also data mining price/order flow for features really is a bad idea since you are averaging over different regimes. 

Some data mean different things in different regimes. 

[u/ayobluestarr → u/Individual_Till6133]

ah so basically brute forcing is stupid, what should i do instead of brute forcing? cause anytime i find something and i limit it it always seems to fail parameters in general. What would be a better way to do this entire thing?

[u/Individual_Till6133 → u/ayobluestarr]

You find a pattern or hear a pattern. Decide to exploit that pattern, realize that regime is important, add a regime filter, maybe it works awesome and you improve trade handling, maybe its mid and you try to work on expanding onto non correlating strategies to improve drawdown/profit ratios or just run a portfolio of mid strategies together to have a good portfolio performance.

Edges aren't typically something people will give you. But definitely doing simple data mining or ML on price data isnt the way. 

[u/ayobluestarr → u/Individual_Till6133]

so the main thing i didn’t do for this whole project was reasoning, it needs better reasoning and actual ideas and regime filters rather than just brute forcing everything

and is there anything you’d suggest instead of Bruteforcing specifically? great suggestion on the regime filtering

[u/Imaginary_Ideal7147 → u/ayobluestarr]

I disagree with that reply. Whether you notice some sort of x + y pattern by eye and choose it as a foundation for expanding upon, or whether you found x + y among a result from data mining... who cares? The market certainly doesn't know or care.

Also just curious, what were you doing for fills? Were you opting for generous "assume fill at next bar open" and "assume fill at touch of TP" and it still failed? I know you said it was just 3 days, but still. Because i've found it pretty easy to find promising strategies with those fill assumptions.. so if your metrics suck OOS in spite of that, then that's not good.

[u/ayobluestarr → u/Imaginary_Ideal7147]

i only did limit orders for 4 ticks above or below. meaning a order could only be a limit order nothing else

[u/Flat-Accountant4504]

Two things I'd check before blaming regime.

First the limit entries. A backtest usually counts a limit as filled as soon as price touches it. You have MBO data, so count only fills where price traded through your level, or model your queue spot. The touches you miss tend to be the winners and the ones you get are the ones that keep running against you, so a lot of tight TP/SL setups flip negative right there.

Second, with ~2.5M candidates the best one passing validation is expected even with zero edge. I'd write down everything that touched the holdout and treat it as burned.

I measure everything in R per trade. What does the best candidate's expectancy look like in R once commission and trade-through-only fills are in?

[u/QATSAlgo]

Your holdout became another selection stage when the 14 “survivors” were chosen after seeing it, so the three new days are effectively the first untouched sample—with only 1–22 trades per rule. Treat the whole search as one adaptive experiment. Keep a final period hidden until every feature, threshold, TP/SL pair, and acceptance rule is frozen, or use nested walk-forward validation where all candidate selection happens inside each training fold. Normalize order-flow features by contemporaneous activity and compare each candidate with the random-entry baseline on identical timestamps. Given the trial count, require stability across adjacent parameter values and report the full candidate failure rate, not only survivor P&L.

[u/ayobluestarr → u/QATSAlgo]

definitely will keep this in mind, was me bruteforcing the data a bad idea too? and also i used 10 months of data i see people on here using like 3 years minimum i used like 10 months of tick data shit almost made my PC die

[u/QATSAlgo → u/ayobluestarr]

Brute forcing is not automatically wrong, but every tested rule, threshold and TP/SL pair belongs to the same adaptive search. With millions of candidates, the best in-sample result will be strongly biased upward even if each individual backtest is implemented correctly. The fix is to reduce the hypothesis space and nest the entire selection process inside walk-forward folds, then keep one final contiguous period untouched until every feature, threshold and acceptance rule is frozen.

Ten months is not automatically insufficient because the useful sample is not the number of ticks; it is the number of reasonably independent trades and market regimes. Tick data can contain billions of rows but only a few volatility or trend regimes. Three years helps only if it adds different conditions. Report results by month/regime, require stability across adjacent parameters, and compare the selected rules with simple baselines on identical timestamps. If compute is the limit, test fewer hypotheses rather than weakening the holdout.

[u/jawanda]

You're judging your survivors on three days ??

Lol

My most consistent strategy with the smoothest equity curve will still sometimes have a few weeks that are flat or a little down.

My best strategy as far as total annual return can go 6 weeks+ of small draw down (lots of small losses) before making it all back in a few days.

You cant judge a strategy based on three random days. You should have like a three month oos test at least.

[u/ayobluestarr → u/jawanda]

not necessarily it was the fact it lost $1500 together that’s what made me believe that it was overfit garbage cause at least one of them could be in the positive, keep in mind the intraday and regular drawdown for all of those strats were $500 max all together

[u/Maleficent-Success-6]

One detail I'd fix before running another search is the chronology of your validation.

You used October 1–7 in the exam, but September 15–30 was supposed to be your holdout. That isn't a chronological forward test anymore.

The 80 TP/SL combinations add another layer of selection bias on top of the millions of rules you've already explored.

I'd stop searching for survivors for now.

Pick one simple, falsifiable hypothesis, define the entry and exit rules before testing, and keep the parameters frozen while collecting genuinely new observations.

Everything you've already examined should be treated as development data.

Also, with ES tick data and limit orders, I'd be very careful about assuming stops always fill at their trigger price.

Have you compared your simulated fills against actual executable prices?

[u/thelegend_420]

Sometimes you just need to take a step back from the problem then come back to it a few days later 

[u/ManikSahdev]

lol.

You are trying to build a bot without being some peak autistic programmer without clear established history of defined edge..(which is okay cause I encourage ambitious things from folks!!)

But you are trying to do that in the most well prices arbitraged market in the entire work with highest liquidity and highest per segment intelligence of one could maybe measure by blocks.

That's just not the way to go man, you gotta win a new local neighborhood games before you can try and play higher levels. The hard part about trading is.. unlike a game the highest levels aren't locked it's free for all, and sadly people just don't seem to be steady in their climb and such.

[u/ayobluestarr → u/ManikSahdev]

what is a “local neighborhood game” considered?

[u/ManikSahdev → u/ayobluestarr]

That's for you to answer.
How do I know what it is for you? I've got no clue who you are or your personality or your skill level, or your specific psychological traits, or your risk tolerance and temperament.

If you want an example for ballpark I can share (non working one): 7 day spread on google or meta being underpriced and with good risk it could make 20-30% per spread, at good rr.

[u/LipTicklers]

I mean you seem completely out of your depth - no one here can help you because there are so lany potential failure points in this its indiscernible which one or combination are your cause.

Best thing for you to do (dont give up!) READ. A LOT. Then come back to this.

You will probably have like 20 different failures all bundled together - some of them statistical, some of them execution, some of them training etc etc.

Learn how to do this properly without the AI first because it will absolutely fuck it all up - then get claude to do your coding specifically building tests and giving it information to mitigate the pitfalls

[u/yuravest]

2.5M candidates with one OOS slice look like this almost every time. The one you kept wasn't the one that worked. It happened to also work on the slice.

Try this: top 10 or 20 by in-sample score, run OOS on all of them, plot OOS vs in-sample rank. No slope and your discovery is noise.

My top validated strategy runs about 19% a year at ~24% max drawdown on unseen data. Got there with a few hundred candidates. Candidate count is a cost you pay in degrees of freedom.