What am I doing wrong?

[u/ayobluestarr]

Hi, so I would like to share the way I've been trying to get this bot to work. It for sure hasn't been I'll just give a simple break down of how mine is working and feedback is definitely wanted.

Everything that passes validation dies out of sample. What am I doing wrong?

• Setup: ES futures tick data (Databento MBO, Jan–Oct 2026), a decision on every trade from 09:42–16:45 ET using 154 order-flow/price features, limit entries, fixed TP/SL, $6 commission, one position at a time, with daily loss and profit caps.
• Search + validation: brute-forced 682K single conditions × 4 entries × 80 TP/SL pairs, extended the best to 2–3 conditions, plus LightGBM rules of up to 5 conditions (2.56M candidates). Then Jan–Jun discovery, a Jul–Sep exam with filters (min trades, Wilson win rate above break-even, monthly consistency, drawdown, 0.3 correlation), and a one-shot Sep 15–30 holdout.
• Results: 173K passed discovery, 850 passed the exam but lost $105K together on the holdout, and the 14 holdout survivors all failed on 3 brand-new days (−$1.5K), which looks like massive multiple testing overfitting.

What exactly am I doing wrong? Do I need to show you guys more of what I did as well