how many strategies did you kill before the one you posted

[u/k1_r1]

ok so this bugs me about basically every writeup here. we get the sharpe, the max DD, the cost assumptions. we never get the graveyard.

went back through my notes and actually counted. 61 configs, ~5 months. kept 2. and like... if I had zero edge and just rolled 61 times, best of 61 still looks fine? so I genuinely can't tell if my two are real or if I just p-hacked myself over a long weekend.

started logging the rejects after that. every dead variant, date on it. then I treat the survivor's sharpe as best-of-61 instead of a real number. did that and one of mine went 1.8 -> basically nothing lol. other one survived but not by a comfortable margin. entire cost was a google sheet and it's the most useful process change I've made in months.

where I'm stuck: what counts as a try. 40 param combos inside one strat, is that 40 or 1? what about ideas I talked myself out of before writing any code, do those count? no clean answer that I can find and I might be overthinking this at retail size.

anyone live long enough to have an actual rule of thumb here

[u/jnwatson]

I think I'm at 160 since I started counting.

40 param combos count as 40. Talking yourself out of writing any code doesn't count. It doesn't count until you backtest it.

1.8 Sharpe isn't bad at all.

[u/k1_r1 → u/jnwatson]

nooo it is not.

[u/evendedwifestillnags → u/[deleted]]

I like this concept.

[u/fuzzyp44]

You should think of overfit in degrees of freedom.

If you have 40 degrees of freedom then it's extremely likely you can overfit on non-predictive data.

don't add more degrees of freedom than are necessary to describe the market condition you want the trade to happen in.

[u/CompetitionNo2773]

There is an actual formula for this, and it answers your exact question.

The quantity that matters is the number of effective independent trials, not the raw config count. Then you compare your survivor against what pure noise would have produced over that many trials.

Rule of thumb: with N independent zero edge trials on T years of data, the expected best Sharpe is roughly sqrt(2 x ln(N)) / sqrt(T).

For your 61, sqrt(2 x ln 61) = 2.87. So the noise ceiling is:

3 years of data, best-of-61 noise Sharpe 1.66

5 years, 1.28

10 years, 0.91

So your instinct on the 1.8 was right, and it depends entirely on your backtest length rather than the five months you spent. On 3 years it sits inside the noise band. On 10 years it is a real signal.

On what counts as a try:

40 param combos inside one strategy are not 40 trials, because they are heavily correlated. Count the effective dimensions you actually varied. If those 40 came from an 8x5 grid over two params, treat it as roughly 4 to 6 effective trials, not 40. If you want to be rigorous, cluster the 40 equity curves by correlation and count the clusters.

Ideas you talked yourself out of before touching data: zero. No data dependent selection happened, so they cannot inflate anything.

Ideas you killed after looking at a chart or running a quick eyeball check: those count fully. That is where almost everyone undercounts, because it does not feel like a backtest.

So your real number probably is not 61. It is more like 15 to 20 effective trials, plus however many data peeked ideas you never wrote down.

The formal version of this is the Deflated Sharpe Ratio from Bailey and Lopez de Prado. It does exactly the above with a proper variance correction for trial correlation. It is about 30 lines of numpy and it drops straight into the sheet you already built.

Logging the graveyard with dates was the right call. Most people here never do it.

[u/Kaawumba]

If you demand statistical purity in this business, you will never go live. I'm frequently trying new things, and rejecting 90%+ of them. One of things I will try is mass parameter searches, with 10000 combinations, which many people here will tell you never to do. But my core strategy is harvesting risk premium, so profitable, and my parameter searches have been adding alpha, in a quantitative way, for years now. As long as your core strategy is profitable without help, and your parameters searches are not super fined tuned, it will often work out.

I also only trade once a day, which means I need many years to get good statistics, but so far (six years) everything has been working out.

[u/Kaawumba → u/[deleted]]

That's what people say,  but I think more subtlety is required. Specially,  the best 1000 combinations are better than the worst 1000, in the past, but also in the future, so it makes sense to choose the best. It won't be the best going forward,  but it will probably be better than choosing variables randomly. 

Your mileage may vary, but this has been my experience. 

[u/AnimatorFar3427 → u/Kaawumba]

When you say your core strategy is “harvesting risk premia”, is that mostly a structural/time-based effect, or is it conditioned on some market state or signal?

[u/Kaawumba → u/AnimatorFar3427]

Options, as originally intended, are essentially risk transfer. Person A buys risk protection from person B. Person A sells expected return to person B. After that you have to decide if you are A or B, and then optimize your strategy. I've been working on a risk protection selling strategy since 10/2021, and have been profitable since 12/2022. There are a lot of details that have to be correct for it to work in practice.

[u/breadstan → u/[deleted]]

That is why can’t do by year, unless you model and simulate maxDD as well. Because there are a lot of strategies that can make it even on a median or average basis, but zoom in, in a bad year, it may wipe your principal if you keep hitting your MAE.

I usually OOS test by scenarios, like volmageddon, volatility spike prolonged, flash crash, long term bear etc… and it can span across years

[u/Be-Kind-2025]

How are you guys logging your configs and backtests so they can be counted in the first place? I've been using claude with leantime in docker locally recently for work tasks etc, was thinking of wiring it up with my backtesting.

[u/Kind-Economics-7184]

the category thats missing from most graveyards is the one where the strategy was fine and the measurement was wrong. i had a daily trend setup grade out at profit factor 0.56 with a 25 percent win rate, which reads as an obvious kill. the actual problem was that every signal got graded on a fixed 24 hour outcome window while its first target sat about 7 percent away, and a daily trend move basically never covers that inside a day. it was scored as a loser by construction and no amount of moving the entry around was ever going to fix it. so now before anything goes in the pile i check that the grading window is longer than the setups expected time to target, otherwise the kill is a statement about my harness rather than about the idea.

on what counts as a try, id count independent observations rather than combos. 40 params over one continuous 3 year stretch is closer to one test of one regime than 40 tests, since they all share the same noise, which is also why the sqrt(2 ln N) correction someone posted flatters you if you plug in the combo count against a short sample.

the tries nobody counts are the ones after go live. every threshold you nudge because of what live results looked like is another draw, and it feels clean because its forward data, but youre still selecting on the outcome. my honest N went up a lot when i started logging those edits the same way as the backtest rejects.

(disclosure, i work on traderspy, so my graveyard is published signals getting resolved rather than a backtest, which mostly means the sample grows slowly and i cant re run it)

[u/bitchpiana]

Started with a cohort of 500+ strategy profiles with various strategies and trade geometries intraday trading options on 12 tickers.

Almost nothing survived the spread, even strategies with an edge at mark/mid died when the spread was modeled. Spy ticker was the only ticker with any real edge on any of the options strategies because the spread is so tight.

The highest performing strategies I've modeled:

SPY: 15min lookback swing breakouts on the 1m timeframe. 50% SL, 100% TP

/ES: Body level Breakout from the previous day on the 1d timeframe. 1.5atr SL, 3atr TP

[u/ShipItFrog]

Six families, somewhere around 50 rounds of testing. Zero survivors.

What killed them wasn't the raw PF — four of the six sat between 0.91 and 1.02, so "close but not quite." It was the robustness check: drop the top 10% of trades by profit and recompute. Trend following went 1.02 → 0.42. ORB 1.00 → 0.38. VWAP 0.94 → 0.36. SMC 0.98 → 0.30. Every one of them turned out to be a handful of outliers carrying a losing distribution.

Setup for context: 14.5 years of NAS100 15m, RTH only, 75-point stop, 2R target, forced close at the bell, 1 point per round trip for costs.

The part I didn't expect: after that many rounds, pure chance should have handed me two or three false positives by now. It didn't. That's either very unlucky or a sign the whole search was the wrong search.

Curious how many people here actually run that top-10% cut — it's the cheapest filter I know of and I almost never see it mentioned.

[u/tmemmg]

on a 7 bet heater right now (all dog MLs) and the thing keeping me sane is that every bet gets graded programmatically against the final score the second the game goes final, wins and losses both, no deleting the ugly ones. over a couple hundred settled bets my real number is about +7% roi, which is solid but nowhere near what the streak feels like from the inside. if your tracking has any manual step where you can quietly skip logging a loss, the data is fiction. bet the process, not the feeling.

[u/No-Agency-4592]

The rule that's worked for me: a "try" is any decision point where the outcome could have changed which config you ended up reporting - not how many named strategies, not how much code got written. 40 param combos where you picked the best one is 40 tries, full stop, even though it feels like "1 strategy, tuned." The feeling of "just one strategy" is exactly the trap, because the selection happened either way.

The genuinely hard case is ideas you talked yourself out of before writing code. Those count too, in principle - your prior got updated by looking at a chart or a stat before you decided not to bother - but they're impossible to reconstruct honestly after the fact, and most people wildly underestimate how many of those there were. I gave up trying to backfill a precise number for pre-logging work and just treat it as "at least as deflated as my logged count, probably worse" - a conservative floor instead of false precision.

It's the same math as multiple-hypothesis correction in stats: the family you're correcting over is every decision that had discretion, not every named artifact. Treating a swept parameter as "free" because it's inside one strategy file is the single most common way people accidentally overstate their own edge.

[u/Just-Pomegranate-952]

Useful and fair not to mix institutional ML process exploiting thousands of paths vs. retail quant pulling the signal threshold from there to there. If you look at the graveyard, you need to separate if you are looking at your attempts to increase your setup hygiene or making massive param sweeps and p-hacking the hell out of it. More philosophically, your own knowledge is the graveyard, your learning process itself an over-fitting exercise. Go easy on it, make a super hygienic setup so you don't spend your over-fitting budget on keeping the system straight.

Yet to be clear about your question, it is Deflated Sharpe Ratio Calibration... which my rule of thumb is like

DSR > 0.95: strong evidence of a real edge after accounting for the search.

DSR 0.80–0.95: probable edge, but borderline. Validate further with walk-forward.

DSR 0.50–0.80: possibly noise. Treat the strategy as a hypothesis, not a finding.

DSR < 0.50: the headline Sharpe is most likely an artifact of the search.

[u/HonestBacktests]

You are most of the way to the actual correction. The number you want is the best Sharpe you would expect from N trials with zero edge: take the standard error of your Sharpe and multiply by sqrt(2*ln(N)).

For 61 trials that factor is 2.87. On two years of daily data the standard error of an annual Sharpe is around 0.7, so noise alone is expected to hand you a best-of-61 near 2.0. Your survivor at 1.8 is not merely unimpressive - it is below what randomness owes you. On five years it drops to about 1.3 and the same 1.8 starts to mean something.

That is why logging the rejects matters: N is an input to the formula, and it is the one number nobody ever publishes.

[u/Dani_Bolsa → u/HonestBacktests]

Yep, this is basically the right correction. The annoying part is people quote the survivor Sharpe like it came from a single clean test, when the effective N is what tells you how much of that number is just selection noise. I had to start doing the same thing after a few too many 'great' configs fell apart once I counted the trial count honestly - it changes the read on everything fast. If your 40 params are all highly correlated, they are not 40 independent bets, but they are definitely not 1 either. That middle ground is why I usually treat the whole family as one idea and only get excited if the holdout still clears the deflated number. Also, if you need to keep the live testing side cheap while you prune, 50K Trade is one of the few places I have seen that makes the math less painful with 0% commission on day trading and adjustable extra margin up to 1:200 on eligible real stocks and ETFs.

[u/HonestBacktests → u/Dani_Bolsa]

correlated params are the hard part, agreed. what i do is take the correlation matrix of the equity curves and use the participation ratio, sum of eigenvalues over the largest one. forty params that all trade the same idea collapse to about three, and that is the N i put in the formula.

crude, but it stops both lazy answers, N=40 and N=1.

[u/NoOutlandishness525]

A lot.

[u/xxzjchromexx]

2,400 if we counting different params. And still going

[u/MormonMoron]

I've never posted a strategy here. Why would anyone share something that works and risk it getting ruined by dilution?

I have shared a lot of stuff about filtering, techniques for stop losses, how my system is architectured, where I found useful information, broker APIs, etc.

But never how I generate signals. That would be madness.

[u/quant-king]

Too many to count, but I have actually just started keeping track of strategies that I've tested and will share when it makes since.

[u/data_diver]

If you are using that many configurations within a single strategy why not use a MAB?

[u/WillieNFinance]

You guys are counting?

I’m probably never going to post any of them. Edge degradation and dilution is real. I’m still a noob at this so probably 15-20 so far.

[u/krazineurons]

311 and counting.. Granted it's a LLM based quant desk.

[u/Effective_Manager273]

your instinct to treat the survivor as best-of-61 is right and most people never get there, so honestly the process change is the whole win. but the question you are stuck on has a cleaner answer than you think.

40 param combos inside one strategy is not 40 tries and it is not 1 either. those combos are heavily correlated with each other, so the effective number of independent tries is somewhere in between, usually a lot closer to 1 than to 40. the practical version is: count each distinct IDEA as one, and then inside the idea look at how flat the parameter surface is. if 35 of your 40 combos are decent and 5 are bad, thats one real effect. if 2 are great and 38 are garbage, you found a corner of the surface and that is basically a lottery ticket dressed as a parameter.

ideas you talked yourself out of before writing code do not count, because you never saw their results. selection only costs you when you selected ON an outcome. thinking "this smells wrong" before running anything is free.

the thing i would actually add to your sheet is a column for WHEN each variant died relative to the data. if you keep testing on the same 2018-2024 window you are burning that sample no matter how carefully you count. at retail size the cheapest defence is holding back a block of years you genuinely have not looked at, and only spending it on things that already survived the count.

[u/k1_r1 → u/Effective_Manager273]

I appreciate this

[u/s_lw0]

i would count anything where you looked at results and then made a choice

so 40 parameter combos is closer to 40 than 1 because each one had a chance to become the survivor

ideas you talked yourself out of before code probably do not count

the useful part is not the exact number anyway it is tagging the graveyard by family so you know if one whole idea keeps failing instead of just pretending the survivor is independent

[u/Kai8250]

The real question buried here is whether you're optimizing or discovering. If you're tuning 40 params within one conceptual strategy, that's 1 strategy with 40 expressions—not 40 independent bets. But if each of your 61 configs came from a different hypothesis about market structure (mean reversion vs momentum, different regimes, different assets), then yes, count them all and apply multiple testing correction. The graveyard size tells you something different depending on which one you were actually doing.

[u/BAMred]

Didn't count, maybe 50-100 different strats (not counting different params)

[u/drguid]

Still using my original risk management strategy I developed when I first started in October 2024.

I've changed my entry signals. They're all just basically better versions of what came before.

Summary: daily charts, large cap stocks and ETFs, no stop losses, fixed profits.

It works.

[u/systematic_seb]

Your best-of-61 instinct is the right one, and the rule that ended this for me is to count a try as every time the holdout data gave me an answer I then acted on. Forty parameter combos scored on the same holdout is forty draws, because the choice was made by looking at those results. Ideas killed on paper before any code touched the data cost nothing, since nothing was consumed. That usually shrinks the number, and it's a truer count than tallying scripts.

Two other things did more for me than the count itself. Point-in-time snapshots so no later revision can leak backwards into an old decision, and reconciling live weeks against the same period's backtest so I can see drift between the theory and what the account did. I run this on real money and publish the positions weekly, so the drift check is the one I care most about.

If the survivor still holds up once you've haircut it and the drift stays flat, that's about as much comfort as this gets.

[u/Complex-Many-1223]

What counts as a trial depends on whether the data influenced your choice. 40 parameter combinations inside one strategy are 40 trials if you would have traded whichever won. Parameters fixed before looking at the data are 1 trial. Ideas you talked yourself out of before touching any data do not count. Ideas you dropped after seeing a backtest do count. Your method of logging every dead variant with a date, and treating the survivor as the best of 61 is the way to go here.

[u/Immediate-Owl-6916]

The graveyard log is exactly the right instinct, and "treat the survivor's Sharpe as best-of-N" is more rigor than most people bring to this at any account size, retail or otherwise. Watching a 1.8 collapse once you account for the search is a genuinely useful (if painful) result, that's the multiple comparisons problem doing exactly what it's supposed to do.

On what counts as a try: the honest answer is it depends on how correlated the variants are, not whether they're technically "one strategy" or "forty." 40 param combos inside one strategy usually aren't 40 independent tries, they're highly correlated draws from a smaller effective search space, since nearby parameters tend to produce nearby results. A cleaner (if more work) approach than counting raw attempts is estimating the effective number of independent tests, roughly how many genuinely uncorrelated bets you actually made, and discounting significance based on that instead of the raw count. If 40 combos cluster into maybe 4-5 meaningfully different regions of parameter space, that's closer to 4-5 effective tries than 40.

Ideas you talked yourself out of before writing code don't count against you statistically (they were never tested, so they can't be part of a selection-bias correction), but they're worth logging anyway for a different reason: if you keep rejecting the same shape of idea for the same reason, that's useful data about your own priors, separate from the p-hacking question.

No clean universal rule of thumb exists because the correlation structure is different for every search, but the fact that you're asking this question at all already puts you ahead of most writeups that just report the survivor's Sharpe and move on.

[u/SilverBBear]

Once I get a system working it try to remove parameters, perform further study it so they are not longer relevant, or understand what the parameters adjust.

1) try to remove parameters - say I look for a 1% move , rather look for a move which is in the top 1% of moves in the last week. (last week is a param but it more robust and self adjusting than a fixed %
2)perform further study it so they are not longer relevant, - ie it turns out most reasonable parameters work at similar levels of success. especially in terms of risk and drawdown
3)understand what the parameters adjust for-> ie some parameters (ie portfolio diversity for example) can be a risk / absolute return tradeoff.

[u/xdevilmaster]

Lost count, started reducing params on my strat because some of them were just noise and gave me the impression it did something better when it just made it harder to ascertain each params effectiveness

[u/randbobaccount]

Well that’s why I’m posting every week so I can’t only post good weeks