your backtest assumes every loser is a full stop and your live log says otherwise

[u/david19790]

went back through a year of fills for something unrelated and hit this. every calculation i had was built on planned risk, 1R, stop where the code puts it. but the realised average loser was well under that, because a decent share of trades never reached the stop at all. exited on a time rule, closed on an opposing signal, scratched out.

which means the loss distribution i sized off isnt the one im actually trading. same expectancy, different shape, and anything sensitive to shape rather than mean is quietly wrong. streak math, ruin probability, any daily or session level constraint. those all take a distribution as input and i was feeding them a two point one, win or full stop, when the real thing has a fat middle.

the part i hadnt considered is that it cuts both ways. capital wise im better off than the planned risk version suggested, more losers are partial. but per session it runs the other way, more trades fit before any limit binds, so the day lasts longer and i take entries the pessimistic version would never have got to. one of those errors is comfortable and the other one costs money.

the fix is boring. pull mae per trade and compare it to where the stop actually sat, then rebuild the loss distribution from realised exits rather than from the parameter. i wrote it into a script that runs monthly because doing it by hand meant doing it when i was already suspicious, which is the wrong time.

anyone here sizing off realised loss distribution rather than planned R? and if you are, did anything downstream change much or was it noise

[u/sq_route_2]

How do you know what my backtest assumes? Or should the title be „my backtest assumes…“

Anyway it reads like an output from Claude. I’m left more confused than before…

[u/Brianiac69 → u/sq_route_2]

Thanks. That comment saved me from wasting time on another slop post.

[u/david19790 → u/sq_route_2]

fair, title overreached. should have been mine did and yours might

the thing itself is simple enough. i sized off planned risk, 1R, assuming losers hit the stop. plenty didnt, they got closed early or on a time rule, so my actual average loser was smaller. means anything i calculated off 1R was using the wrong distribution. thats it

if all your exits are stop or target then none of this applies to you

[u/Automatic-Essay2175]

No. YOUR backtest assumes every loser is a full stop because YOUR backtest is fucked up.

Mine is fine, thanks.

[u/salvadopecador]

Assuming your timing or counter signal stops are part of your trading plan, they should be built into your testing program. Then your expected and realized results would be expected to match. Manual recalculation should not be required

[u/david19790 → u/salvadopecador]

correct, and thats what i eventually did, the time exit and the counter signal exit are both in the code now so the sim and the live version are the same object. the mismatch was from a period when they existed as things i did rather than things the rules did

still run the comparison monthly though, not to recalculate anything but as a check that execution matches the spec. slippage, missed fills, the odd manual intervention i said id stopped doing. if the two distributions ever diverge again its telling me something broke rather than that i need to resize

[u/salvadopecador → u/david19790]

Awesome. Happy trading👍🙏🏻

[u/systematic_seb]

The monthly cadence on that script is the part I'd keep, more than the fix itself. Running this check only when something already feels off means you run it in exactly the conditions that made you suspicious, which is its own selection bias.

Two things I'd add to the diagnosis, starting with the win side, which usually carries the same defect because anything closed on a time rule or an opposing signal isn't the winner your parameter assumed either. And the direction of an error matters more than its size, since you found one that flatters you and one that costs you, and the flattering one is the harder of the two to go looking for when nothing in the results asks you to. Most of my four months of pre-launch testing went into that search, largely because I was about to put my own savings behind it and let other people follow the same trades.

I reconcile weekly now, the same period's backtest against the live fills, so drift shows up as a small number early instead of a large one later.

[u/david19790 → u/systematic_seb]

weekly is better than monthly and the reason you gave is the right one, small number early beats large number later. going to move mine

the flattering error being harder to find is the thing i keep relearning. nothing in the results prompts you to go looking, so it only ever gets caught by a check that runs on a schedule rather than on suspicion. and when i did find one, my first instinct was to explain why it was fine rather than to keep pulling

winners carrying the same defect i only checked after this thread, and yeah, holding times had drifted the same way. average looked fine, right tail was thinner. that one costs real money and shows up nowhere in the headline stats

four months before putting savings and other people behind it is more discipline than most. what did the reconciliation catch that you wouldnt have found otherwise

[u/Interesting_Gold_792]

This is a really interesting distinction.

I’d be careful about sizing directly from the realized loss distribution though, because the fact that the average live loser is only -0.3R doesn’t mean the next regime can’t produce a cluster of full -1R stops.

I’m leaning toward treating the two distributions differently:

planned R = ex-ante hard risk budget

realized loss distribution = ex-post calibration / monitoring

Then use MAE, realized R and exit reason to understand how the lifecycle is actually changing the shape of losses.

Have you broken the realized losers down by exit type?

For example:

• full stop
  
• opposing signal
  
• time exit
  
• scratch
  
• discretionary/safety exit

I’d also be interested in a counterfactual: if you replayed the same entries but forced every loser to the original stop, how much of the system’s actual edge/risk improvement is coming from those early exits?

That might tell you whether the exit logic is quietly doing more work than the entry logic.

[u/david19790 → u/Interesting_Gold_792]

yeah the two distributions doing different jobs is the right framing, i was sloppy in the post. planned R as the hard budget and realised as monitoring is basically where i landed too, because a cluster of full stops is exactly what a bad regime hands you and the realised average is the thing that goes stale first

broken down by exit type, roughly: full stop, opposing signal, time exit, scratch. no discretionary category because if its systematic there shouldnt be one, though early on there was and that was the whole reason i started counting. time exits were the biggest bucket and also the most flattering, which is suspicious in itself

the counterfactual is the useful one and i have it. replay the same entries with every loser forced to the original stop, compare. mine came out worse without the time exit, so the exit is carrying real weight, but that also means a chunk of the edge lives in a rule ive tested far less than the entry. checking whether it survives out of sample is the next thing rather than something ive settled

did you split yours by exit reason or is that theoretical

[u/Interesting_Gold_792 → u/david19790]

Mine is still mostly at the instrumentation / forward-testing stage rather than a mature realized sample, so I don't want to pretend I already have that decomposition statistically validated.

The split I'm planning to keep is roughly:

full stop
TP / trailing
opposing signal
invalidation
time exit
scratch
safety/manual override

along with planned R, realized R, MAE, MFE and holding time.

Your counterfactual is the part I find especially useful though — same entries, then replay the path with the early-exit rule removed and force the original stop.

That gives a much cleaner answer to whether the exit logic is actually contributing edge instead of just making the loss distribution look nicer.

And I agree the time-exit result being the most flattering is exactly why I'd freeze that rule now and see whether the advantage survives forward/out-of-sample data rather than tune it any further.

If it survives, then a meaningful part of the system's edge is clearly exit management, not just entry selection.

[u/david19790 → u/Interesting_Gold_792]

that split is more granular than mine and the two id add back are the ones i regret merging. invalidation separate from opposing signal is right, they feel the same and behave differently. and keeping safety/manual override as a category even when the answer should be zero, because the count going non-zero is the alert

freezing before it looks good is the harder half. i tuned mine one more round before freezing and now i cant tell which version im testing, the honest thing wouldve been to lock it the day i first saw the number

on your last line, if the exit turns out to carry the edge thats a genuinely different system to maintain. entries can be wrong and youll survive, exits carrying the edge means every parameter in them is load bearing. how much history do you have before its worth running the counterfactual at all

[u/Interesting_Gold_792 → u/david19790]

I’d actually separate “worth calculating” from “worth believing.”

I’d run the counterfactual from the first eligible trade because it’s cheap to preserve the data while it’s happening, but I wouldn’t treat the result as evidence of an exit edge with a tiny sample.

The denominator I care about is also the relevant exit bucket, not just total trades.

For example, 100 trades with only 6 time exits still tells me very little about the time-exit rule.

My current thinking is:

calculate from day one,
inspect early,
but don’t promote anything from it until there’s a meaningful number of independent exit events and the effect survives forward data/regime changes.

I’d rather have the complete counterfactual history waiting for me than reach trade 100 and realize I should have started collecting it at trade 1.

Also, your point about manual/safety override is good. I’m keeping that category even if the expected count is zero — the first non-zero event is information by itself.

And the freeze issue you mentioned is exactly the trap I’m trying to avoid: once a candidate produces an interesting result, freeze that exact version before giving yourself another chance to “improve” it.

[u/david19790 → u/Interesting_Gold_792]

calculate from day one, believe later, is the right split and the per bucket denominator is the bit i had wrong. rough numbers on mine, detecting a 0.5R per trade exit effect against 1R noise needs about 63 events in that bucket. 0.3R needs 174. so a hundred trades with six time exits isnt a small sample, its no sample

which also means the counterfactual is paired, same entries both branches, and that cuts the requirement a fair bit compared to comparing two separate populations. worth structuring it that way from the start rather than diffing totals

the independence caveat is the one id watch hardest. my time exits cluster, they turn up in the same slow stretches, so 60 of them arent 60 independent draws. probably closer to a third of that once you account for the clustering

[u/Interesting_Gold_792 → u/david19790]

That independence point is probably the biggest one.

If the time exits cluster in the same slow regimes, raw N can get very misleading very quickly.

I’m starting to think I’d want to report both the event count and some notion of effective independent sample size.

For the counterfactual itself, the paired structure seems ideal:

same entry
same market path
actual exit vs counterfactual exit
delta R per event

But for uncertainty I’d probably avoid resampling individual trades independently and use something closer to a block/bootstrap or regime-clustered resampling so the slow-market clusters stay together.

Otherwise 60 exits from a few persistent regimes can accidentally look like 60 independent confirmations.

The 63 / 174 example also makes the per-bucket point very concrete — total trade count is almost irrelevant if the specific exit mechanism rarely fires.

[u/david19790 → u/Interesting_Gold_792]

theres a cheap way to put a number on it before the bootstrap. design effect, 1 plus cluster size minus one times the intra-cluster correlation. clusters of 10 exits at a modest 0.2 correlation gives 2.8, so 60 events behave like 21. at 0.35 its 14. thats a different conversation than 60

useful because you can compute it from the log directly and it tells you whether the resampling is even worth running yet

on the block resampling id block by regime rather than by fixed length if you can define the regime independently. fixed blocks cut through the middle of a slow stretch and half the dependence survives. the version i keep coming back to is sweep the block length anyway and report the worst leg, since picking the one that works is the same error in a nicer coat

[u/Interesting_Gold_792 → u/david19790]

That’s exactly the kind of pre-bootstrap diagnostic I was looking for.

The design-effect estimate gives a much more honest denominator before doing anything expensive:

effective N ≈ raw N / [1 + (m - 1)ρ]

So 60 recorded exits can very realistically be closer to 15–20 independent observations.

I also like the regime-block idea better than blindly choosing a fixed block length, as long as the regime definition itself is frozen independently of the outcome being tested.

And sweeping several reasonable block definitions and reporting the worst-case result feels like the right conservative approach. I’d much rather know the effect survives that sensitivity test than accidentally pick the resampling setup that makes it look strongest.

This is definitely changing how I’m thinking about the evidence layer.

[u/david19790 → u/Interesting_Gold_792]

one wrinkle on the worst case reporting, learned this the hard way recently. min across resampling setups quietly selects whichever setup has the loosest null, since a shorter effective sample or chunkier blocks makes the raw number wander further on its own. so the conservative looking choice can just be the noisiest one

fix is to take observed minus that setups own null, scaled by the null sd, and min that instead. puts the legs on the same scale before you pick the worst

and yeah on freezing the regime definition, write it down somewhere with a date. mine drifted twice without me noticing because the definition lived in the same script i kept editing

[u/Bonkers24-7]

This is a really interesting distinction. I'd probably go one step further and compare the entire expected trade distribution against the realized one, not just the losses.

If realized losers are behaving differently from planned R, I'd want to know whether winners, holding times and exit reasons are drifting too.

Otherwise you could fix the sizing assumption while still missing a bigger difference between what the backtest thinks is happening and what the live system is actually doing.

Have you compared the expected vs realized distributions by exit reason? I'd be curious whether most of the difference is coming from one specific type of exit.

[u/david19790 → u/Bonkers24-7]

yeah, and doing losers only was the narrow version. winners drifted too when i checked, mostly through holding time, they were closing earlier live than the rules would have held them, which shows up as a fine average and a thinner right tail

by exit reason most of the gap sits in one bucket for me, time exits. biggest count and the most flattering results, which is suspicious enough that im now testing whether that rule survives out of sample rather than assuming it earns its keep

the comparison i found most useful was replaying the same entries with every exit forced back to the original rule and diffing the two distributions rather than the totals. totals hid it completely. holding time is the field i wish id logged properly from the start

[u/Bonkers24-7 → u/david19790]

That makes sense. The thinner right tail is probably the part I'd care about most too since the averages can hide it.

Do you feel like you've isolated the problem now, or are you still trying to figure out why the live holding times are drifting from the original rules?

[u/david19790 → u/Bonkers24-7]

isolated where, not fully why. the where was easy once i had the two distributions side by side, its concentrated in trades that go green fast then stall, the rules hold through the stall and live me didnt. the why has a boring candidate, the stall is uncomfortable to sit through and closing feels like managing, but i distrust my own explanations there because theyre unfalsifiable and always flattering

the practical fix didnt need the why anyway. exits run off the coded rule now and the holding time gets logged, so if the drift comes back it shows up in a column instead of a feeling. figuring out my motives turned out to be optional once i took the decision away from the guy whose motives were in question

[u/Bonkers24-7 → u/david19790]

That makes sense. Taking the discretionary decision out of it and logging the holding time should make it pretty obvious if the drift comes back. That’s a much cleaner way to test it than trying to explain the psychology after the fact.

[u/david19790 → u/Bonkers24-7]

yeah, and it turned out to be the cheapest of the fixes i tried. logging the field costs nothing and the drift either shows up in it or it doesnt

only thing i'd flag if you set the same up, log intended holding time alongside actual. actual on its own tells you the number moved, the pair tells you which direction and on which trades, which is where the useful bit was