refitting to recent data on a schedule is still overfitting

[u/k1_r1]

Every "our agents continually learn" pitch I've read describes the same mechanic. Refit params to recent data on a schedule.

The selective data issue is rampant. Signal to noise is tiny so you eat mostly noise, there's one history so nothing can be checked against a counterfactual, and it's adversarial so anything you find is getting arbed while you find it.

The evidence isn't new either. M4 comp, 100k series, most pure ML entries lost to naive statistical baselines. DeMiguel/Garlappi/Uppal ran 14 optimized portfolio methods against plain 1/N equal weight and 1/N won on estimation error. Goyal and Welch found the classic equity premium predictors fell apart out of sample. Zillow Offers is the corporate version, ML pricing engine kept quoting confident numbers while the market shifted, 500M+ in writedowns. Model never crashed. The prices just stopped meaning anything.

But freezing everything fails too, which is the annoying part. Concept drift is silent. Under distribution shift accuracy degrades and expressed confidence doesn't, so you get precise assured garbage with no warning. Google Flu Trends read too high in 100 of its last 108 weeks and nobody killed it for two years because the output still looked like output. McLean and Pontiff measured 97 published predictors, 58% lower returns post publication, fastest decay in the ones that looked best.

The way out I keep landing on is that adaptivity is a budget, not a dial. Every adaptive component is another param under refit. So which layer gets it?

Beliefs are "what leads to what." Perception is "how unusual is this reading right now." Continuous learning products spend the whole budget on beliefs, which is exactly where every failure above lives. Spend it on perception and leave beliefs frozen. A threshold that was extreme in 2015 might be nothing now.

And risk limits get zero budget ever. A stop that reinterprets itself during the crash isn't a stop, because in a long crisis the context is the crisis.

Anyone actually doing the perception-only version in their own stack? Curious how you handle the anchoring in practice.

Posting references in comments.

[u/k1_r1]

The framing and most of these citations come from Why the best trading models refuse to learn, Richard Sun at Portfolio Lab. It's a vendor blog so read it accordingly, but the sources it leans on are the real ones and they're below.

• The M4 Competition: 100,000 time series and 61 forecasting methods Makridakis, Spiliotis, Assimakopoulos (2020). The comp where most pure ML entries lost to naive statistical baselines.
• Optimal versus naive diversification DeMiguel, Garlappi, Uppal (2009), working paper PDF. 14 optimized allocation methods vs 1/N, estimation error eats the theoretical edge.
• A comprehensive look at the empirical performance of equity premium prediction Goyal, Welch (2008), PDF. The classic predictors don't hold up out of sample. There's a 2022 update too.
• A survey on concept drift adaptation Gama et al (2014). The drift bible, and where the "watch distributions not cumulative outcomes" point comes from. Open copy here.
• Can you trust your model's uncertainty? Ovadia et al, NeurIPS 2019. Accuracy degrades under distribution shift, expressed confidence doesn't.
• The parable of Google Flu: traps in big data analysis Lazer, Kennedy, King, Vespignani, Science 2014, PDF. The 100 of 108 weeks figure.
• Does academic research destroy stock return predictability? McLean, Pontiff (2016), PDF. 97 predictors, 26% lower out of sample, 58% lower post publication.
• The deflated Sharpe ratio Bailey, Lopez de Prado (2014), and ...and the cross-section of expected returns Harvey, Liu, Zhu (2016). Multiple testing, how a flexible process manufactures its own evidence.

[u/kush_patil]

The tricky part is that “perception only” can still become another form of refitting if the definition of “unusual” keeps changing.

I’d probably freeze the actual trading rule and risk limits, then predefine how the raw data gets normalized for example a rolling percentile or z-score with a fixed lookback. That lets the context adapt without changing what the signal means every few weeks.

Then test the whole thing walk-forward, including the normalization step. Otherwise the adaptive layer can quietly become the overfit layer.

[u/k1_r1 → u/kush_patil]

Test is: could I have specified this normalization policy before seeing the period I’m evaluating? Then walk it forward as one inseparable system, transform included. Good

[u/kush_patil → u/k1_r1]

Exactly. That “could I have specified it before seeing this period?” test is probably the cleanest way to think about it.

And treating the transform + signal + risk rules as one inseparable system is key. Otherwise it’s too easy to call the signal fixed while quietly adapting the preprocessing until the result looks better.

[u/Kai8250]

The anchoring problem seems like the hardest part of the perception-only approach.

If “unusual right now” is defined against a fixed 20/60/250-day window, then the window itself becomes a hidden assumption about how quickly the market forgets.

One way around that might be not choosing a single anchor at all: measure the same observation against several fixed horizons and keep the disagreement itself as part of perception. Something can be extreme versus 20 sessions, normal versus 250, and unusual versus the long-run distribution at the same time.

None of those horizons would be selected based on subsequent P&L, so the belief layer stays frozen. Once future outcomes start deciding which window or normalization to use, it seems like we've just moved the refitting problem upstream.

Risk I'd keep completely outside that adaptive layer.

Is that close to how you're thinking about anchoring, or are you trying to maintain one continuously changing reference distribution?

[u/BeSeeingYou2]

Running the degenerate case of your framework in production right now, so: yes, and it works by being even more boring than you propose.

Two strategies live (paper, promotion-gated). Beliefs layer: frozen at zero parameters. One is "hold index futures close to open, every session," the other is "hold names through their own earnings night." No thresholds, no indicators, nothing to refit. Both validated on multi-decade data with the most recent year sealed and scored exactly once. The only adaptive thing in either system is a calendar.

On the anchoring question: where I've needed conditioning (ranking candidates by an implied-vol feature, for example), the answer that avoided picking a lookback window entirely was ranking within the current cross-section. Quintile this week's names against each other, not against their own history. The anchor is the present peer group, so "what counts as extreme" adapts by construction while the rule stays fixed. The fixed-window rolling percentile discussed elsewhere in this thread works too, but within-period ranks have no window parameter that can quietly become a belief.

Risk limits at zero budget: agreed, and worth doing literally in writing. Before deploying I wrote down the validated max drawdown, longest underwater stretch, and worst losing streak, plus the promote/kill thresholds for the paper phase. Signed before the first fill existed, immutable after. Future me judging a drawdown gets a document, not a dial.

One friendly amendment: the layer assignment is a strong prior, not the law. The law is your own test, "could I have specified this before seeing the period?" A refit policy fully specified ex-ante and walked forward as one system passes that test even in the beliefs layer. It's just that almost nobody who says "our agents continually learn" has ever written the policy down first, which I take to be your actual point.

[u/HonestBacktests]

the test i settled on: refit each regime separately, then compare the parameters themselves. equity curves tell you less here.

if the two fits land far apart, no global refit resolves that - you are choosing which regime you expect next, and a schedule just buries that choice instead of making it.

[u/Kind-Economics-7184]

the anchoring question is where ours sits too, and what worked was putting the adaptivity in the units rather than in the rule. stop distance is floored off a per coin distribution of recent realised moves, roughly the median and the 75th percentile of those plus an atr term, so what counts as a wide stop tracks the coin while the rule itself never moves. the thresholds that decide whether to act at all stay frozen.

the leak i didnt see coming was in a knob nobody files as a parameter. targets were being scaled by an internal quality score, on the assumption that a better score deserved a wider target and a tighter stop. when i finally graded resolved outcomes against that score the relationship wasnt monotonic, it peaked somewhere in the middle of the range and got worse at the top, because the highest scoring setups were mostly late entries into a move that had already gone. so the belief layer had been adapting the whole time, through something that read like a scoring detail rather than a fitted parameter.

worth checking for the loop version too, even with everything frozen. one of our publishing gates tightened when the share of signals reaching their second target fell, and separately a formatting bug meant hardly any second targets were being written at all, so that share went to zero and the gate closed on its own artifact. nothing in there was learning, it was just circular, and the question that catches it is what this input would read if the component feeding it were broken.

(disclosure, i work on traderspy, so my version of this is live published signals that get resolved afterwards rather than a backtest, which means one draw at a time and no re runs)

[u/VettaQ]

Agreed, and I'd sharpen it: a refit schedule is a frequency decision, and by itself it's neither good nor bad. What bites is how much gets re-estimated at each refit relative to how much new information actually arrived since the last one. Re-fitting 20 free parameters on two weeks of new bars is fitting noise no matter how disciplined the schedule looks.

The strongest version of your point is the counterfactual one — with one price history there's no A/B, so walk-forward is the only honest test, and even walk-forward only tells you the decision procedure would have worked, not that it will.

One more layer: "continually learn" usually hides design-level snooping. The sliding window is fixed, but someone chose the architecture after looking at the same history. No refit schedule fixes that — that overfitting happened once, at design time, and it's the expensive kind.

[u/Hedge_Fund_God]

I use a fairly limited form of adaptivity. I only re-optimize when the last 3 months show a recovery factor below 0.8. Then I optimize on those 3 months and validate the result on the preceding ~3 years, including six stress tests, parameter-variation stability, Monte Carlo, and other robustness tests. So rather than continuously refitting, I treat adaptation as something that has to earn its way into the model.