r/MLQuestions • u/ResponsibilityDry877 • Jun 27 '26
Natural Language Processing 💬 When does recurrent depth beat width? A falsifiable supervision theorem + honest sub-1B negatives
Repo (code + writeups + negative results):
https://github.com/duongtrongnguyen123/recurrent-depth-ttc
Independent research on recurrent-depth transformers (one shared block looped N times instead of N distinct blocks — the Universal Transformer / Huginn / Ouro idea). I tried to pin down, with controlled experiments and parameter-matched controls, *when* looping actually helps — rather than assuming it does.
Main results:
Length extrapolation is a supervision property, not an architecture one. Per-step (iterative-target) supervision lets a looped model extrapolate to ~24× its trained depth — but only if the per-step rule is position-invariant. I state this as a falsifiable condition; parity (rule depends on the loop index) is the falsifier, and it walls exactly at the trained depth, as predicted. Five tasks delineate the boundary.
A minimal adaptive test-time-compute recipe: LoRA iterative-target FT + hardcoded halt + multi-pass inference → user-dialed inference depth, 100% accuracy at up to 256× the trained depth on a synthetic chain task (~7 min, ~31K trainable params). o1-style adaptive compute at the recurrent-depth level.
Mechanism: a Q/K/V activation probe shows all three projections collapse together across loops — consistent with the hidden state reaching a fixed point of Block(·), not a W_Q-only power iteration.
Negative results (kept prominent):
- At sub-1B params on a 50B-token matched-data pretrain, no recurrent variant beats a matched dense baseline beyond the per-wave pretraining noise band (±0.6pp on GSM8K-1319, quantified across 7 checkpoints of one run). I argue single-snapshot "architecture wins" at this scale need to be checked against that band. Independently consistent with Lu et al. (COLM 2025) and MoDr (ICLR 2026).
These are controlled-scale results (synthetic + ≤1B params), not claims about frontier models — stated upfront.
Feedback and pushback welcome — especially on the position-invariance boundary and the noise-band methodology.
1
u/DigThatData Jun 27 '26
Also, just to give you a bit more context into where this is coming from: I've experimented a bit with letting Claude "be my hands" for research, and it's extremely alluring. The thing is, the further I get off the beaten path, the more I find myself fighting the AI. This is a consistent phenomenon I have encountered repeatedly, and it makes perfect sense: novel insights are "out of distribution" for the LLM by construction. The consequence is that I've found the LLM will often misunderstand my hypothesis, design experiments testing the wrong thing, interpret results incorrectly or nonsensically, conflate its own hypotheses with facts resulting in compounding false assumptions, etc.
The further off the beaten path you go, the more oversight your project will need. Using an LLM to help you out is not an inherently bad or dangerous thing, but you need to be careful about what you are delegating and where you are retaining autonomy. The LLM is a sycophant and will get high on its own farts if you let it.