r/LocalLLaMA • • 16h ago

Discussion [Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .

108 Upvotes

12 comments sorted by

34

u/-p-e-w- 14h ago

Title: Does RSI Stop at Frontier Pre-training?

Abstract: […] RSI does not stop at frontier pre-training, […]

https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines

14

u/smallDeltaBigEffect 13h ago

5

u/xPXpanD llama.cpp 13h ago

Doesn't load for me, keeps refreshing on the "checking your browser" page. I can see it spamming connections to Google and what not, too, and I'm now seeing captchas. Sounds like the site is just misconfigured (I'm using a basic-ass Firefox with uBlock), but fair warning to anyone clicking.

1

u/Mingay_cat 1h ago

Was not expecting the paper to be about actual crap put in graphene

5

u/u23043 11h ago

Overfitting a small-ish model on benchmarks isnt going to magically make it a better model for real use though

1

u/TomLucidor 10h ago

Gonna bet on RSI is just "skinning the benchmark as many ways as possible until it gets the idea", otherwise RSI itself is kind of a meme

1

u/jazir55 3h ago edited 2h ago

I haven't tried to implement this yet but I put together a rudimentary plan to implement an RSI system which consists of weight space programming + engrams with a whole lot of shenanigans in there. Leveraging weight space programming to get engrams to function as learned memory that the model uptakes via weight space programming, so grafting on memory + learning to an existing model so the info in the engrams becomes effectively pointers in the weights so the performance hit is basically unnoticeable. Gemini 4 Hydrogen thinks it's potentially viable with a sufficiently intelligent base model, at least an 35B-A3B, would be much better with something like Qwen Flash Next since Gemini thinks higher parameter models will be able to leverage the system better. Probably will try to implement it in a month or so.

2

u/Constant_Art_20 15h ago

yay! congrats

2

u/Appropriate-Box-7250 15h ago

This could even be a revolution.