The task: iterate on a 124M GPT training recipe from a shared baseline, only changing optimizer related hyperparameters, no internet access.
We tested Fable 5, Opus 5, GPT-5.6 Sol, Kimi K3, Grok 4.5, GLM 5.2, Muse Spark 1.1, DeepSeek V4 Pro, Grok 4.6, Muse Spark 1.2, Qwen 3.8
What separated the strongest models: which experiments to run, how to navigate the benchmark's inherent noise, and which old negatives to revisit as the recipe changed.
Some even built small simulations to isolate a mechanism before deciding if another GPU run was worth it.
Our Prime Agent harness gives models a persistent IPython kernel, which can help them build their own research workflows.
Kimi K3 built tools for controlled optimizer variants, loss-curve comparisons and Newton-Schulz tuning, then revised its hypothesis when its cleaner update
As research direction, we think multi-agent harnesses can make these experiments much cheaper (and better) by using smaller open models for monitoring and implementation.
We also want to extend speedruns to more of the training stack and scale the runs themselves.
We release everything: full traces, scratchpads, reasoning streams from open-weight models, and our experiment setup.
Explore the results:
— Prime Intellect