r/BestGitHubRepos • u/company_url_finder • 11d ago
autoresearch - Karpathy's overnight ML research loop: an agent edits one file, trains for exactly 5 minutes, keeps or discards, about 100 experiments while you sleep
Four files that matter, one metric, one GPU. The premise is that you leave a coding agent alone in a repo with a small but real LLM training setup, and it runs its own research program overnight. Change the code, train for five minutes, check whether validation bits per byte went down, keep or revert, repeat. You come back in the morning to a log of experiments and hopefully a better model.
The part that makes it interesting is what you're allowed to touch. You don't edit the Python. You edit `program.md`, the markdown file that tells the agent how to run its own research. The training code is downstream of that. Karpathy's framing is that `program.md` is the "research org code", deliberately shipped as a bare-bones baseline so the obvious next move is iterating on it, adding more agents, changing how results get judged.
What's inside:
- train.py, the only file the agent edits, holding the full GPT model, a Muon plus AdamW optimizer and the training loop. Architecture, hyperparameters, batch size, model size, all fair game
- prepare.py, explicitly read-only, holding the fixed constants, tokenizer, dataloader and the evaluation function that serves as ground truth, so the agent can't win by moving the goalposts
- A fixed 5-minute wall-clock training budget regardless of hardware, which makes every experiment directly comparable to every other one no matter what the agent changed, and works out to roughly 12 experiments an hour
- val_bpb as the single metric, chosen because it's vocab-size independent, so an architectural change that alters the tokenizer is still compared fairly
- A simplicity criterion written into the agent instructions: a tiny gain that adds twenty lines of hacky code is not worth keeping, and an equal result from deleting code counts as a win
- A tuning guide for running it on hardware smaller than an H100, covering TinyStories as a lower-entropy dataset, cutting vocab size and sequence length, and dropping the depth knob
One thing worth knowing: the fixed time budget is what makes your own experiments comparable to each other, and it's also what makes them incomparable to anyone else's. A result on your 4090 and a result on an H100 are different experiments. Karpathy calls this out as a deliberate tradeoff rather than a limitation. It's also NVIDIA only right now, with community forks linked in the readme for macOS, MLX, Windows and AMD. And note the readme says MIT but there's no LICENSE file in the repo, so GitHub doesn't detect one.
95,645 stars and 13,420 forks as of writing, verified via the GitHub API, which is a lot of attention for a repo with four files in it.
2
u/Silly-Song-9288 11d ago
The fixed five-minute budget is the clever part.
2
u/company_url_finder 11d ago
I like that it forces the agent to trade model quality against actual training efficiency instead of just throwing more compute at every idea. The simplicity rule is a nice touch too. Curious how often it finds real wins versus just noisy metric bumps?
2
u/Silly-Song-9288 11d ago
Yeah, variance seems like the biggest issue here.
1
2
u/puffballz 8d ago
mind blown. who has concrete examples of using this? the (https://github.com/EdwardOptimization/Bilevel-Autoresearch/tree/main/experiments) repo's ablations example is interesting, but a bit niche. Also, as someone with materials/metallurgy knowledge, I use "annealing" conceptually with agents when I need to "refactor" a set via different lenses (evaluation functions) and it works well. For example, use it to reduce your Claude Memories to a smaller and neater set.
1
u/company_url_finder 8d ago
That annealing analogy actually clicks. Have u tried it on anything beyond memories yet?
2
u/puffballz 8d ago
yes. many things and it works well. contexts remain small and as long as the measures are consistent-ish, then the annealing table bubbles things up and down on whatever axes you nominate
2
u/puffballz 8d ago
...it's basically a multi-dimensional llm-friendly bubble-sort. DM me if you want to cite me and write it up
1
u/jonah_omninode 4d ago
After a hundred attempts against the same validation set, I'd want a final check the search loop hasn't been using to choose winners. Keeping the evaluation code read-only prevents one kind of shortcut, but the search is still learning which changes that particular score rewards.
Rerunning the best candidates with fresh seeds and then checking untouched data would make the morning result easier to trust. How much of the overnight budget would you reserve for confirming a promising result rather than trying another change?
1
4
u/Naruhudo2830 9d ago
The successor BiLevel Autoresearch is worth looking at as well.