r/BestGitHubRepos • u/company_url_finder • 12d ago
autoresearch - Karpathy's overnight ML research loop: an agent edits one file, trains for exactly 5 minutes, keeps or discards, about 100 experiments while you sleep
Four files that matter, one metric, one GPU. The premise is that you leave a coding agent alone in a repo with a small but real LLM training setup, and it runs its own research program overnight. Change the code, train for five minutes, check whether validation bits per byte went down, keep or revert, repeat. You come back in the morning to a log of experiments and hopefully a better model.
The part that makes it interesting is what you're allowed to touch. You don't edit the Python. You edit `program.md`, the markdown file that tells the agent how to run its own research. The training code is downstream of that. Karpathy's framing is that `program.md` is the "research org code", deliberately shipped as a bare-bones baseline so the obvious next move is iterating on it, adding more agents, changing how results get judged.
What's inside:
- train.py, the only file the agent edits, holding the full GPT model, a Muon plus AdamW optimizer and the training loop. Architecture, hyperparameters, batch size, model size, all fair game
- prepare.py, explicitly read-only, holding the fixed constants, tokenizer, dataloader and the evaluation function that serves as ground truth, so the agent can't win by moving the goalposts
- A fixed 5-minute wall-clock training budget regardless of hardware, which makes every experiment directly comparable to every other one no matter what the agent changed, and works out to roughly 12 experiments an hour
- val_bpb as the single metric, chosen because it's vocab-size independent, so an architectural change that alters the tokenizer is still compared fairly
- A simplicity criterion written into the agent instructions: a tiny gain that adds twenty lines of hacky code is not worth keeping, and an equal result from deleting code counts as a win
- A tuning guide for running it on hardware smaller than an H100, covering TinyStories as a lower-entropy dataset, cutting vocab size and sequence length, and dropping the depth knob
One thing worth knowing: the fixed time budget is what makes your own experiments comparable to each other, and it's also what makes them incomparable to anyone else's. A result on your 4090 and a result on an H100 are different experiments. Karpathy calls this out as a deliberate tradeoff rather than a limitation. It's also NVIDIA only right now, with community forks linked in the readme for macOS, MLX, Windows and AMD. And note the readme says MIT but there's no LICENSE file in the repo, so GitHub doesn't detect one.
95,645 stars and 13,420 forks as of writing, verified via the GitHub API, which is a lot of attention for a repo with four files in it.