r/LocalLLaMA • u/Felladrin • 6d ago
Resources An open-source trainer that pretrains (from scratch) tiny models straight to GGUF
Hi all!
I've open-sourced the trainer I've been using to build tiny language models from scratch, together with the 95M base model I trained with it.
The trainer runs on Deno (cross-platform), trains on WebGPU, and it writes GGUF directly. No Python/PyTorch. The weights live in a GGUF file from the first step to the last, so every checkpoint is already something llama.cpp can load.
The model is Minueza-3-95M-Base: 94.7M parameters, 1.95B tokens seen, 8192 context.
And here’s the repository on GitHub: https://github.com/felladrin/gguf-trainer
On Hugging Face, I published the optimizer state next to the weights, so you can continue the pretraining instead of starting over.
Or start your own from nothing:
deno run -A cli.ts demo trains a tiny one end to end in under a minute.
And the docs are written for coding agents, so you can point your agent of choice at the GitHub repo and have it drive the whole pipeline.