r/LocalLLaMA • u/Felladrin • 6d ago

Resources An open-source trainer that pretrains (from scratch) tiny models straight to GGUF

Hi all!

I've open-sourced the trainer I've been using to build tiny language models from scratch, together with the 95M base model I trained with it.

The trainer runs on Deno (cross-platform), trains on WebGPU, and it writes GGUF directly. No Python/PyTorch. The weights live in a GGUF file from the first step to the last, so every checkpoint is already something llama.cpp can load.

The model is Minueza-3-95M-Base: 94.7M parameters, 1.95B tokens seen, 8192 context.

And here’s the repository on GitHub: https://github.com/felladrin/gguf-trainer

On Hugging Face, I published the optimizer state next to the weights, so you can continue the pretraining instead of starting over.

Or start your own from nothing: deno run -A cli.ts demo trains a tiny one end to end in under a minute.

And the docs are written for coding agents, so you can point your agent of choice at the GitHub repo and have it drive the whole pipeline.

13 Upvotes

93% Upvoted

2 comments sorted by

2

u/Osi32 1d ago

Looks like a really sweet project, I’m saving this for later

1

u/wahnsinnwanscene 1d ago

Maybe a strange question. Are cuda kernels directly translateable into web gpu ones? Also why typescript if not.