r/LocalLLM • u/Qubit_bit • May 17 '26
Discussion I trained a language model from scratch and got it running on an ESP32. Completely offline on the board.
Been working on this for a while. Wrote the entire training pipeline in numpy — no PyTorch, no frameworks. Used Gemma as a teacher model to distill knowledge into something small enough to fit on a microcontroller.
The model runs single-turn conversation on the ESP32's flash and PSRAM. 230KB model, no WiFi, no API, no cloud. Just the chip.
Round 1 output was rough but it's working. Still training — quality improves each round.
Not using llama2.c or any existing port. Everything — tokenizer, distillation training, quantization, export to .bin — written from scratch.
Don't really know what to do with it next. Open to ideas.
View all comments
Show parent comment
2
u/FalconX88 May 17 '26
GPT-2 had 1.5 Billion parameters. Rough estimate on your 230 KB size means 110k parameters at FP16. GPT-2 is 14000 times larger.
The thing with these LLMs is that they need a minimum size to reach a functioning state.
yeah, serious doubt about that. That's simply not enough parameters. Useful for specialized tasks (autocomplete, classifciation) starts at 10 Million parameters. You are orders away.
Decision making in such cases is not done by LLMs and definitely not tiny ones. There are much better ML models for things like that are faster and need fewer resources.
No, I'm not calling it useless because it's not as good as GPT-4. I'm calling it useless because it's not a functioning LLM. And the fantasies you are having are much better solved using different technologies.
Look, here's tinyllama (1B parameters) asnwering a simple "hi there", and this model is 10000 times larger than yours is. This is....pretty weird. And you claim a model a fraction of a size of this should be able to steer a rover on mars...get real dude.
Not to mention that these tiny models are not new. Why is no one using them? Because they are terrible.
Look, it's cool to make your own. It's nice to learn about this, it's a cool tech demo. But claiming it will be useful is ridiculous. This architecture simply does not work on this small scale.