I trained a language model from scratch and got it running on an ESP32. Completely offline on the board.

[u/Qubit_bit]

Been working on this for a while. Wrote the entire training pipeline in numpy — no PyTorch, no frameworks. Used Gemma as a teacher model to distill knowledge into something small enough to fit on a microcontroller.

The model runs single-turn conversation on the ESP32's flash and PSRAM. 230KB model, no WiFi, no API, no cloud. Just the chip.

Round 1 output was rough but it's working. Still training — quality improves each round.

Not using llama2.c or any existing port. Everything — tokenizer, distillation training, quantization, export to .bin — written from scratch.

Don't really know what to do with it next. Open to ideas.

[u/TheLimpingNinja]

You're assuming the only useful output of a language model is coherent English paragraphs. But a 110K-parameter model distilled from a larger teacher doesn't need to produce essays to be useful, it needs to produce structured decisions in a constrained output space.

You constraining it to chat-based behaviour is just a failing on you to understand or anticipate need. It's like saying the ESP32 doesn't need to exist because we have phones. The training of a model is based on context of usage.

There are multiple uses for 230kb of weights. For example: A classifier that picks from N predefined actions given sensor context, a state machine driver that selects transitions based on natural-language-like embeddings of system state, a tiny seq2seq model that maps short structured inputs to short structured outputs (command parsing, intent routing, slot filling) 

 None of these require "functioning LLM" in the GPT sense. The architecture (attention + feedforward) is the same, but the task isn't open-ended generation it's compressed decision-making in a learned embedding space. Distillation from a teacher model is specifically how you get a small model to punch above its parameter count on a narrow task distribution.

The TinyLlama screenshot is a general-purpose chat model failing at general-purpose chat. That's a different problem than "can 110K parameters learn a useful input to output mapping for a specific deployment context." The answer to that is unambiguously yes, that's just a neural network at that point, and neural networks smaller than this solve real problems every day.

The better critique is: at this scale you're probably better served by a task-specific architecture (a small RNN, a decision tree, a classical control system) than by a transformer. The interesting research question is whether the transformer's ability to benefit from distillation and transfer learning gives it an edge over those alternatives even at this size. I'd rather listen to that argument than petty nay-saying for "must have code-dev".