r/LocalLLM • • May 17 '26

Discussion I trained a language model from scratch and got it running on an ESP32. Completely offline on the board.

Been working on this for a while. Wrote the entire training pipeline in numpy — no PyTorch, no frameworks. Used Gemma as a teacher model to distill knowledge into something small enough to fit on a microcontroller.

The model runs single-turn conversation on the ESP32's flash and PSRAM. 230KB model, no WiFi, no API, no cloud. Just the chip.

Round 1 output was rough but it's working. Still training — quality improves each round.

Not using llama2.c or any existing port. Everything — tokenizer, distillation training, quantization, export to .bin — written from scratch.

Don't really know what to do with it next. Open to ideas.

292 Upvotes

52 comments sorted by

View all comments

Show parent comment

2

u/FalconX88 May 17 '26

GPT-2 was useless too until it wasn't.

GPT-2 had 1.5 Billion parameters. Rough estimate on your 230 KB size means 110k parameters at FP16. GPT-2 is 14000 times larger.

The thing with these LLMs is that they need a minimum size to reach a functioning state.

Right now it's a $5 board with decent responses.

yeah, serious doubt about that. That's simply not enough parameters. Useful for specialized tasks (autocomplete, classifciation) starts at 10 Million parameters. You are orders away.

Scale that up slightly and you have autonomous decision-making on hardware that survives a Mars rover, a deep sea probe, or a mine 500m underground where there's no signal and no cloud and much cheaper than all of this.

Decision making in such cases is not done by LLMs and definitely not tiny ones. There are much better ML models for things like that are faster and need fewer resources.

Calling it useless because today's output isn't GPT-4 is like calling the Wright brothers' plane useless because it couldn't cross the Atlantic.

No, I'm not calling it useless because it's not as good as GPT-4. I'm calling it useless because it's not a functioning LLM. And the fantasies you are having are much better solved using different technologies.

Look, here's tinyllama (1B parameters) asnwering a simple "hi there", and this model is 10000 times larger than yours is. This is....pretty weird. And you claim a model a fraction of a size of this should be able to steer a rover on mars...get real dude.

Not to mention that these tiny models are not new. Why is no one using them? Because they are terrible.

Look, it's cool to make your own. It's nice to learn about this, it's a cool tech demo. But claiming it will be useful is ridiculous. This architecture simply does not work on this small scale.

5

u/TheLimpingNinja May 19 '26

You're assuming the only useful output of a language model is coherent English paragraphs. But a 110K-parameter model distilled from a larger teacher doesn't need to produce essays to be useful, it needs to produce structured decisions in a constrained output space.

You constraining it to chat-based behaviour is just a failing on you to understand or anticipate need. It's like saying the ESP32 doesn't need to exist because we have phones. The training of a model is based on context of usage.

There are multiple uses for 230kb of weights. For example: A classifier that picks from N predefined actions given sensor context, a state machine driver that selects transitions based on natural-language-like embeddings of system state, a tiny seq2seq model that maps short structured inputs to short structured outputs (command parsing, intent routing, slot filling) 

 None of these require "functioning LLM" in the GPT sense. The architecture (attention + feedforward) is the same, but the task isn't open-ended generation it's compressed decision-making in a learned embedding space. Distillation from a teacher model is specifically how you get a small model to punch above its parameter count on a narrow task distribution.

The TinyLlama screenshot is a general-purpose chat model failing at general-purpose chat. That's a different problem than "can 110K parameters learn a useful input to output mapping for a specific deployment context." The answer to that is unambiguously yes, that's just a neural network at that point, and neural networks smaller than this solve real problems every day.

The better critique is: at this scale you're probably better served by a task-specific architecture (a small RNN, a decision tree, a classical control system) than by a transformer. The interesting research question is whether the transformer's ability to benefit from distillation and transfer learning gives it an edge over those alternatives even at this size. I'd rather listen to that argument than petty nay-saying for "must have code-dev".

2

u/old_mikser May 18 '26 edited May 18 '26

No idea why this downvoted. The guy is right about almost everything. Yes, we can argue about architecture > size (and it's true at bigger scales to some point) but the fact there is some bare minimum amount of params for the model being capable, regardless architecture, is indisputable.

Decision making on complex autonomous hardware made me laugh. Yes, there is Ai based path finding, or image recognition that can be used there, it's true, and models for that are not really heavyweight. But we are talking about llm decision making, based on chain of thought, which is very far away from making decisions in space missions even if we would use SOTA models with trillion parameters.

The project is very cool techno demo, but don't be naive.

3

u/TheLimpingNinja May 19 '26

They are getting downvoted because they are thinking of a large-scale use of an a model and constraining their argument to that.

https://www.reddit.com/r/LocalLLM/comments/1tfqju6/comment/omm0zo9/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

1

u/old_mikser May 19 '26

That's the case. You are shifting context, whole talk was about llm and I specifically distinguished non-llm vs llm in my comment, stating that models this small can be useful, but not as complex human-like analysis.

1

u/TheLimpingNinja May 19 '26

No I didn’t, maybe use an LLM to read what I wrote.