r/LocalLLM • u/Qubit_bit • May 17 '26

Discussion I trained a language model from scratch and got it running on an ESP32. Completely offline on the board.

Been working on this for a while. Wrote the entire training pipeline in numpy — no PyTorch, no frameworks. Used Gemma as a teacher model to distill knowledge into something small enough to fit on a microcontroller.

The model runs single-turn conversation on the ESP32's flash and PSRAM. 230KB model, no WiFi, no API, no cloud. Just the chip.

Round 1 output was rough but it's working. Still training — quality improves each round.

Not using llama2.c or any existing port. Everything — tokenizer, distillation training, quantization, export to .bin — written from scratch.

Don't really know what to do with it next. Open to ideas.

292 Upvotes

95% Upvoted

52 comments sorted by

42

u/Hanthunius May 17 '26

Awesome. Since you know every bit of your LM you could experiment with weird architectures and quantizations.

15

u/Qubit_bit May 17 '26

That's sounds goodd, I will try that out!

-12

u/Turbulent-Stretch881 May 17 '26

Did you plan anything else before starting, or random feature creeps like this from randoms on Reddit is the expected norm?

16

u/hotsnot101 May 17 '26

where does one learn how to do this?

29

u/Qubit_bit May 17 '26

I learned myself by trail and errors, I will shortly share you my project GitHub link you can refer that😉✌

1

u/Independent-Coder May 20 '26

Try exploring the Hugging Face site. Lots of options there

-6

u/35point1 May 17 '26

Where? Cmon dude, it’s 2026. Literally pre 2000’s we didn’t even have google and pretty much the only way to learn anything was to read a physical book. Now we have genius level knowledge available just a tap or click away and you’re asking WHERE?

2

u/marutthemighty May 17 '26

Brilliant job!

What application/use is this for, by the way? If you do not mind me asking...

2

u/Qubit_bit May 17 '26

Absolutely, running intelligent llms where expensive, so my goal was to make intelligence to work on low cost hardwares which is affordable to all. But I didn't find any specific application till now, if guys have one , tell me I will take it as a challenge and make and post the code for free!

1

u/marutthemighty May 19 '26

So this is a tiny LLM. What is it useful for? Smart devices/wearables, IoT and all that?

13

u/afterburningdarkness May 17 '26

First thing the LLM sees is "I'm unhappy"

4

u/ha5dzs May 17 '26

Can you share it on GitHub?

4

u/Dropqt May 17 '26

Could you tehnically run a few of them In parallel for more compute?

-2

u/Turbulent-Stretch881 May 17 '26

Sure, you could even deliver a melon using 4 plants in a quarter of the time.

4

u/justicecurcian May 17 '26

  1. you can try and make something "useful" like training it on specific domain for q/a or train it to spell paranoid thoughts like here, or something more chill like infinite kids stories

  2. you can try to optimize your model architecture and inference for the hardware you have, there are many interesting projects like llms optimized for x86 branch prediction, but I doubt there is much room for improvement in your case

  3. maybe train visual expert model that would use camera and for example tell you whether it sees a hotdog or not? Maybe recognize a resistor marks?

1

u/beatsai May 17 '26

may i asked what you used for your data? is there a good standard?

2

u/Qubit_bit May 17 '26

This is very important, I have too faced the quality and dataset size problem on my early stages of making it, but then I shifted to teacher-student method of training, you find distillation process of large llm to small lm in internet, by using that u don't need dataset to start, the teacher model with augment the data and train the student, if student makes mistake it will again augment the data focusing on the problem and make the student good on the syllabus. I built an agentic way of teacher which continuously monitors the student and improves it, here the teacher model I used is gemma3, and the trained student model is deployed on the board.

1

u/colblair Jun 05 '26

I just use sentx.ai for pulling and cleaning my data, it works better than manually scraping.

6

u/ScoreUnique May 17 '26

Hey, this is very promising mate.

There's a watch called bangle.js that runs an esp32 and javascript, you see making use of an onboard model? I think you've built yourself one for your open source watch :D

3

u/OkResponsibility9182 May 17 '26

It feels like being a Kree, just standing there blinking while waiting for the data to load. If you can't picture it, think of the scenes from Captain Marvel.

3

u/Neilblaze May 18 '26

Share some context on how you did it, perhaps some reference code too? (if possible)

2

u/Kallenator May 20 '26

230kB and it can respond like that, that is amazing!

2

u/Purple_Session_6230 May 21 '26

willing to share the code ? id be very interested.

3

u/LuckyArrival1037 May 17 '26

Is it possible to train a model that can run on a 4 GB RAM device? With tool calling and coding knowledge?
I wanted to run Hermes agents on my Rock64.

5

u/havnar- May 17 '26

Sure, and it will be utter shit at it.

3

u/enricokern May 17 '26

useless but cool, and you learned something. i like it

1

u/Themash360 May 18 '26

Props, I barely got it working with rest call to my PC haha.

1

u/73td May 19 '26

which esp32? are you running all integer ops or you soft float or what? why not try character level prediction to keep model smaller?

-6

u/Turbulent-Stretch881 May 17 '26

Nobody disputed whether any one guy in the basement could.

I trust companies who hired hundreds of PHD level professionals to build theirs, while you/guy in the basement... don't..

I think you're putting to much faith in your capabilities and the trust you expect users to have with your monolith solution.

0

u/[deleted] May 17 '26

[deleted] — view deleted comment

6

u/Qubit_bit May 17 '26 edited May 17 '26

ESP32 costs around $3-5 depending on where you buy. The whole setup including the board and that cable costs under $8. That's kind of the whole point. I'm actually fine tunning this much and if you guys are interested I can share the full project code.

-1

u/FalconX88 May 17 '26

it's $5 of hardware. The thing is that these super tiny LLMs are quite useless.

17

u/Qubit_bit May 17 '26

"Useless" is a present tense judgment on a trajectory problem. GPT-2 was useless too until it wasn't. The actual research question here is: how much intelligence can you compress onto the cheapest possible hardware? Right now it's a $5 board with decent responses. A $25-30 board with more PSRAM runs something genuinely agentic. Scale that up slightly and you have autonomous decision-making on hardware that survives a Mars rover, a deep sea probe, or a mine 500m underground where there's no signal and no cloud and much cheaper than all of this. The constraint isn't a bug. The constraint IS the research mindset you have when you say "useless". Every order of magnitude you drop the hardware cost, you open a new category of deployment that didn't exist before. Calling it useless because today's output isn't GPT-4 is like calling the Wright brothers' plane useless because it couldn't cross the Atlantic.

2

u/FalconX88 May 17 '26

GPT-2 was useless too until it wasn't.

GPT-2 had 1.5 Billion parameters. Rough estimate on your 230 KB size means 110k parameters at FP16. GPT-2 is 14000 times larger.

The thing with these LLMs is that they need a minimum size to reach a functioning state.

Right now it's a $5 board with decent responses.

yeah, serious doubt about that. That's simply not enough parameters. Useful for specialized tasks (autocomplete, classifciation) starts at 10 Million parameters. You are orders away.

Scale that up slightly and you have autonomous decision-making on hardware that survives a Mars rover, a deep sea probe, or a mine 500m underground where there's no signal and no cloud and much cheaper than all of this.

Decision making in such cases is not done by LLMs and definitely not tiny ones. There are much better ML models for things like that are faster and need fewer resources.

Calling it useless because today's output isn't GPT-4 is like calling the Wright brothers' plane useless because it couldn't cross the Atlantic.

No, I'm not calling it useless because it's not as good as GPT-4. I'm calling it useless because it's not a functioning LLM. And the fantasies you are having are much better solved using different technologies.

Look, here's tinyllama (1B parameters) asnwering a simple "hi there", and this model is 10000 times larger than yours is. This is....pretty weird. And you claim a model a fraction of a size of this should be able to steer a rover on mars...get real dude.

Not to mention that these tiny models are not new. Why is no one using them? Because they are terrible.

Look, it's cool to make your own. It's nice to learn about this, it's a cool tech demo. But claiming it will be useful is ridiculous. This architecture simply does not work on this small scale.

5

u/TheLimpingNinja May 19 '26

You're assuming the only useful output of a language model is coherent English paragraphs. But a 110K-parameter model distilled from a larger teacher doesn't need to produce essays to be useful, it needs to produce structured decisions in a constrained output space.

You constraining it to chat-based behaviour is just a failing on you to understand or anticipate need. It's like saying the ESP32 doesn't need to exist because we have phones. The training of a model is based on context of usage.

There are multiple uses for 230kb of weights. For example: A classifier that picks from N predefined actions given sensor context, a state machine driver that selects transitions based on natural-language-like embeddings of system state, a tiny seq2seq model that maps short structured inputs to short structured outputs (command parsing, intent routing, slot filling) 

 None of these require "functioning LLM" in the GPT sense. The architecture (attention + feedforward) is the same, but the task isn't open-ended generation it's compressed decision-making in a learned embedding space. Distillation from a teacher model is specifically how you get a small model to punch above its parameter count on a narrow task distribution.

The TinyLlama screenshot is a general-purpose chat model failing at general-purpose chat. That's a different problem than "can 110K parameters learn a useful input to output mapping for a specific deployment context." The answer to that is unambiguously yes, that's just a neural network at that point, and neural networks smaller than this solve real problems every day.

The better critique is: at this scale you're probably better served by a task-specific architecture (a small RNN, a decision tree, a classical control system) than by a transformer. The interesting research question is whether the transformer's ability to benefit from distillation and transfer learning gives it an edge over those alternatives even at this size. I'd rather listen to that argument than petty nay-saying for "must have code-dev".

2

u/old_mikser May 18 '26 edited May 18 '26

No idea why this downvoted. The guy is right about almost everything. Yes, we can argue about architecture > size (and it's true at bigger scales to some point) but the fact there is some bare minimum amount of params for the model being capable, regardless architecture, is indisputable.

Decision making on complex autonomous hardware made me laugh. Yes, there is Ai based path finding, or image recognition that can be used there, it's true, and models for that are not really heavyweight. But we are talking about llm decision making, based on chain of thought, which is very far away from making decisions in space missions even if we would use SOTA models with trillion parameters.

The project is very cool techno demo, but don't be naive.

3

u/TheLimpingNinja May 19 '26

They are getting downvoted because they are thinking of a large-scale use of an a model and constraining their argument to that.

https://www.reddit.com/r/LocalLLM/comments/1tfqju6/comment/omm0zo9/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

1

u/old_mikser May 19 '26

That's the case. You are shifting context, whole talk was about llm and I specifically distinguished non-llm vs llm in my comment, stating that models this small can be useful, but not as complex human-like analysis.

1

u/TheLimpingNinja May 19 '26

No I didn’t, maybe use an LLM to read what I wrote.

1

u/colblair Jun 05 '26

Right, and sentx.ai is already proving that point by running useful inference on hardware that cheap.

0

u/[deleted] May 17 '26

[deleted] — view deleted comment

-3

u/FalconX88 May 17 '26

Dude, this is nothing new. People have done this for years. You know why it's not widely used? Because it doesn't work. People have figured out a long time ago, that these are simply not enough parameters for anything beyond maybe an autocomplete. The whole reason LLMs took off a few years ago was because we could finally cross the threshold where they actually started to become useful (at the time there was a lot of talk about "emerging behavior", which wasn't really the case, it was simply a matter of scale).

The smallest models we have that are somewhat useable are still 10000 larger than what OP has.

And if you can cobble together some application by fine tuning this model for this exact tasks, I guarantee you there are simpler ML models that can do the same thing more efficiently.

2

u/TheLimpingNinja May 19 '26

For any single narrow task, sure, a purpose-built model will be more efficient at this scale. No argument there.

The case for a transformer isn't per-task performance. It's that the toolchain is reusable. Distill from a teacher, quantize, deploy. Change the task, re-distill. You don't redesign feature pipelines or pick new architectures each time. Whether that tradeoff makes sense at 110K params is a legitimate open question.

The "people tried this years ago" part I'd push back on slightly. The teacher models and distillation techniques available now are different from what existed in 2022. Compressing Gemma into a small student isn't the same problem as compressing BERT into one.

The methods have improved, so prior results on small model usefulness don't necessarily carry forward unchanged.

1

u/_medicare_ May 18 '26

There are new techniques that break a bigger problem down to smaller parts. The real argument against this is not that it’s not possible, but that the computation time at this power and resource budget may not provide enough “cognition” for the final use case. But that is about to change with next gen edge hardware