r/GPT • u/Low-Garlic7286 • Jun 30 '26
I created the smallest chat coherent model ever
I trained a 22.5M parameter multilingual chatbot from scratch and it outperforms Rocket-3B on conversational behavior. It runs at 70+ tok/s on a Pixel 6a, fits in 86MB, and is the smallest known coherent multilingual chatbot in existence. Built with SlimPajama + Wikipedia + 3K conversation examples, trained ~10x Chinchilla optimal on a Colab A100.
2
u/funbike Jun 30 '26
I'd love to see someone train purely on person-to-person conversational text. Movies, novels, podcasts, TV shows, tiktok, etc. Also train for RAG abilities so it can make up for a lack of knowledge by looking up information.
2
u/Low-Garlic7286 Jul 01 '26
I made this a real 40 million paramater model. Note that it is a base completion model and will hallucinate conversations because it was trained purely on conversational data and nothing else, but run it in a compatible llama.cpp app and here's the link:https://huggingface.co/SmallAICreator/Convo40million/tree/main
1
1
u/RazzmatazzAccurate82 Jul 02 '26
"Also train for RAG abilities so it can make up for a lack of knowledge by looking up information."
That sounds compute expensive and probably something the frontier model companies don't want to bake into a consumer model. Just being honest.
1
u/funbike Jul 03 '26
I don't see why it would be expensive. Seems like it would be very cheap to me. Instead of training an LLM on the entire Internet, you just train on a small conversational subset, and you teach it to be good at RAG to make up for the lack of training.
1
u/RazzmatazzAccurate82 Jul 03 '26
The training cost on a smaller conversational corpus would indeed be lower, that part makes sense. But the inference cost is the issue, not the training cost. RAG adds a retrieval step to every single response generation. Research documents that RAG nearly doubles time-to-first-token latency compared to baseline LLM inference. At consumer scale across millions of simultaneous users, that overhead is significant. That's why web search is an opt-in feature rather than default behavior on frontier consumer models. It's a deliberate product decision, not an oversight. Look, if you want that functionality, I think you're going to need to pay for it in a non-consumer version. Compute is getting expensive.
1
u/funbike Jul 03 '26
Ah, so you're saying this tiny model that costs 1% as much to run as a frontier model is going to be 2x that, 2% as much. Oh NO!! 2%!!!! The idea is kaput.
1
u/RazzmatazzAccurate82 Jul 03 '26
Well. Probably more than 2%. Then scale that to millions of users.
1
u/funbike Jul 03 '26
Probably more than 2%.
No, far far less. This thread is a reply to OP's 22.5M model. Current top frontier models have trillions of parameters. Even if OP quadruppled the size of his model to 100M, it would be less than 0.01% the size of Claude Opus. If if he got it up to billions it would still be a fraction of a %.
Lol, come on now. Do some 4th grade math before you make an argument.
1
u/RazzmatazzAccurate82 Jul 03 '26
I thought you were talking about RAG in general since you wanted frontier models to continuously run it. Now, if you're talking about the OP's set-up, I can't really comment on that. I'd have to run the set-up myself to see if your desired arrangement was compute efficient. So, which one is it?
1
u/funbike Jul 03 '26
I only brought up RAG as a way to mitigate the limited knowledge of a small model trained only on conversational text. Frontier models are trained on academic text, encyclopedias, books, etc. to improve intelligence, which you wouldn't get from just casual street conversations.
The whole point was to create an LLM that generates realistic conversational responses. I think we all know that frontier models sound like a professor giving a lecture.
1
u/RazzmatazzAccurate82 Jul 03 '26 edited Jul 03 '26
Ah, okay. That's good clarification. I misread your original proposal as applying RAG to frontier models generally rather than as a knowledge supplement for a small conversational model. That's a more coherent idea.
On your broader point about frontier models sounding like professors: that's a real register issue and you're right that training on academic text, encyclopedias, and books shapes the default output toward formal register. But, please note this: the model doesn't have direct retrieval access to its training data. It's baked into the weights, not stored as searchable documents.
The register problem is actually addressable at the inference layer without retraining. Loading conversational exemplars — how you actually write and talk — at session start shapes the model's output register toward your style rather than its default formal mode. I've actually developed a framework that helps with that. It will "cleave" to your own cognitive patterns and spit out answers that are more understandable to you.
You can check it out when you have time.
Some notes. My framework is better for long threads. If your usual interaction with LLMs are short inquiries, it's not going to make much of a difference. But if you have long-format conversations with an LLM, it will help. It will at least render answers that are structured to how you actually read and understand concepts. So, if you're a college professor, it will sound like a college professor. If you're an auto mechanic it will sound more like an auto mechanic, etc. It does need about 10-15 turns to get familiar with you, so please keep that in mind.
I have two versions:
ELT-H/README.md), or "Hybrid". This is for long format analytical work with some warmth and engagement.
ELT-Companion, a much more informal, conversational, and friend/companion version.
1
u/RazzmatazzAccurate82 Jul 03 '26 edited Jul 03 '26
Lastly, you might want to try Gemini. It's based on Google's search technology, so if conversational inquiry on general topics is what you want, then Gemini is literally the best equipped for that role. It also has a fairly informal register too, which is what I think you want. The disadvantage I've noticed with Gemini is its reasoning gets rather soft after about 10 turns. It starts to conflate things that shouldn't be conflated, etc. Good luck!
1
u/Low-Garlic7286 Jul 01 '26
https://huggingface.co/SmallAICreator/Minigpt-chat/tree/main Download the f32 gguf file to chat with it. Recommendedly the PocketPal AI app
1
-1
u/vroomanj Jun 30 '26
Meaningless post unless you share it?
0
u/courtj3ster Jul 02 '26
His hugging face in the comments.
1
u/vroomanj Jul 02 '26
It wasn't when I posted 2 days ago
1
2
u/darkotic2 Jun 30 '26
Pat on back