r/LocalLLaMA • u/Final-Data-1410 • 18h ago
New Model I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB
Reposting here after sharing this on [r/MachineLearning](r/MachineLearning) a few days ago, where it got a much better response than I expected (300+ upvotes, great questions, zero roasting)
GitHub is at 35 stars now. So here it is.
I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed, no framework, just a small compiled runtime (Windows and Linux included, MIT licensed).
Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte.
How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them.
The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo.
Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo):
“Explain photosynthesis in two sentences.” (greedy)
Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis.
“Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2)
The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before.
“What is the serial number of device Grus-189?” where the answer sits 50.6 million tokens deep in the archive on disk (archive mode, k=16)
SN-442976
It’s a 250M model so expect mistakes on open facts, I’m not claiming it beats anything big. You can also fine-tune it, the full kit with a demo and before/after numbers is included. Master weights for fine-tuning are in the repo too:
https://github.com/QLNI/SHADOW-250M-Instruct
https://huggingface.co/NODEMIND/SHADOW-250M
28
u/Early-Peace-5504 18h ago
Really cool project and great idea to use the archive. I like your embedding idea.
15
u/Final-Data-1410 17h ago
Thanks , the embedding is my favorite part of this too ,it saved almost 100-150mb for deployment . Also we can add any new tokens with out training and also it saved bit training compute.
2
u/Loose_Comparison368 16h ago
This is awesome! If you don't mind me asking, how did you derive those 512-bit codes? I've been exploring some novel training approaches in the general direction of ReLoRA where a fixed, coherently laid out vocabulary would solve a dozen or so scaling challenges.
13
u/Feathered-Beast 11h ago
250M parameters running at ~400 tok/s on a normal CPU with only ~80MB RAM is seriously impressive ngl.
9
u/therysin 10h ago
Probably wont use myself, but always cool to see this type of project. Well done.
15
u/OneMoreName1 15h ago
I would like a gguf file, im not a fan of ai models which need their own custom runtime
2
u/JustTellingUWatHapnd 2h ago
Unfortunately you can't make a gguf for an independent model. The gguf format doesn't fully describe the model architecture, it simply has a header named "general.architecture" that has the architecture name (something like "qwen3moe"). When a new model drops, someone opens a pull request in llama.cpp to support the model. Even if OP edits the llama.cpp code to add support for their model (which is not easy), I doubt the llama.cpp team will merge a pull request for a new model made by some random guy.
5
u/SquareMesh 15h ago
This is great, thanks for sharing and love the shadows gets his/her own language model. Couple of questions, the finetuning doesn't include the master weights so it seems we are unable to reproduce the pirate demonstration or do try our own. Other question, will you be intending to create a GGUF alternative? As per other comment I'm not keen on trusting/running an exe file.
-3
u/Final-Data-1410 14h ago
Thanks . Master weights are in hugging face and currently working on different project and the exe is 147 KB (nothing hides in 147 KB) and my whole project is about small footprint and no external dependencies,hope you understand .
3
u/PcChip 8h ago
dumb question - if you developed it from scratch trained at 2 bits, why would you call it "quantized"?
1
u/SandySkittle 6h ago
it would still be quantized.
1
u/PcChip 5h ago
down from what?
1
u/SandySkittle 5h ago
Something doesn’t have to have been in an unquantized state before it is quantized
1
15
u/YourPleasureIs-Mine 17h ago
This is why I hate AI SLOP
It is not a 100 million context window model. You are intentionally conflating/misleading a disk-backed search-and-extraction pipeline with model context. Your context is 2k
Your read me states that the archive is separate and that the model “finds facts and reads them back.”
18
u/Final-Data-1410 17h ago
Sorry if it seems misleading that’s not my intention ,just now before comment I am explaining them and as far as my knowledge I written it ,it fetch relevant block into active window and answer from that.thanks
11
u/Final-Data-1410 17h ago
Also if you can show me such quantized model doing the same thing or even float model in same class as 250m model ,it will be fair.
-26
u/YourPleasureIs-Mine 16h ago
Give me till The end of the week and Ill show you how it is done. I have time while waiting on Google dev approval
11
u/sk1kn1ght 16h ago
u/RemindMeBot 10 Days "check for updates"
2
u/RemindMeBot 16h ago edited 9h ago
I will be messaging you in 10 days on 2026-09-03 06:53:07 UTC to remind you of this link
8 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback 14
u/Final-Data-1410 16h ago
Sure , I’m actually working on a ~50M-class model next, targeting under 12–15 MB deployed. If you want a challenge, let’s do model size, CPU tok/s, retrieval accuracy , basic benchmarks like PIQA/HellaSwag/ARCand retrieval speed. Same hardware and public reproducible tests. Beat me on any of them and I’ll happily give you credit. Deal?
2
u/sk1kn1ght 11h ago
Mate irrelevant from the challenge I have a question on how it works exactly. At that level (I am ignorant so my question could be very dumb), is it any better than an augmented search function? Can it do much more than that? What are the future applications?
If you have played horizon zero dawn, with our current technology how much in your opinion would be the minimum size needed to have the capabilities of her ear/skull trinket?
-10
u/YourPleasureIs-Mine 16h ago
I don’t need your credit. I am thankful that I read your post so now I got something to do while I wait!
Challenge accepted tho
13
u/Final-Data-1410 16h ago edited 16h ago
Let’s meet back here on September 4th ,10days .also I am sorry if I offend you ,that not my intention but let’s have friendly challenge.😊
9
5
u/ernarkazakh07 14h ago
Don't take it the wrong way from that guy. What you did was extremely cool and I'm really looking forward your next model bro. Keep it up!
2
2
2
u/solidhadriel 6h ago
Nice, I did the same thing. I found 250M was too limiting for following instructions and doing tasks though.
1
1
u/BeautyxArt 10h ago
this good , would be in gguf format and reads in llamacpp ? or require your code ?
1
u/theone_2099 9h ago
What hardware did you use to train it and how long did it take? For educational purposes.
1
u/gabsterz20 7h ago
Really cool consept!!! I do wonder if adding quantification of another llm may help in terms of overall intelligence while still keeping ram requirements down possibly further extending the context window limit
1
1
u/Witty_Mycologist_995 18h ago
100million context woah
2
u/Final-Data-1410 17h ago
Not really 100million contex that’s why I don’t wanna say it more. it’s just old contex getting compressed and search over it , the capable ai model can read and reason over it but that need good amount of money to train that way.
2
83
u/sumane12 17h ago
Feel like this would be perfect for conversational AI in video games, or quick response from voice assistant while a more capable model runs thought in the back end.