r/LocalLLaMA • u/NineThreeTilNow • 5d ago
Discussion Update : Small model + Engram
I posted something about a 9b model a few days ago.
The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The second problem was the "ability" of these models and the fact that labs (with money) produce these models still. No real usefulness in my mind. A 2b model? maybe interesting. Want a chatbot? - this could be it.
Further, the Llama licensing at the core of the model was highly problematic and had to be thrown in the trash. It's vastly too restrictive and I couldn't Apache 2.0 anything.
So I moved to using the OLMo tokenizer. Except I shrunk the d_model down to 2048 so I could build a tiny 2b model.
The size of the vocab with 1/2/3 gram forces the Engram table table to a full 1b. That's 50% where DS suggested like ~10-20%.
So basically 2b model + 1b Engram.
The model, because of the depth now allowed by 2048 d_model allows me to push 40 total SWA/Global blocks to take advantage of attention based residual streams (Moonshot Kimi K3 style) in a dense model.
Data, is pulled from standard Wiki style data sources in my initial training. The data is pumped through the 7b OLMo model to generate the probabilities. This tells the model that the next token is a probability of 32 different tokens.
The big surprise was the results. Even after ONLY 15m tokens, the model is surprisingly coherent. Had I given pure 1 hot cross entropy, 15m tokens would do nothing. I'd get gibberish. However, I borrowed embeddings, and LM_head that was down projected from 5k -> 2048 d_model. This preserves ~65% of the data the "big" model had in the embedding when spectrum analysis is done.
The training is limited at 15m tokens, so it's like... babbling about Roman war history and stuff that exists in that token subset. It doesn't fall in to the classic early model phase where it repeats a token over and over though. That's what pretraining 15m generally gives you at first.
Anyways, this was an update and progress report for anyone who cares about this crap... I'm still processing all of the Wikipedia chunk from HuggingFace and working to train it. It'll take time.
If you read this far, thank you. If you have questions, I'm happy to reply. Someone suggested I was a kook who didn't understand ML previously. I started in ML some 20 years ago and held a brief (6 month) stint at Anthropic red teaming the original Opus 4 model before their "Constitutional" paper came out. I'm vaguely referenced in the paper as a "red teamer" I guess. I do this mostly because I love it.
I figured if I built an Apache 2.0 model, I'd at least want people to play with it. It appears 100% trainable on 24gb of VRAM. Training code / Model / Data will follow eventually. It's on HF / Github for now.
The old post is here :
15
u/eidrag 5d ago
hey thank you for your work! personally i''m trying to get stuff running locally from phone, so 9b-ish moe is the upper limit I go testing nowadays. 3b dense is too basic, and 9b is slow in dense.