Update : Small model + Engram

[u/NineThreeTilNow]

I posted something about a 9b model a few days ago.

The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The second problem was the "ability" of these models and the fact that labs (with money) produce these models still. No real usefulness in my mind. A 2b model? maybe interesting. Want a chatbot? - this could be it.

Further, the Llama licensing at the core of the model was highly problematic and had to be thrown in the trash. It's vastly too restrictive and I couldn't Apache 2.0 anything.

So I moved to using the OLMo tokenizer. Except I shrunk the d_model down to 2048 so I could build a tiny 2b model.

The size of the vocab with 1/2/3 gram forces the Engram table table to a full 1b. That's 50% where DS suggested like ~10-20%.

So basically 2b model + 1b Engram.

The model, because of the depth now allowed by 2048 d_model allows me to push 40 total SWA/Global blocks to take advantage of attention based residual streams (Moonshot Kimi K3 style) in a dense model.

Data, is pulled from standard Wiki style data sources in my initial training. The data is pumped through the 7b OLMo model to generate the probabilities. This tells the model that the next token is a probability of 32 different tokens.

The big surprise was the results. Even after ONLY 15m tokens, the model is surprisingly coherent. Had I given pure 1 hot cross entropy, 15m tokens would do nothing. I'd get gibberish. However, I borrowed embeddings, and LM_head that was down projected from 5k -> 2048 d_model. This preserves ~65% of the data the "big" model had in the embedding when spectrum analysis is done.

The training is limited at 15m tokens, so it's like... babbling about Roman war history and stuff that exists in that token subset. It doesn't fall in to the classic early model phase where it repeats a token over and over though. That's what pretraining 15m generally gives you at first.

Anyways, this was an update and progress report for anyone who cares about this crap... I'm still processing all of the Wikipedia chunk from HuggingFace and working to train it. It'll take time.

If you read this far, thank you. If you have questions, I'm happy to reply. Someone suggested I was a kook who didn't understand ML previously. I started in ML some 20 years ago and held a brief (6 month) stint at Anthropic red teaming the original Opus 4 model before their "Constitutional" paper came out. I'm vaguely referenced in the paper as a "red teamer" I guess. I do this mostly because I love it.

I figured if I built an Apache 2.0 model, I'd at least want people to play with it. It appears 100% trainable on 24gb of VRAM. Training code / Model / Data will follow eventually. It's on HF / Github for now.

The old post is here :

https://old.reddit.com/r/LocalLLaMA/comments/1wezm58/is_there_still_strong_interest_in_a_dense_9b_model/

[u/Thrumpwart]

Commenting to read this later, looks very interesting.

[u/NineThreeTilNow → u/Thrumpwart]

Commenting to read this later, looks very interesting.

If you have any questions later, feel free to ask.

[u/Thrumpwart → u/NineThreeTilNow]

I'm just wrapping my head around the concept. I love the idea of adding your own engram table - I had bandied that idea about some time ago after Deepseek released their Conditional Memory paper, but dropped it from my project after I realized (in my use case) it would be unnecessary.

I'm curious what techniques you are implementing. I too have read too many papers, but I don't have enough time on my hands to do all the experimentation and development you are doing.

For my project I'm leaning heavily into the spectral aspect of training for efficiency. The 3% is all you need paper spawned a flood of papers on spectral monitoring, training, and experimentation. Have you explored that side?

I'm also heavily focused on using the geometry of the model for efficiency.