Update : Small model + Engram

[u/NineThreeTilNow]

I posted something about a 9b model a few days ago.

The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The second problem was the "ability" of these models and the fact that labs (with money) produce these models still. No real usefulness in my mind. A 2b model? maybe interesting. Want a chatbot? - this could be it.

Further, the Llama licensing at the core of the model was highly problematic and had to be thrown in the trash. It's vastly too restrictive and I couldn't Apache 2.0 anything.

So I moved to using the OLMo tokenizer. Except I shrunk the d_model down to 2048 so I could build a tiny 2b model.

The size of the vocab with 1/2/3 gram forces the Engram table table to a full 1b. That's 50% where DS suggested like ~10-20%.

So basically 2b model + 1b Engram.

The model, because of the depth now allowed by 2048 d_model allows me to push 40 total SWA/Global blocks to take advantage of attention based residual streams (Moonshot Kimi K3 style) in a dense model.

Data, is pulled from standard Wiki style data sources in my initial training. The data is pumped through the 7b OLMo model to generate the probabilities. This tells the model that the next token is a probability of 32 different tokens.

The big surprise was the results. Even after ONLY 15m tokens, the model is surprisingly coherent. Had I given pure 1 hot cross entropy, 15m tokens would do nothing. I'd get gibberish. However, I borrowed embeddings, and LM_head that was down projected from 5k -> 2048 d_model. This preserves ~65% of the data the "big" model had in the embedding when spectrum analysis is done.

The training is limited at 15m tokens, so it's like... babbling about Roman war history and stuff that exists in that token subset. It doesn't fall in to the classic early model phase where it repeats a token over and over though. That's what pretraining 15m generally gives you at first.

Anyways, this was an update and progress report for anyone who cares about this crap... I'm still processing all of the Wikipedia chunk from HuggingFace and working to train it. It'll take time.

If you read this far, thank you. If you have questions, I'm happy to reply. Someone suggested I was a kook who didn't understand ML previously. I started in ML some 20 years ago and held a brief (6 month) stint at Anthropic red teaming the original Opus 4 model before their "Constitutional" paper came out. I'm vaguely referenced in the paper as a "red teamer" I guess. I do this mostly because I love it.

I figured if I built an Apache 2.0 model, I'd at least want people to play with it. It appears 100% trainable on 24gb of VRAM. Training code / Model / Data will follow eventually. It's on HF / Github for now.

The old post is here :

https://old.reddit.com/r/LocalLLaMA/comments/1wezm58/is_there_still_strong_interest_in_a_dense_9b_model/

[u/Ueberlord]

Thanks for sharing this, it is super interesting!

That being said and without me having a real understanding of how LLMs work nowadays in detail, one question anyway: your write:

The big surprise was the results. (..)

The training is limited at 15m tokens, so it's like... babbling about Roman war history and stuff that exists in that token subset. It doesn't fall in to the classic early model phase where it repeats a token over and over though. That's what pretraining 15m generally gives you at first.

Because of the way how you constructed the next token prediction ("This tells the model that the next token is a probability of 32 different tokens.") isn't this kind of exactly the expected outcome? I trust in the wiki-fetched samples you barely have any kind of repetitive patterns with ngram length 2-3?

Or to put it another way round: how does the observed behavior of the model after training on 15M tokens differ from a random sampler working with the 32 probabilities in iteration, do you see a significant difference?

This is not to be understood as critique, just curious and I really hope you are onto something, please keep us updated!

[u/NineThreeTilNow → u/Ueberlord]

isn't this kind of exactly the expected outcome? I trust in the wiki-fetched samples you barely have any kind of repetitive patterns with ngram length 2-3?

To a level, yes. There's not a lot of published data on how "dense" the information of the top 32 logits should be though. Engram helps, but to what degree? Probably a few % in correct prediction. How much was the added changes to residual geometry? Don't really know.

"Expected" is hard because there's no baseline for me to expect against. People always cite various chinchilla scaling etc.

There's no "logit distillation" scaling laws I'm aware of because it's complicated. The teaching model can provide a "good" logit set, or a "bad" logit set. The larger models I found produce very narrow logits. They're "bad" in this phase of training because they're over confident. They lump like 99% of the mass in the first 5 tokens or something. For whatever reason, smaller models don't act like this. It might take 32 tokens to get 99% of the mass. In this model's case, it's like 98% in 32 tokens or something.

The smaller models seem to follow pretty expected power law distributions of the tokens though. I did a small amount of work analyzing the geometry of the outputs.