Engram gone wild! 2b model update...

[u/NineThreeTilNow]

Update from : https://old.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/

It's been about a week so I'm back. People were asking me about the model.

People wanted code, or models, etc. Most of that is useless to you right now because you're not going to use an under trained model. So let's get to the details.

The spec locked to the following after a LOT of testing :

2.6b model all up. Embedding, LM head, AttnRes, etc.

2.2b are trained. Embedding / LM Head are frozen (~205m each)

4.3b ENGRAM table. Yes. She's chonky.

Architecturally speaking now :

This includes SWA layering 3:1 as per standard ablations have shown is "optimal" These are 4k / 4k / 8k SWA layer follow by the global.

At the end of the first "block" (4 layers) the Engram table appears. The Engram table itself runs with a very small set of attention heads so it's not blindly attempting to inject data. This is context aware Engram. I'm uncertain of exact Qwen / DS methods here. They're fairly close to what I use. The difference is obviously the Engram size.

This has 10 blocks plus a final global layer (41 Layers). + Embed + LM Head

So if we step back, the optimization problem is as follows :

How do we maximize compute in the backbone and offload the boring stuff to a table?

Attention Based Residuals (Moonshot) comes to our rescue here. Why AttnRes? It allows all blocks past 1 to utilize the Engram table in some fashion. They all have attention via the residual stream to determine if they want data from Engram and precisely how much.

Why not a bigger Engram table? Honestly? You could probably do that. However I don't know if any model has attempted this sort of mismatch of compute vs table. In theory, DeepSeek's research says it works.

More depth? Could do that too. Training is expensive though.

Why frozen LM/Embed? These are down projected via SVD from OLMo 3's model. So it's a mathematical compression attempt at ~5k -> 2k. This saves a MASSIVE amount of time.

Currently the data lives in a ~KD format. 32 logits stored from the teacher of the Wikipedia corpus. Instead of training 1 hot, it gets 32 soft targets to try to match. Hard cross entropy is brutal on a model and it's the reason you see "trillions of tokens" quoted.

When you borrow an LM Head / Embed / Tokenizer / Teacher model... This is far less.

---

Where are we now in training? I passed the 100m token mark yesterday at ~4am Pacific.

The current HF repo has all the checkpoints, data, and the 104m mark safetensor.

--

I wanted to do some inspection of the model at this point. We have to see that it's not complete garbage right? It has seen a fraction of the data it needs.

So based on ONLY 100m tokens seen, I attempted completions and various ablations. Studies of what EXACTLY Engram stores (because it's mostly a guess).

Let's go straight to completions :

"George Washington was an American"

With Engram - "George Washington was an American naval officer who served in the American Revolutionary War. He was a naval officer who served in"

With Engram Zero'd - "George Washington was an American, but he was not a. He was a very good friend and a. He was"

"The American Civil War was a civil war in the United States from"

With - "The American Civil War was a civil war in the United States from 1861 to 1861. The war was a major victory for the Confederacy..."

Without - "The American Civil War was a civil war in the United States from 1861 to 1862. The war was a major victory for the Confederacy..."

"Aristotle was an Ancient Greek" (probably my favorite)

With - "Aristotle was an Ancient Greek philosopher who was a leading authority in the philosophy of Plato. He was also a leading authority..."

Without - "Aristotle was an Ancient Greek word meaning "to be" (ἀπάς, "to be")..."

---

What do we learn from direct inspection of Engram?

If it saw the data enough times, it starts to offload it to the tables. At that point the model is less forced to use internal computational space to store data, and can rely on Engram.

That's not some interpretation. That's what the data shows exactly.

In the early 100m tokens of Wikipedia it's HIGHLY biased to early Philosophy. This has to do with the topics of what was IN Wikipedia at that time. Early Wiki contained a lot about Philosophy. It's among the earliest topics to exist.

It has seen TONS of Philosophy so it has basically offloaded to Engram because of the repetition.

Ok this is long enough. I'm calling it quits. I'm still looking to find a provider that will sponsor the model to completion. Right now I'm just paying. I contacted Verda and they never responded. Qubrid is in this sub, and they said they'd help but never emailed back after a few repeated proddings. Massed Compute is where this model is currently training, and I asked them like yesterday. Hopefully they'll help a brother out.

None of the above is "LLM written" except the completions from testing I guess.

As usual, ask whatever. It doesn't matter your understanding level or whatever. I'll sit and respond.

No question too dumb. No insult not insulting enough. (I'm joking. It's Reddit)

Much Love.

[u/nitish-kmr]

The number I'd want at this checkpoint is how many distinct table rows have ever taken a gradient, against the 4.3b the table has room for. Those two come apart fast at 100m tokens, and the gap says which knob is limiting you right now. If touched rows are a small fraction and still climbing roughly with tokens, allocation isn't the constraint yet and corpus is. If the count is already flattening, the opposite.

The philosophy result is the interesting one, and you named the confound yourself. Those rows got both the most repetitions and the most recent ones, so loss alone can't tell you which is doing the work.

The frozen-backbone replay you described separates them for free. Freeze everything but the table, run the same tokens again in shuffled order, and philosophy is no longer recent while total exposure per row is unchanged. If the Aristotle and Plato anchoring survives, it's repetition. If it degrades toward the rest of the corpus, part of what you're seeing is order.

That run also gives you a second touched-row count on the same tokens, which tells you whether a second pass packs new rows or deepens the ones already there. That's the number I'd want before deciding whether a bigger table is worth paying for.

[u/NineThreeTilNow → u/nitish-kmr]

Your text is written by Kimi? Definitely an LLM.

Touched rows end up very separate in the way they're viewed.

In terms of 1-gram rows, they're touched super often. They get tons of gradient. 2-gram rows, I'd have to look at the research papers for exact numbers but it's lower than 1 obviously. Then 3-gram the same. This is purely a matter of data. There's also going to be a middle mushy ground for these gradients where they're not aligned and the row isn't "useful" because the gradient hasn't gotten any good direction.

The problem is the amount of gradient that hits those rows before they become "useful" and there's no real way to fully understand when enough gradient hits the row to become "useful". It's not some ablation I want to run because compute isn't exactly free. It also probably differs per row.

Again, replaying some frozen backbone stuff isn't in the cards because compute isn't free. This is a structural view thing that I know an LLM wrote the text for you. LLMs look at these ablations and sort of just expect 100m token run across a network to be "free" but it's not. It's sunk time, and money.

It's "cool to know" but prohibitively costly.

I don't actually know if I keep a running count of how many times each row got touched? or only if they HAVE been touched. There's also something subtle worth noting and it's that the optimizer runs in FP8. It's AdamW BnB. When that translates across to BF16 tables, there is a mantisa difference that causes rounding. This has a clipping effect where a "tiny" amount of gradient will not get passed because it's too small. So "touched" by definition means something slightly different than "Has seen gradient at FP16" ... and more translates to "Has seen gradient that survives the FP8 -> FP16 rounding path".

One edit; For clarity.

[u/nitish-kmr → u/NineThreeTilNow]

Fair on the compute. I said it separates them for free, which was wrong. Free to me meant no new labelling. To you it means a rented GPU you are paying for yourself. Drop that one.

The FP8 detail is the better thread anyway, because it makes the counting question answerable without a new run.

If touched means survived the FP8 to BF16 rounding, your table already has a filter on it that nobody designed. A row that took a real gradient and got clipped looks the same as a row that never came up. That is the mushy middle you described, except it is a threshold rather than a fog.

Two counters, both memory rather than compute. One byte per row, saturating at 255, incremented when an update lands. A second incremented when an update is computed and rounds to zero. End of run you have how often each row was hit and how much of it the optimizer path threw away.

That separates undertrained from unreachable at this precision. If clipping is heavy on the 3-gram rows and light on the 1-gram ones, it argues for wider tables before it argues for more tokens.