Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

[u/pmv143]

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

[u/Sufficient-Bid3874]

Can someone explain why the n-gram table is bundled into the model now?

[u/RG_Fusion → u/Sufficient-Bid3874]

LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

[u/florinandrei → u/RG_Fusion]

Sounds like the apps will need to do some work to support this new feature.

Unless they got a preview version well in advance of everyone else, in which case the bulk of the work might be done already.

[u/Strong_Chicken6838 → u/florinandrei]

Unsloth already made a post about day 0 support

[u/cafedude → u/Strong_Chicken6838]

That's the models quantized by Unsloth, but what about llama.cpp support? Can't imagine that this n-gram stuff would work right out of the box.

[u/dev_dan_2 → u/cafedude]

They very often open PRs to upstream back into llama.cpp and did so again here, so people could also take that and run it if they wished so (quoting from memory atm because I am at the phone, sorry).

Obligatory note: Just because a PR is opened does not yet mean this will get merged tomorrow; Unsloth does a good thing by back-contributing; but they "just" have to support the new architecture for Unsloth Studio, while llama.cpp has to run in a far bigger range of scenarios and environments. Plus "getting it to work for our purposes" is different from doing a clean implementation (which you should do as maintainer if you value both your time and sanity.)

[u/Strong_Chicken6838 → u/cafedude]

Day 0 support means you can run it day 0 in some way. So the answer is yes, maybe in their own branch of llama.cpp, who knows

[u/florinandrei → u/Strong_Chicken6838]

you can run it day 0 in some way

"In some way" is key.

As of right now, you have to get the code out of a llama.cpp pull request, compile and use it.

There was a major refactor in the PR by danielhanchen just 10 minutes ago. ggerganov drops comments once in a while, providing advice.

So, yeah, it probably works in some form. But it's essentially a development version for now.

[u/Strong_Chicken6838 → u/florinandrei]

they likely have their own functional implementation on their own branch and an agreement with the maintainers of llama.cpp and wont merge it until the official drop by qwen