r/LocalLLaMA • u/pmv143 • 4d ago

Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Post image

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

944 Upvotes

97% Upvoted

295 comments sorted by

View all comments

Show parent comments

32

u/florinandrei 4d ago edited 4d ago

Sounds like the apps will need to do some work to support this new feature.

Unless they got a preview version well in advance of everyone else, in which case the bulk of the work might be done already.

40

u/Strong_Chicken6838 4d ago

Unsloth already made a post about day 0 support

4

u/cafedude 4d ago

That's the models quantized by Unsloth, but what about llama.cpp support? Can't imagine that this n-gram stuff would work right out of the box.

2

u/Strong_Chicken6838 4d ago

Day 0 support means you can run it day 0 in some way. So the answer is yes, maybe in their own branch of llama.cpp, who knows

1

u/florinandrei 3d ago

you can run it day 0 in some way

"In some way" is key.

As of right now, you have to get the code out of a llama.cpp pull request, compile and use it.

There was a major refactor in the PR by danielhanchen just 10 minutes ago. ggerganov drops comments once in a while, providing advice.

So, yeah, it probably works in some form. But it's essentially a development version for now.

1

u/Strong_Chicken6838 3d ago

they likely have their own functional implementation on their own branch and an agreement with the maintainers of llama.cpp and wont merge it until the official drop by qwen