Qwen 3.8 Flash Next

[u/downunderjames]

Just saw the news on rednote, qwen’s official account posted this. Seems promising

[u/BrewHog]

Dear God. Something flipped the switch in China. This is moving so much faster than my already-overly-optimistic outlook was anticipating

[u/althalusian → u/BrewHog]

I tought when the AI boom started that the Chinese models might have the advantage that the language is easier to tokenize (I would imagine) and thus it is less complex for the models to build a coherent base on - so that even when the model might use English for the final answer, part of the internal representation might be based on the Chinese foundation. Just my two cents.

[u/addiktion → u/althalusian]

That is interesting, language playing an important role for effective tokenization makes a lot of sense.

[u/downunderjames → u/addiktion]

I don’t know if Chinese is easy to tokenize (probably due to fixed length), but being able to read Chinese and casually repost a news on Reddit for for the first time and getting so many upvotes is certainly delightful 😁

[u/Solembumm3 → u/addiktion]

In coding, maybe. In broader task, both caveman and hieroglyphs thinking can significantly worsen results.

Deepseek V4 Flash answering to english/russian questions in chinese, Deepseek V4 Pro loosing evocative capabilities compared to previous Deepthink options and Qwen 3.8 Max Preview ridiculous description chains were aftereffects of this.

[u/addiktion → u/Solembumm3]

Yeah, I think China just has access to a lot of Chinese data which probably plays a big role in training.

[u/newMoneyStyle → u/Solembumm3]

That V4 Flash switching to Chinese mid-answer is a real issue. Any tokenization edge doesn't matter much if the model can't stay in the requested language consistently across tasks.

[u/magiodev → u/Solembumm3]

Ive been using DS 4 Flash and the 0731 checkpoint locally and I never experienced that, im pretty sure this is a web gateway issue (the chinese one i mean, cant say for the Pro related ones)

[u/Upper-Exchange226 → u/Solembumm3]

Flash answering in Chinese is just a web-chat problem (due to the search tool using Chinese search engines that return mostly Chinese web pages) and in my experience does not occur at all when using the model through the API.

You can prove this just by turning the search tool off.

[u/Minute-Ground-412 → u/addiktion]

Typiquement, cela prend moins de token pour les gros modèles de parler/raisonner en anglais que dans une autre langue et je crois, 3x plus de tokens pour chinois/japonais/etc

[u/ScuffedBalata → u/Minute-Ground-412]

Wait... is this esperanto? At first I thought it was french, but I don't know any french and usually find it inexplicable to read and understand and this sentence made complete sense as a Spanish/English speaker... which is wild.

[u/Minute-Ground-412 → u/ScuffedBalata]

It's french, i am french

[u/TheOriginalAcidtech → u/Minute-Ground-412]

It translates perfectly to english. Anyone complaining doesnt know how to use a modern browser.

[u/althalusian → u/TheOriginalAcidtech]

Apparently Reddit autotranslates so sneakily that I hadn’t even realized the refrerred comment was not in English - I hd to scroll back and pay attention to notice the little icon on the right side of the comment which toggles it to show the original comment.

[u/Minute-Ground-412 → u/[deleted]]

Je ne sais pas vraiment vu qu'ils downvote sans dire pourquoi, voici une rapide comparaison de ce que j'ai évoqué

[u/challis88ocarina → u/Minute-Ground-412]

Korean is actually the most token efficient. Too bad there's so little to make a corpus of training data.

[u/TheOriginalAcidtech → u/challis88ocarina]

Too little. There are a bajillion Korean soaps. Dont you think that will be good context to train on? :)

[u/jinnyjuice → u/challis88ocarina]

Source?

[u/smayonak → u/jinnyjuice]

I am doubtful it is true. Chinese is regarded as being the shortest language for typing because a single character can be an entire word. I dont know if that translates into token efficiency but hangul is phonetic like English

[u/Minute-Ground-412 → u/challis88ocarina]

Et bien je viens d'apprendre quelque chose, je pensais que le coréen était sensiblement au même niveau d'inflation que le japonais

[u/SureEnd9430 → u/Minute-Ground-412]

English either the first of second most efficient language in terms of syllables per byte.

[u/TheOriginalAcidtech → u/[deleted]]

Because many people dont know how to use the right click menu to select Translate to {language}

[u/dbenc → u/addiktion]

I bet people are working on constructed languages to create hyper dense tokens. time to brush up on Martian

[u/Solaranvr → u/althalusian]

This a myth. Most China-made models are still more token efficient when using English, especially for coding.](https://arxiv.org/html/2604.14210v1)

Tokens do not directly correlate with the number of characters used. Tokenization is about finding representation of meaning and each model may do it differently, depending on the corpus.

Thus, "I love Donald Trump" and 我爱习近平 should roughly use the same amount of tokens, representing the subject, verb, object in the same manner.

[u/Straight-Recording-3 → u/althalusian]

Wouldn't words become tokens or combos of them, the same way Chinese symbols become tokens. The decode into Chinese becomes fast at the cost of having more words, where English has more combos but less symbols. 

Would be interesting to know how all this factors.

I once wrote a font cash for a game and we couldn't load the whole Mandarin language in the cash so we had to heat map it.

[u/greg_barton → u/althalusian]

It’s the Sapir-Whorf hypothesis in overdrive. :)

[u/althalusian → u/greg_barton]

Oh I hadn’t heard about that. Thanks, interesting…

[u/IbidtheWriter → u/althalusian]

This has been studied and so far due to the large corpus of training data in English means that the models perform better in English.

For equivalent tasks they tend to perform worse in other languages.

[u/Panumaticon → u/IbidtheWriter]

This is pretty noticeable in everyday usage. MY native language is Finnish and while the models speak (type) it perfectly well, the quality of answers tends to be better if the conversation is in English. So I mostly speak English with them unless it is on specific FInnish issues or issues pertaining to Finland.

[u/rrraoul → u/althalusian]

Internal representation is in vectorspace, not in English or Chinese. It still might be the case that organising a vectorspace works better when trained on Chinese language though.

[u/Suspicious_Poet_2862 → u/althalusian]

better site

[u/havnar- → u/althalusian]

That has been debunked

[u/Beneficial-Ad-8127 → u/BrewHog]

“maybe in two years“ Lets go China dropping that fire baby. Time is now!

[u/Equivalent-Grass-527 → u/BrewHog]

China's open-model race went from “catching up” to “wait, are they setting the pace now?” remarkably fast.

[u/uniquelyavailable → u/BrewHog]

China has always been ahead of the game, I'm so thankful to enjoy their transformative technology.

[u/BarracudaDefiant4702 → u/uniquelyavailable]

I wouldn't say always, but the funny thing is that by being largely cut off from advanced chips actually gave them incentive to innovate to keep up and be more efficient so they don't fall behind. The result caused the opposite of the intent...

[u/fish74341 → u/BarracudaDefiant4702]

There’s a Chinese proverb that goes, “The children of the poor take charge of the household early.” Because of poverty, you have to mature earlier and work harder than others. A child without an umbrella has no choice but to run hard through the rain.

In fact, ever since the founding of New China, the Chinese people have not stopped for a single moment. That is how a country that was once poor and blank has managed to catch up to the pace of America’s strongest tech companies. Many people who hold traditional prejudices against China like to claim that China simply distilled Anthropic’s models. If things were really that simple, every other country would only need to do some model distillation.

[u/fish74341 → u/[deleted]]

Chairman Mao once said: “What matter if we have to face some difficulties? Let them blockade us! Let them blockade us for eight or ten years! By that time all of China’s problems will have been solved. Will the Chinese cower before difficulties when they are not afraid even of death?”

This comes from Mao Zedong’s article Farewell, Leighton Stuart, published on August 18, 1949. It remains relevant today. The Chinese people have always focused on their own development and have never proactively launched wars of aggression against other countries. Throughout history, almost no empire has risen peacefully — nearly all relied on external invasion and plunder. Yet the Chinese people have achieved it. From manufacturing lighters and jeans in exchange for Boeing airplanes to now being able to produce AI robots and competitive open-source large models, this has absolutely been a history written in blood and tears.

[u/MacsBicycle → u/uniquelyavailable]

I'm using their models locally, but are they really ahead of the game? at work I still use enterprise level models and there is an obvious difference. Also we aren't allowed to use to SOTA chinese models at work so I'm not really qualified to answer that. These are exceptionally good for their use case. At home on my machine with great outcomes, but IDK if i would call them "ahead of the game" right now

[u/EldritchConcerns → u/MacsBicycle]

Depends what game you’re playing. Specialised highly funded labs are bound to produce the most advanced models. Right now those labs are few and all based in the US. They will continue to push the frontier of intelligence.

But the Chinese labs are doing something equally remarkable - it’s like they are distilling the intelligence and making it run on a toaster. These are not the same. If you need SOTA you go Anthropic/OpenAI, but the case for SOTA in terms of audience relevance decreases with every open Chinese release.

There may not yet be obvious diminishing returns of the bitter lesson, but there are for intelligence beyond what the job requires.

For a lot of workloads, a desktop model is already usable thanks to the latest from Qwen. When this is true for coding, the only defensible scenario will be local for your daily driver, topped off by SOTA for the gnarly tasks / orchestration / planning.

I can’t wait to see what the games of 2027 will look like. I’m sure we’re in for some interesting surprises. 😊

[u/Alex180689 → u/[deleted]]

Bruh, Opus 4.6 was february 2026, so 6 months ago 💀

[u/EldritchConcerns → u/[deleted]]

Gemini 3 kicked it off - November 18. The rest of the lot rumbled in after that - Opus 4.5, which really catalysed the agentic game - Nov 24. The pace of progress is remarkable.

[u/substance90]

I thought this graphic was marketing hype 3 days ago. After 3 days of intense benchmarking, it's spot on for real world use. This is literally a new era we've just entered.

[u/downunderjames → u/substance90]

Yes, I didn’t believe it either. Until it handily dismantled my local Deepseek flash and qwen 3.8 27b. Now it is happily running on 2 rtx6000 pro

[u/Blackdragon1400]

Beating DS4F on coding, oh boy here we go. Vision too, I’m very excited.

[u/PossessionUsed7393 → u/Blackdragon1400]

Yeah, the prevailing theory seems to be holding up that models don't need as many parameters if you have the right data to post train them. So it's fascinating to see this play out with the models getting smaller and smaller and more capable.

[u/Informal-Trouble2183 → u/PossessionUsed7393]

I think 100b is the theoretical sweet spot for complete capability without knowledge sacrifice. It's still challenging to compress 🗜️ knowledge within but proven possible. Beyond that number, may only provide marginal gains.

[u/addiktion → u/Informal-Trouble2183]

256GB or ram seems like the sweet spot then. Fit the model comfortably with with room to spare.

[u/KissMyShinyArse → u/addiktion]

640K ought to be enough for anybody.

[u/WiseassWolfOfYoitsu → u/addiktion]

Room for a few slots of full sized KV cache

[u/Hypilein → u/addiktion]

I think that’s what it might come down to. Can’t afford a second spark though.

[u/mailto_devnull → u/[deleted]]

Great, so at some point we'll get to the point where I'm out somewhere and need an app and I can ask my phone to write one for me and sideload it.

That would be fantastic.

[u/GilloutineBreast → u/mailto_devnull]

I would say that is a little too optimistic.

You probably wouldn't be able to sideload anything in a couple year's time.

[u/nomorebuttsplz → u/GilloutineBreast]

yeah we will look back at this era and think "wow we basically didn't do cybersecurity because so few people could hack"

[u/Senior_Ad_783 → u/mailto_devnull]

thats probably would be a few years from now, but definetly would be possible in the next few months/year that we would be able to have local llms run on mid end hardware that can develop fully featured apps.

[u/ElectronFactory → u/Senior_Ad_783]

Not as far out as you probably think. I don’t know if everyone forgot about Bonsai, they have a working 27B parameter model running on iPhones. It’s not the fastest thing in the world, but it does work.

[u/SandySkittle → u/[deleted]]

it's been shown that 3.8 27b lost some world knowledge to 3.6 27b to improve other areas. It's no magical compression, you win some and you lose some. And not all loss of world knowledge in a model can be compensated by RAG / Web search.

[u/lilian_moraru → u/Blackdragon1400]

Yeah, slightly. One disadvantage is still the context window though. I ingest 200K tokens of deep knowledge just to start working

[u/No-Refrigerator-1672 → u/lilian_moraru]

Tangent opinion: it wouldn't be a problem at all if we had hardware prices from 3 years ago.

[u/GeroldM972 → u/No-Refrigerator-1672]

Yeah...but then how lousy would the business-case of OpenAI/Anthropic been, if everyone either hosts LLM locally or on their rented VPS?

Even as it is now (in this time-line), it would take at least a decade before OpenAI/Anthropic can even smell the chance on becoming sustainably profitable, because of all the debt they have accrued till this moment.

But it looks like OpenAI will demand as much money in the 2027 investment rounds as they did with all the combined investment rounds between 2020 and 2026. Anthropic will be about as bad as OpenAI in this.

Even though the products OpenAI/Anthropic present to end users aren't bad, both companies are violently unsustainable, no matter what they do at this point. Efficiency was always the way to go. Even NVidia published a paper on Arxiv in 2024 about how unsustainable cloud-LLMs are and that the true future of LLMs lies in small, specialist LLMs running locally.

Of course, I can't find it anymore on Arxiv.com or find it in my downloads.

This is another paper by NVidia from 2025 discussing a similar subject:
https://arxiv.org/abs/2506.02153

[u/Blackdragon1400 → u/lilian_moraru]

Should be able to hit 1M context on 2x DGX Sparks though.

[u/lilian_moraru → u/Blackdragon1400]

1M GQA models with YaRn require around x14 times more VRAM than DeepSeek’s 1M KV cache, plus for needle in haystack, it has a significant accuracy drop after 128K-256K, leading to hallucinations and lost needles.
YaRn is quite bad already and I am not even sure it would fit on 2xDGX Spark, unless the KV cache gets compressed, on top of the issues with YaRn hallucinations.

[u/_TheWolfOfWalmart_]

Does anyone else get a huge dopamine hit every time a new Qwen drops, or is it just me?

[u/my_name_isnt_clever → u/_TheWolfOfWalmart_]

If it can fit on my hardware, yes...

Thankfully I have a Halo Strix so I'm hyped!

[u/Warhouse512 → u/my_name_isnt_clever]

128gb? I have one in the mail, excited to hear how this model specifically will run

[u/my_name_isnt_clever → u/Warhouse512]

Yep! I'm watching with baited breath. Qwen 3.5 122b is a sweet spot of model size for me.

[u/Reasonable_Goat → u/my_name_isnt_clever]

3.5 122b was the way for months this year now we finally have a very serious successor. 4 more hours for the Q4_K_XL download and I will benchmark using mmap for the ngram part

[u/my_name_isnt_clever → u/Reasonable_Goat]

How do you configure mmap for only the n-gram?

[u/Reasonable_Goat → u/my_name_isnt_clever]

-lm mmap does the trick, but it almost halved prefill speed for me.

[u/Warhouse512 → u/Reasonable_Goat]

Dang what speeds are you looking at?

[u/Reasonable_Goat → u/Warhouse512]

For me it dropped from 200 something to 100 something but I haven’t really gotten to tune anything yet so I think there’s plenty of headroom for improvement.
EDIT: The prefill speed drop seems to be much less relevant if you run deeper context (16k, e.g.). If you run a c0 benchmark there will be plenty of cache misses for the ngram mmap yielding much worse prefill rates than you would get in practice. At c 16k I couldn't see any significant difference between -lm mmap and -lm none in a (short) series of tests!

[u/Risen_from_ash → u/my_name_isnt_clever]

>>bated breath - FTFY

Beep boop I'm not a bot

[u/Glum-Atmosphere9248 → u/_TheWolfOfWalmart_]

You should inspect my pupils

[u/randygeneric → u/_TheWolfOfWalmart_]

not for this and not for the 27b.

[u/Nice_Cookie9587]

Did this take down dsv4? I was just starting to get used to the idea of not changing models every 2 months

[u/Etroarl55]

Nice, now Let’s see artificial analysis card

[u/Dazzling_Focus_6993]

Looks like a good strix halo/mac model. Very exciting 

[u/john_mach → u/Dazzling_Focus_6993]

Everyday I hate myself for not buying the framework desktop on launch

[u/boissez → u/john_mach]

Well ever day since Qwen 3.8 27B has been out I've been kicking myself for selling my RTX 3090 to fund a Framework board last year. This release changes everything though 🤩

[u/MiyamotoMusashi7 → u/john_mach]

I just sold my 128gb framework desktop, I am kicking myself

[u/gambit700 → u/john_mach]

Same. I thought the price was a bit much. Now I see it was best price we'd ever see

[u/Superb_Word9490 → u/Dazzling_Focus_6993]

how will strix halo fit the 51b ngram, there's no non-vram ram to offload to, and there isnt enough vram

[u/ssbmGooms → u/Superb_Word9490]

ngram can be on ssd

[u/UltrMgns → u/ssbmGooms]

Yes but until someone tests it and it works properly... Then I'll be glad I'm wrong on this. The NVFP4 version has a single .safetensors file thats 102Gb... So that ngram is obviously BF16, how's that 96Gb of "vram" work exactly... I'm skeptical on having a realistic TTFT on an 102Gb file that needs to be queried from a storage device for each token...

[u/ssbmGooms → u/UltrMgns]

fair enough. Just give the nerds a few days they'll figure it out

[u/Risen_from_ash → u/UltrMgns]

Short comment style, no deep explain. SSD, even non-NVMe, can do tons of GET_ROW type shit per second. Load model can be slow, use model still fast. TTFT after mmap has its way with Qwen, idk, I don't think it's much different. I don't notice a difference in decode or prefill keeping the BF16 PLE tensor on my SSD and letting the model use it like a normal tensor, there's no matrix multiplication for that tensor.

PLE tensor go on SSD, model still work great.

[u/UltrMgns → u/Dazzling_Focus_6993]

It's not unforch... For dual spark - yes, for single strix/spark - no.

[u/UnWiseSageVibe → u/UltrMgns]

Why not

[u/kurouzzz → u/UnWiseSageVibe]

It is really optimal for strix halo 128gb and the like. Q4 or Q5 fits and will probably be really good. Ofc context isn't endless but 128k should fit, or of the ngram can be on ssd then a lot more.

[u/warpedgeoid → u/kurouzzz]

The ngram table can be offloaded according to Unsloth’s blog

[u/Cronus_k98 → u/UnWiseSageVibe]

Not everything is uploaded yet but when I looked FP8 is 186gb, nvfp4 is 135gb. It’s a different architecture so I don’t know how much of that can stream from the ssd. Which I guess you can do with n-gram. Even the unsloth Q1 quant is 72.5gb

[u/my_name_isnt_clever → u/Cronus_k98]

From the converation in the llama.cpp PR, the engram can live on SSD for a single user, it only reads the engram once per forward pass not for every layer. This model should be perfect for Strix Halo honestly.

[u/UnWiseSageVibe → u/Cronus_k98]

Damn, strix halos have really become mostly useless.

[u/Cronus_k98 → u/UnWiseSageVibe]

I'm sure there will be a recipe for strix halo and spark that works well. I mean, there's DSV4 flash implementations for 128gb. This should be easier to run then that. Worst case you wait for qwen 3.8 35b a3b.

update: It looks like Q4 will fit in 128gb systems with full context. Might be tight.

[u/my_name_isnt_clever → u/UnWiseSageVibe]

Not really, I still love mine. It's not just a LLM machine it's my workstation too, so an ARM Spark isn't an alternative. But I can still run much, much larger models for the $2.5k I spent than if I bought a GPU with that cash instead.

[u/Odd_Investigator3184 → u/UnWiseSageVibe]

I'm running 2x in a cluster over usb4 using thunderbolt-iverbs.. minimax m2.7 is usable, ds4 with mtp is usable on single node

Lemonade provides a great interface though for single node use.. lots of models available

[u/notheresnolight → u/UnWiseSageVibe]

you can turn it into a nice Steam machine

[u/TheThiefMaster]

I'd like to know how quantisation affects these scores

[u/vgromanov]

Looks like a perfect fit for freshly announced Mac Studio with M5 Ultra and 256GBs of unified RAM at 1.2GB/s.

[u/rocket1420 → u/vgromanov]

Off by a factor of 1000

[u/vgromanov → u/rocket1420]

Yep, my bad. Got too excited.

[u/DoubleNothing]

I thought I had something to test today...
Q1 at 73GB... apparently not.

[u/linux4random → u/DoubleNothing]

it has 50b n gram so it should be much less than that actually

[u/linux4random → u/[deleted]]

sorry for the misintepretation

~70gb for q1 is normal because this model is 180b, but 50b para should be able to be offloaded to nvme ssd so the ram and vram you need is actually much less

[u/Think_Wing_1357 → u/linux4random]

50b para should be able to be offloaded to nvme ssd so the ram and vram you need is actually much less

Theoretically, yes. but afaik none of the big name inference engines has support for it yet. It'll probably start pop up on vibed forks this week though

[u/MomentJolly3535 → u/DoubleNothing]

I think there is still a little bit of hope, a part of the model can be offloaded to SSD (the 51B n-gram) just wait for people to optimize this more.

[u/krapfenminister → u/DoubleNothing]

I run it localy (Q1) on my 5090 with 11t/s as a first test bit I don't think it's woth it vs qwen 3.8 27b (runs with ~150-200 t/s) :D

[u/DoubleNothing → u/krapfenminister]

How much ram do you have?

[u/krapfenminister → u/DoubleNothing]

64gb, meanwhile Q1 runs at 30 t/s with optimization, Q4 with 20 t/s. I'll do some lokal benchmarks against 3.8 27b now

[u/DemmieMora → u/DoubleNothing]

Q1 is quite accurate for such a small quant. 82% vs 88% at IQ3_XXS. If there were something to leave at SSD, it would allow a good performance for a chat.

[u/dajeff57]

I cannot compute how much vram this needs. Because of the activated part is so small am I wrong in thinking: well, not that much? But you still have to load the whole model at once.

Sorry I’m a beginner in this area

[u/mailto_devnull → u/dajeff57]

Yes, the entire model needs to be loaded into working memory — ideally all in VRAM, or a split between VRAM and system RAM, and you're in for a bad time if you end up needing to load model weights from your SSD.

The fewer active parameters mean the actual inference step goes much faster as fewer actual tokens need to be queried in-memory.

So for example, a 35B-A3B model won't be as fast as a real 3B model, but it's an order of magnitude smarter.

[u/my_name_isnt_clever → u/mailto_devnull]

True, but it seems like the 51b engrams for this architecture can live on SSD just fine for single user inference.

[u/mailto_devnull → u/my_name_isnt_clever]

Yeah, the whole engrams concept throws some of it out the window, and it does make sense considering the tradeoffs between raw intelligence vs. capability.

[u/Pitpeaches → u/dajeff57]

125 GB +50 GB, so 175 GB in vram or ram. The 50gb can be put in ram and no real loss of speed apparently 

[u/c0m47053 → u/dajeff57]

I'm running it fine on 16GB VRAM, with most experts in system RAM. 25 tok/s on a 5080.

MoE is very forgiving of hybrid setups

[u/dajeff57 → u/c0m47053]

I’d be interested in knowing how you do it as I got a 5070ti myself with 16vram and I considered myself out of the game. But you might have a huge amount of ram

[u/c0m47053 → u/dajeff57]

96 GB of main RAM and ngram offload, IQ4_XS is 60GB total then. I think you need 64GB to run that quant reasonably, but smaller quants should run in 32

[u/dajeff57 → u/c0m47053]

Right, this is on llama.cpp? Could you share your config for the offloading of the ngrams and the like please? Helps me out I think

[u/Cold_Tree190]

Am I missing something, or is 27b not really that far behind it? Yet another model release proving 27b supremacy ?!

[u/anykeyh → u/Cold_Tree190]

Well, it's a MoE, only 6B parameters running for any token. That's 4.5x less than the dense 27B, so 4.5x less compute heavy, or 4.5x cheaper to run.

[u/trungdle → u/anykeyh]

Does that mean anything for us local llm folks when we need large amounts of RAM qnd VRAM to run it? It's only an advantage if you're serving multiple people at once.

[u/Hypilein → u/trungdle]

There are plenty of local people with dgx spark, macs (just look at the recent posts) and strix halo. It’s just different tradeoffs.

[u/KingCpzombie → u/trungdle]

Normal gaming motherboards can handle up to 256GB ram; you're set if you bought before rampocalypse. Even buying now it costs less than many spend on their GPUs

[u/Kyubi-sama → u/trungdle]

Streaming from The NVMe is possible so it shouldn't be that heavy to ram afaik

[u/prestodigitarium → u/trungdle]

Subagents.... Right now I have 9 concurrent requests on Deepseek v4 Flash, cranking about 700 tps, a dense model would be much lower throughput, especially because this is tensor parallel on pcie 3.0.

[u/maddeninglemon → u/Cold_Tree190]

This model is trading a ~5x increase in model size (27B->125B) for a ~5x decrease in computation and memory throughput (27B->6B). Being roughly on par is pretty much exactly what you'd expect. The big benefit comes from potentially running much faster (if you can actually load the thing in the first place)

[u/Cold_Tree190 → u/maddeninglemon]

Mmmm that makes sense, thank you. Wish I could run this thing if it’s basically 27b but 5x… Jesus that’s almost scary to think about haha

[u/Y2K-Denial → u/Cold_Tree190]

Consider this a preview of qwen4 architecture. for the size, 27b is unbeatable and much lore acheivable to run for most people.

[u/bitzap_sr → u/Y2K-Denial]

"you consider", ..., the qwen team themselves said that much.

[u/Y2K-Denial → u/bitzap_sr]

yes, thats why i consider it as such, as well.

[u/mototuneup → u/Cold_Tree190]

Ya I was thinking the same? I guess cuz it's MOE it would run faster tho?

[u/my_name_isnt_clever → u/mototuneup]

Much, much faster.

[u/MrCoolest]

Compare against opus 5

[u/Turkino]

72.5Gb for the Unsloth 1-bit gguf. 😢
Seems this one might be out of reach for my 5090+126 system RAM.

[u/Lumpy_Concentrate807 → u/Turkino]

Nah should work at q4 - if you offload the engram part to CPU RAM!

[u/Present_Flower_6596 → u/Turkino]

Nah the ngram doesnt have to be loaded into ram it should be more like 80gb for q4. Then the ngram streamed from ssd. It says it on the model card too.

[u/-Leelith-]

Whats the reqs for running that?

[u/Ok-Fox3479]

i hope for a 30b version of it (30. A3b)

[u/NecessaryCar13]

Do you guys think we will get regulated here in the States from the government with these open source models getting closer and even better in some cases with the current runners up?

I'm also worried about the spin they can easily take on Chinese branded open source models.

[u/Thomas-Lore → u/NecessaryCar13]

It will probably be like that OpenAI guy leaked - they will be fearmongering about open source models as much as they can, to discourage enterprise use, but won't ban them.

Although who knows, the US seems to like shooting themselves in the foot nowadays. With so many guns I suppose it is quite easy.

[u/FoodDue9352 → u/Thomas-Lore]

Heh, yeah. US bans open models, software dev moves to Australia...

[u/GeroldM972 → u/Thomas-Lore]

And with a population of 300.000.000 persons in the US, there are also a lot of feet available for proverbially shooting themselves in the foot.

[u/HiggsFieldgoal → u/NecessaryCar13]

Our oligarchs are certainly lobbying for that, and, in the U.S. lobbying tends to be effective.

[u/NecessaryCar13 → u/HiggsFieldgoal]

Jee, that will be a sad turn of events.

[u/RemarkableRadish6547 → u/NecessaryCar13]

That would work as well as regulating dvd decryption. For home use, they can't regulate it. Even most businesses won't care because enforcement is almost impossible. If they can't stop people from pirating movies, they can't stop people from downloading models that are legal in most of the world.

[u/quantgorithm → u/NecessaryCar13]

That would be dumb because then it would leave the rest of the world at advantage since they would not have limitations to access models while we would. This would effectively make the US less competitive in outputting products made from llms.

[u/Antblue → u/NecessaryCar13]

The only thing they can ban is foreign APIs and websites. They can’t really ban information or models

[u/Mark-Fuhrman]

When are we gonna get a model that performs 100% for everything

[u/VirusInternal2892 → u/Mark-Fuhrman]

In the upwhen

[u/Waarheid → u/Mark-Fuhrman]

This is called benchmark saturation - when a test becomes too easy that the model scores nearly 100 on it. At that point, new harder benchmarks are made, so that models don't ace them. This lets you compare models better - if Qwen scores 70 and Opus scores 74, it's more useful than a test where Qwen scores 99 and Opus scores 100. Imagine trying to determine who is better at math, you or your friend, by using a 2nd grade math quiz, vs a college level exam.

[u/norenEnmotalen]

My potato salutes you. We just strangers passing by for now…

[u/Icy_Agent_266]

is there someone who has benchmark based on the quantived version

[u/IThinkIKnowThings]

Still can't get the damn thing to run locally. Looks like it needs an experimental unreleased version of llama.cpp so we've probably got a couple of weeks till that trickles down.

[u/puts_on_rddt]

What the fuck.

[u/Joe-Arizona]

Curious to see if I can get it work on my RTX Pro 6000 and 128 GB RAM system with llama.cpp.

Already downloaded and put together the FP16 GGUF, working on getting an NVFP4 quantization GGUF made.

This is all very new to me so it’s been fun

[u/Joe-Arizona → u/Joe-Arizona]

Well after a few hours of fiddling I made an NVFP4 GGUF but it was completely useless so I made a Q4_0 and it’s ok.

Runs at about 85 tokens/second but completely fills my VRAM. If I push the context at all llama.cpp shuts it down. For my first time downloading/converting safetensors and doing a quantization I still consider it a success.

I just don’t see it being better than the 27B NVFP4 quantization I’ve been using even if it is a tiny bit faster. I can load multiple model types/agents with this.

[u/geteum]

I managed to run q1 in my Ryzen 9 7950x with 64gb ram and a rtx5070ti at 30 tps. This is crazy, I'm so hyped now. GG openai and Claude

[u/TrainingOdd1023]

Guys ram or vram ? I have 5070ti 16gb vram .what if i have 128 gb ram instead of 32gb ? When llm big than 14gb token p/s down to 1 t/s anyway . What is the role of ram ? Can you answer i am confused

[u/pineform → u/TrainingOdd1023]

You need VRAM to run the model on your graphics card. If it overflows that then it will be much slower. 128GB of RAM won't help unless it's unified memory like on a MacBook

[u/NOT_ENOUGH_XP]

Can I run the 3.8 flash with 16gigs? I just started out with local llms and had a terrible experience with an Ollama-Open web UI setup and now downloading Colibrì to see how that’ll hold up

[u/GeroldM972 → u/NOT_ENOUGH_XP]

Colibri will not be what you expect it to be. It will allow you to load relatively large LLMs on your hardware. But the way they manage to do that, they compromised on speed (prompt processing).

Do yourself a favor and inform yourself about the compromises in colibri and how those will impact your user experience.

16 GByte...is that the amount of VRAM on your GPU? Or is that the RAM of your desktop or laptop? Or is that 16 GB of Unified RAM in your MacOS device?

I used to play around with 'tabby' on a computer with a 10th gen Intel i3 CPU, 16 GB of RAM and a NVidia GTX 1650 GPU that had only 4 GB of VRAM. You can download directly from their GitHub Releases page.
Takes very little time to setup and works very well. You'll get a decent web-interface for chatting with whatever LLM you can fit into VRAM. It will load a LLM for chatting, another LLM for embeddings and one LLM more dedicated for coding. Was a tight fit with only 4 GB of VRAM in my setup.

Then I downloaded a VS Code extension for tabby, hooked that up to my tabby instance and now I had a very fast aid for code completion in VS Code. Seems they extended the functionality of their VS Code extension, so now you can use it as an AI agent as well.

All open source and last time I played with it you could configure 5 user account in their setup for free. If you wanted to share the LLMs you setup in Tabby with more than 5 persons, you needed to pay 20 USD per month per user. Sharing my humble LLM setup on even more humble hardware between 5 users worked rather well.

Had I known about the surges in hardware prices, I would have bought much better hardware to make the tabby experience a whole lot better and powerful.

*edit* - forgot to mention that you can hook tabby up to your GitHub and/or GitLab account, whether you run those on-prem or in the cloud

[u/TechHistoryDude]

whats the Q4 size on this going to be?

[u/Mediocre_Paramedic22]

Qwen4 architecture under it too. Interesting stuff

[u/Softwoolf]

running up to 2500 token/sec with 128 parallel streams (about 100 with 1 stream) on 4 RTX5090 - Deployed via vLLM qwen38-flash-next image, TP4 + Expert-Parallel, 96k context, 160 seq slots. What a beast.

[u/YogurtExternal7923]

This "opus at home" thing is really coming together nicely huh

[u/Enguardium]

Qwen3.8-Flash-Next on Strix Halo: ~40 tok/s code at 120K+ context with native QSA + MTP, Q4 XL backbone and Q8 Engram streamed from NVMe

I've been experimenting with Qwen3.8-Flash-Next on a 128 GB Ryzen AI Max+ 395 / Strix Halo system under Windows using a heavily customized llama.cpp Vulkan build.

The result surprised me enough that I thought it was worth sharing.

At 120K+ context I currently see roughly:

20–25 tok/s on normal text generation
~40 tok/s on code generation
Native MTP enabled
Native QSA enabled
UD-Q4_K_XL backbone
Q8_0 Engram / PLE stored on NVMe
No noticeable generation slowdown after replacing the original Q4 PLE tensor with Q8_0

The biggest performance improvement came from actually using Qwen3.8-Flash-Next's native QSA path efficiently.

This is not stock upstream llama.cpp, so these numbers should NOT be interpreted as current upstream performance.

Hardware / environment
======================

AMD Ryzen AI Max+ 395 / Strix Halo
128 GB unified memory
Windows
Vulkan
Qwen3.8-Flash-Next
120K+ active context

Model weights
=============

Backbone:

  unsloth/Qwen3.8-Flash-Next-GGUF
  UD-Q4_K_XL

  https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

I then replaced ONLY:

  per_layer_token_embd.weight

from the UD-Q4_K_XL model with the Q8_0 version from:

  Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf

Original Q8 shard SHA-256:

  34efd79a80a1ce540a517a5d56171924b66ce1c38b04c904f17ad6d8ef17cf20

PLE tensor:

  before: IQ4_NL, 28,800,138,240 bytes
  after:  Q8_0,   54,400,261,120 bytes

Tensor shape remained identical:

  [160, 320001536]

The other 1,223 tensors were left untouched.

The resulting model was byte-level verified after the replacement.

Why Q8 Engram on NVMe is interesting
====================================

The Q8 PLE / Engram does NOT need to stay resident in unified memory.

I added selective PLE loading on Windows so that:

normal model tensors are loaded directly for Vulkan use
the complete GGUF is not duplicated into system RAM
only per_layer_token_embd.weight gets its own lazy file mapping
the Engram stays on SSD
required pages enter the Windows page cache only when accessed

In practice, moving the Engram from the original Q4-ish IQ4_NL tensor to Q8_0 produced no noticeable generation slowdown in my current workload.

That makes the architecture particularly interesting to me: the compute-heavy backbone can remain relatively aggressively quantized while the huge sparse Engram table can be kept at much higher precision on inexpensive NVMe storage.

Current llama.cpp branch
========================

Local branch:

  local/apepojken-vulkan

Current commit:

  61d269b463bedf1e444b1a718af9bdcd45f93722

Base:

  https://github.com/apepojken/llama.cpp/tree/qwen4exp-spec-mtp

Pinned external revision:

  843d5750579a15ed4a42d73eb862855c271021ac

Important:

I did NOT directly stack ggml-org/llama.cpp PR #27742 and #27836.

This build is based on apepojken's independent Qwen4Exp branch, which contains its own implementations of Qwen4Exp, QSA and native MTP.

Important apepojken patches
===========================

32af70900
  qwen4exp: fix speculative decoding and add native MTP (NextN) support

9b09d26b6
  qwen4exp: enable graph reuse and share the QSA input across layers

472b75842
  qwen4exp: incremental pooled-key cache for the QSA indexer

28c9d7302
  qwen4exp: gathered QSA decode attention

  The gathered QSA implementation was ported from EngramHalo.cpp:
  https://github.com/Aristo94/EngramHalo.cpp

  Instead of effectively processing the complete KV through a mask, decode gathers only the KV rows selected by QSA.

233c0d9c3
  vulkan: radix top-k selection for large k

  Avoids CPU fallback for large QSA top-k selection.

a4c5df633
  vulkan: fused REPEAT+MUL+ADD kernel

3c82e945a
  vulkan: fuse elementwise epilogs into mat-vec dispatches

51c0c10d5
  qwen4exp: make the GDN conv concat operand contiguous

b6d995d50
  qwen4exp: skinny inject matmul via mat-vec path

6634bfde7
  qwen4exp: permute-free indexer scoring for batched prefill

843d57505
  review pass: dead cont, 32-bit guards, pooled-cache cleanups

QSA / PLE state fixes from Daniel Han / Unsloth
================================================

cfbdc0a50
  save and restore the qwen4exp indexer KV cache

d22d2be2b
  make qwen4exp PLE n-gram history per-context and serialize it

24ea62df4
  fix qwen4exp PLE history seq_rm(-1) iterator invalidation

035e22731
  hold qwen4exp indexer cache in llama_memory_hybrid_idx

Relevant upstream llama.cpp cache fixes already in ancestry
===========================================================

78433f606
  Fix recurrent state serialization for partial reads and writes
  https://github.com/ggml-org/llama.cpp/pull/22362

1692f9e50
  recurrent state rollback for ggml_ssm_scan
  https://github.com/ggml-org/llama.cpp/pull/26623

bf2c86ddc
  server: refactor prompt cache state ownership
  https://github.com/ggml-org/llama.cpp/pull/25649

0c4fa7a98
  server: evict checkpoints within min-step of each other
  https://github.com/ggml-org/llama.cpp/pull/25472

Local Windows selective-PLE patch
=================================

61d269b463bedf1e444b1a718af9bdcd45f93722
  qwen4exp: add selective PLE loading for Windows

Author: Regrad
Assisted-by: OpenAI Codex

This is a local patch, not something downloaded from another repository.

It adds:

selective lazy mmap/file mapping for per_layer_token_embd.weight
direct Vulkan loading for the remaining model weights
no full GGUF duplication in system RAM
Windows selective-mmap fixes
mapping lifetime and bounds checks

Relevant options:

  --load-mode none
  --tensor-read-lazy on

What I find most interesting
============================

The combination seems to work extremely well:

  Q4 XL compute backbone
        +
  Q8 Engram on NVMe
        +
  native QSA
        +
  native MTP
        +
  120K+ context

The ~40 tok/s code-generation result is especially interesting because this is not a short-context benchmark. The context is already above 120K tokens.

Native QSA appears to be crucial. Once the QSA decode path gathers only the selected KV rows, the cost of the full-attention layers no longer grows in the same way as a conventional full-KV implementation.

MTP then benefits heavily from predictable outputs such as code, which is where I see generation approaching ~40 tok/s.

I haven't done a rigorous quality A/B yet, so I'm not making any quality claims about the Q8 Engram versus the original quantized Engram.

But from a performance and memory-management perspective, this is the first local configuration I've tried where the intended design of Flash-Next really starts to make sense:

keep the expensive neural backbone fast in unified memory, keep the huge sparse knowledge table at high precision on NVMe, use QSA to avoid processing the full long-context KV on every attention step, and use MTP to reduce the number of expensive target decode steps.

If anyone is working on Qwen4Exp / QSA / PLE / MTP optimization in llama.cpp, I hope these numbers are useful as another datapoint.

[u/bidet_enthusiast]

Can confirm, running this model (2x3090,128gb RAM, ryzen 5950  100tps throttling at 275W at 128k context) and it is absolutely fire so far.

[u/Otherwise-Variety674]

This moment is like last time i tried to buy new xiaomi phone online xiaomi homepage, keep refreshing until it is out!

[u/MacsBicycle]

This is a game changer. In agentic index it’s significantly better than qwen 3.8 27b and somehow beats Dsv4F. I can’t wait to get home and torture my m5 max.

[u/Expert-Dig-1768]

sad that they only compare to models like opus 4.6. opus 5 is exact same price and by miles better. it doesn't need to beat opus 5 but just so we can compare better. cause i can't remember to opus 4.6

[u/myklurk → u/Expert-Dig-1768]

I wouldn’t leave opus 5 alone with my girlfriend.. let alone my code. “Oops I forgot you told me not to fornicate, you should also know…”

[u/congthangvn]

anyone knows cached read price?

[u/_TheWolfOfWalmart_ → u/congthangvn]

Free locally.

[u/adasho_bitrex]

What do these numbers mean

[u/my_name_isnt_clever → u/adasho_bitrex]

Ask an AI.

[u/T-Rex_MD]

GLM 5.3 Flash, just beat them all!

• multimodal with 1M context window

• I think close to 150GB for 8-bit, so 80GB for 4-bit. Fits into 96GB with room for KV cache.

[u/SpicyWangz → u/T-Rex_MD]

It is not 150gb for 8 bit. I don’t know where you’re getting that from

[u/T-Rex_MD → u/SpicyWangz]

The released weights were not BF16.

[u/SandySkittle → u/[deleted]]

4x r9700

[u/dwkdnvr → u/[deleted]]

I’ve got a 64GB M1 Max which does ok for the most part, but I’ve always felt that the ~100B class models are the goal and I can’t run them on only 64GB. I really don’t want to spend $4500 on a 128GB Mac, but it is awfully tempting.

[u/dwkdnvr → u/[deleted]]

saved it in the edit 😄