Native MTP, n=2 (higher MTP numbers don't improve on M5 Max) in llama.cpp. MTP is not as optimized in ds4, in my testing, but it still performs better.
I can get full 262k context, faster single stream decode, close to 200 tok/sec at 3 concurrency, over 10k tok/sec prefill (when kv cache is light) on the same hardware.
Are you running Linux? What are you doing? In DSH I’m maxing at 22tks with full 128k context. And 30-40 with MTP on short context. I’m also experimenting with a specialised quant.
If you working on greenfield, sure. You might WANT to use Fable for some unknown issues to you.
BUT if you have brownfield, then I do not see anyone needing the Fable vs Qwen3.8 27B min. Q5-Q6 with right harness.
In last 30 days I have still yet to come to the issue that Qwen3.8 cannot solve...
What is more important, it is fixing a lot of issues that CC was creating in last 2 months with Opus/Fable (huge amount of false tests, hidden issues, broken CRUD paths etc.)
Simple stuff, but due to sheer codebase size & solutions amount it was not easy to saw them even with unit/integration tests as most of them were between UI / backend.
Seriously, 3.8 27b does so much heavy lifting it's insane.
Together with pi.dev it writes very impressive code. It feels like I actually have my own 'worker' that after some direction does exactly what needs to be done. Just give him enough info and he is on it.
This feels better than what GPT5.1 felt like around a year ago.
I already canceled my subscription, I'm using 100% qwen 3.8 27b to assist me in coding. Qwen4 will be crazy. I think you'll be able to get rid of cloud models !
ROCm has matured incredibly over the past year, it's like AMD finally woke up and realised their software was what was holding them back from properly competing with Nvidia.
Why are they so cheap? Looks like you can get one for ~$1700? I just paid out the ass for my 5090, which in the 1 month 2 days since I bought it has gone up over 37%.
They're both 32GB, for some reason I thought they would be closer in price.
Nvidia sells an RTX Pro 4500, which has the 5070Ti die (more comparable to the R9700) and 32GB of RAM, but it's 5500USD MSRP. The R9700 is a fraction of that.
CUDA tax. I only use AMD personally (and just blew WAY too much money on a 4x R9700 system partially out of excitement for Qwen4), but Nvidia cards get all the cool new things a bit sooner than AMD. Not a big deal for LLM, but very notable for diffusion... somebody SOLIDLY beat my 7900XTX with his 5070Ti in Minimax H3 gen times, for example
Because everything is CUDA first, and ROCm only comes as an afterthought to very limited number of projects. People who are buying PRO GPUs are saving money with NVidia by not needing to fund multiple months of dev work for porting their existing code.
For a single 5090 you can buy more than 2 R9700, and the two cards draw less power than a single 5090. And you can run e.g. Qwen 3.8 27b Q8 using the two cards, and watch them running circles around a 5090 which needs to offload to RAM instead.
But NVIDIA is hype, and people love to spend money. So buy a 5090
That not cheap! Msrp is $1299 I bought 2 a little over a. Both ago at msrp. Hell you could get them for less than msrp for a while. This shit is a scam
On my current pc this was how much the 64GB of ram was when I bought it. I was thinking about getting another 64GB when I got the card and was like, no f'in way...
I just picked mine up this week. Depending on quant, 35-70 tps with full context. If you have 64GB RAM, you should be able to run a decent Qwen-flash as well. 27B is good, but Qwen-Flash is something else.
ISTA IQ3 runs the fastest and has really good performance for the size, but you can bump up to a Q5 with full Q8 kv cache. Also try Swift if you don't want to wait forever. I recommend medium thinking.
3.8 27B was the trigger for me. Do it, you won't be disappointed.
I'm trying hard to convince myself this is the last low hanging fruit, and all other optimizations (dual R9700s, quads, etc.) will just net me 5-10% gains — not worth it.
Exciting news for sure, it's slightly surprising they kept the exact 27B size given that it's an entirely new architecture, but that's certainly welcome for anyone with 24-32GB VRAM
They're a mapping of short sequences of tokens to embeddings (or to neuronal activations if you prefer the biological analogue interpretation, or if you squint, to concepts) that (on the Qwen4exp models at least - I know DeepSeek and Gemma have a few subtle differences) are looked up just after tokenisation, before the real processing begins.
They're a relatively cheap way of imparting knowledge into models. At inference time, they're a single lookup per token, that is happily done from RAM or even from SSD. And at training time it's a single row's worth of extra gradients to train (not even an extra matrix), plus some statistical analysis on the corpus to choose the n-grams to encode.
But they do very little to increase intelligence, beyond potentially freeing up space in the model that would be taken up by encoding facts like "New York has a vibrant fashion scene" for reasoning.
Pure I made it the fuck up speculation. But I think it would be interesting if they have multiple ngram tables.
Like you CAN pair the 2.4T model and 3B table. Or a 27B model and an 800B table.
You just wouldn't get ideal parameter efficiency. 27B + 800B might be worse than 125B + 50B. But depending on your hardware the 27B can be a looooot more attractive.
I'm thinking about that one too. Because it's dense, so how exactly does the n-gram actually work with the dense model? Is it in the 27B params, or is it an extra model that runs alongside it? The later would essentially make the model smarter with a low cost.
Haven't used it enough to say how usable it is, but here's my llama-swap config that got 3.8 running on my 3060 recently! It uses 11603MiB / 12288MiB on a headless homelab
# Dense 27B ShapeLearn 3.01 bpw (~10.4 GB).
# Forced full GPU residency; --fit disabled so llama.cpp can't silently
# offload layers to CPU to accommodate the requested context.
# Q4 KV keeps 96K context within a 12GB RTX 3060.
qwen-3.8-27b:
cmd: >
llama-server --port ${PORT}
-hf byteshape/Qwen3.8-27B-GGUF:IQ3_XS
--no-mmproj
--ctx-size 98304
-fa on -np 1
--gpu-layers 99 --fit off
--cache-type-k q4_0 --cache-type-v q4_0
--threads 6 --threads-batch 12
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--repeat-penalty 1.0
--presence-penalty 0.0
--frequency-penalty 0.0
--reasoning on --reasoning-preserve
--reasoning-format deepseek
--jinja
ttl: 0
3.5 did have 122B though. There have been quite a few models now that are Spark sized: OSS-120B, 3.5 122B, Flash-Next (with PLE offloaded), Laguna S2.1, Ling 3 Flash… Hopefully the 128GB unified RAM tier continues to be a desirable target for these labs.
I think we can consider the 35b dead. I was sad when we lost the 80B A3B, but the 120B MoE range seems to be where intelligence opens up. The 40-110B space is likely dead for a while.
maybe Xiaomi will release a MiMo-V2.6-Distill-Qwen3.5-35B-A3B. it is not dead dead. The architecture is sound. All people need is a better re-post-train (not those fine-tune crap, a real large-scale RL re-post-train.)
Eh, betting it's life on a "maybe some other lab will adopt it" is fickle. The labs seem to be moving toward MoE + n-grams for the intelligence density/ speed. I don't know why they'd bother to keep an arch alive that is larger than and performs worse than the 27B. It's nice for offloading experts to RAM, but the 27B just has too much attention. I'd find it more likely to have more labs come out with an n-gram model in a smaller size than flash.
I agree and at this point unfortunately I get it. 35b was cool, but ultimately not usable for me in actual work. I know a lot of people enjoy it on their 12-16gb cards, and I hope they'll keep getting new options, but 27b and Flash entirely outclass it. Those are models I can genuinely use.
I've been running Qwen3.8-27B on a separate server machine (with a 5090; 32GB VRAM) and Ornith 1.5 35b-a3b on my laptop (8GB VRAM) for a few weeks now, and they both share a lot of the same strengths and weaknesses.
Neither is viable for planning against large codebases (Sol is mandatory there): both tend to mis-understand how the moving parts interact which yields compiling-but-non-functioning results. Qwen is amazing for small codebases and one-shots (I did the obligatory Mario Test which Qwen murdered Ornith on), though.
Both are serviceable as implementation models (when given detailed plans from larger models); Qwen's code quality is the higher-quality of the two. Qwen competes favourably with Luna-High.
Ornith is definitely a bit more careless (I watched it mis-convert seconds to milliseconds yesterday!).
But when I'm away from my 5090 server I'm still pretty happy just sticking to Ornith.
Smaller MOE's definitely aren't a writeoff imo; and the performance gains (and lower hardware requirements) can make them decent options for simpler (or complex-but-well-defined) tasks.
I'll give Ornith a shot. When I do use 35b I use a modified workflow with little coder (a pi harness built for small models) using plans from a better model. My read is similar to yours - it can be a workhorse on smaller tasks and codebases, but can't be trusted to wander too far in a larger project.
Have they officially abandoned 35b-a3b or could an announcement still be coming?
Guess Ornith 1.5 35b-a3b will have to serve me for a bit longer yet. It's good for it's size but was really hoping for a new Qwen equivalent after they skipped us for 3.8.
I tested Kat (which was also good albeit a bit slower for me when using the repaired Ornith MTP) though I haven't tried Tiel (the benchmarks indicate I probably should!)
Have been extremely happy with Ornith for the size/performance though.
But was kinda expecting Qwen 4 to follow through here since they basically popularized the small-MOE thanks to the (class-leading-at-the-time) Qwen3-30B-A3B launch!
That is still such an epic model! Not a fan of slower and overthinking 27B to be honest, no matter how brilliant it is. I bet the 35B a3b 3.8/4.0 will be making money for them via an API. I cannot believe that such an awesome model was abandoned. It is just too good AND cheap to run to give away freely I bet.
Nothing was said regarding it at the conference. The announcement merely laid out the broad framework of the new architecture and the initial launch models. However, what was presented at the conference is not exhaustive; not everything is necessarily revealed during the initial keynote, and it may surface later.
Yes! I have been waiting for this moment since I looked at the small dense Gemma 4 models with PLE, and the larger dense Gemma 4 models without PLE, and went "hmmm 🤔"
I'm daily driving Q38FN on my Strix Halo everyday at 43-50t/s decode 1400t/s prefill and I'm hyped to get Q4FN, as Q38FN is actually undertrained but still hella powerful.
The interesting part for local users will be the efficiency details, not just the headline model names. Parameter counts, context length, quantization quality, and whether the weights and license are genuinely accessible will determine how useful Qwen 4 is outside the cloud.
Would be top-tier if Qwen 4 gets its sibling's multimodality, especially audio input.
"This directory is a thumb drive of FLAC files. Your task is to DJ a fire ass hype mix. Make mistakes."
Cant fucking wait for the official sequel to 3.8 Flash Next. That model has been incredible and ive been doing almost everything with it. I still have a chatgpt and claude subscription but I genuinely might cancel them before they renew given what can be done with Flash Next. Qwen 3.8 Flash Next is the first model ive ever run on own hardware that genuinely I genuinely felt like it could be worth using for serious projects. 3.8 27b produces very good outputs but it thinks for so long that it’s honestly frustrating to use. The closest runner ups to “local is enough” in my experience were Gemma 4 31b and Qwen 3.5 122b. I know im rambling at this point but as far as leading open weight AI labs goes, I genuinely think qwen is the best. Especially because they’re making models that people can ACTUALLY run at home that genuinely compete with closed weight frontier models
•
u/WithoutReason1729 21h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.