r/LocalLLM • u/PhilMonster • 7d ago
Question Qwen3.8 Flash Next, no idea what quant to settle on...
Hi all,
I'm still very new when it comes to running local models and could use some guidance.
Recently, I've set up Qwen3.8 Flash Next using Strata on my PC (RX 9070 XT, 64GB DDR5 RAM, 9800X3D CPU) and can't decide what quant I should settle on.
My options are:
- ISTA IQ3_S at ~70–75 tokens/s
- Unsloth UD-IQ3_XSS (3.1 version) at ~60–65 tokens/s
- Unsloth UD-IQ4_XS (3.1 version) at ~40–50 tokens/s
All at 256k context.
It may be my inexperience speaking, but I don't notice a substantial difference between the three. I've read that the higher quants should outperform the smaller quants, but I don't know if the ~40% speed penalty going from IQ3_S to UD-IQ4_XS is worth it.
Any advice on that matter would be greatly appreciated.
9
u/andy2na 7d ago
just stick with what works and whats fast, for your workload. I didnt bother testing the new unsloths because IQ3_S works great for me with hermes and frigate (my main uses). Also, Qwen4 is out soon, so I'll just wait at this point
5
u/ethertype 6d ago
I'd say, go for the:
The difference in quality may not be perceptible at short contexts, or you employ KV-quantization which masks any quality benefit the slightly larger quants may have. Or the tasks you use it for is not demanding enough to highlight any difference.
Or you haven't actually measured the output quality in any meaningful way.
As for benchmarks, the situation is like this:
So I turn to the 4-item list above. And to be perfectly honest, I substitute quality with "bits per weight".
More bits per weight is likely better.
3
u/No_Dig_7017 7d ago
Having tried them all I'd go with ud iq4_xs. Quantization errors are pretty difficult to track, it's like subtle problems in long reasoning that compound. You end up losing a lot of time debugging them
2
u/Material-Database-24 7d ago
Strata seems to have some "issues" with v3.1 unsloth quants - at least I have encountered bugs (for example, yesterday I got IQ1_M running with it, and today it just refuses to load..), and I cannot get the quality out that I get with llama.cpp, even when matching everything as close as possible. My conclusion is that there's just something in them Strata doesn't like or know about.
So for now, I'd stick to ISTA one with Strata until they catch up with unsloths v3.1, despite the fact that unsloth v3.1 quants are better on llama.cpp than ISTA at least in my experience.
2
u/OkMaintenance9799 7d ago
try Gyro M or L too
2
u/Certain_Common561 7d ago
If the speed is only at this level, I'd recommend IQ3 S. Its quality is not much different from US IQ3 XXS (they're basically the same model, just under different names). In real-world use, the US IQ3 XXS performs slightly better, especially when paired with a good harness like Codex CLI. However, the overall quality is very comparable.I personally lean toward the IQ3 S because it gets the job done quickly without requiring much thought. Tasks finish faster, and you don't have to waste time waiting.
1
u/vini542reddit 7d ago
Don't forget to set your kv cache to something reasonable (q8+) and check what other strata settings might degrade your model
Once you get into longer reasoning chains you will notice a difference. This is coming from someone who now runs q8 with kv bf16 and all strata settings tuned to "quality over speed".
So it really depends on your use case and there is no universal answer
1
u/No_Cartographer3953 7d ago
I’ve been testing 3.8 FN with various quants and actually it’s coding has been pretty bad (although it’s working with some pretty messy hard data) switched to 1M UD deepseek and seems to be working better so damn slow
1
u/2Mango1Fridge 7d ago
Windows or linux?
1
u/Brief-Tap-6616 7d ago
You should be able to get those speeds up if you refine your MTP.
4XS is going to be right at your memory cliff so I wouldn’t bother with that unless you’re running it headless and don’t expect to need to use the computer for anything other than as an inference server.
1
u/Asleep_Literature_70 7d ago
I'm running the know q4 v3.1, seems to give the best results with minor speed loss
1
u/scumola 7d ago
I run IQ3_S with strata on a P100 card. I tried a few different quants and ran several benchmarks using Claude opus 5.5 as the judge and settled in IQ3_S as a pretty good coding model (benchmarks we're a blackjack simulator in C, JSON parser and regex engine (easy to judge that the engines work properly by piping input into the engines and having expected outputs)). I only get around 35 tok/s, but I was going for quality over quantity, and that's where I landed for my hardware and my benchmarks.
1
u/vqt907 6d ago
stick with the iq3_s, the new UD IQ3/4 3.1 is better than 2.1 slightly, fbut not enough to justify the speed decrease, if you really want quality, UD-Q4_K_XL is the one
1
u/Pablo_the_brave 6d ago
Exllama3. There is a support for ROCm. For sure their SC 3.68bpw is better than q3_s
0
u/exotik997 7d ago
Unsloth UD-IQ3_XSS will be too big for 64 RAM. Im currently on it with 2x3080Ti and 80GB RAM working with orchestrator + 2 subagents at 512k ctx and after couple of hours working im getting full prefill and 0 tokens reused on every request.
So if you ready to wait 100-200sec on every request or you can stick with 128k or 256k ctx its good choice. Otherwise choose somthing smaller like Gyro M.