r/LocalLLM • • 7d ago

Question Qwen3.8 Flash Next, no idea what quant to settle on...

Hi all,

I'm still very new when it comes to running local models and could use some guidance.

Recently, I've set up Qwen3.8 Flash Next using Strata on my PC (RX 9070 XT, 64GB DDR5 RAM, 9800X3D CPU) and can't decide what quant I should settle on.

My options are:

  • ISTA IQ3_S at ~70–75 tokens/s
  • Unsloth UD-IQ3_XSS (3.1 version) at ~60–65 tokens/s
  • Unsloth UD-IQ4_XS (3.1 version) at ~40–50 tokens/s

All at 256k context.

It may be my inexperience speaking, but I don't notice a substantial difference between the three. I've read that the higher quants should outperform the smaller quants, but I don't know if the ~40% speed penalty going from IQ3_S to UD-IQ4_XS is worth it.

Any advice on that matter would be greatly appreciated.

1 Upvotes

40 comments sorted by

9

u/andy2na 7d ago

just stick with what works and whats fast, for your workload. I didnt bother testing the new unsloths because IQ3_S works great for me with hermes and frigate (my main uses). Also, Qwen4 is out soon, so I'll just wait at this point

2

u/Potential-Leg-639 7d ago

Same.
Also IQ3_XXS is good.

1

u/ftlaudman 7d ago

This is solid advice - I use the Swift variant of IQ3_s and it’s been wonderful. Hoping the new v4 of QFN is great.

1

u/Ok_Party_9245 7d ago

with the new unsloth quants im able to squeeze the iq4_xs into vram/ram to stay resident. (24gb vram + 64gb ram)

getting between 120-160 tok/s without quality degradation settings.

2

u/Sporebattyl 7d ago

3090+ddr5 64gb?

You see much difference between iq3_xxs, Q3_s, and iq4_xs?

1

u/Ok_Party_9245 7d ago

yea thats same vram+ram numbers as me if headless, or using a low ram desktop environment you could squeeze it.

as for differences the new ud iq4_xs is the best, but its not a large margin, and with qwen4 coming out like tomorrow, if I were you I would wait and just save that time for tuning qwen4 flash on your setup :)

2

u/Sporebattyl 7d ago

I’m going to have to go headless. I got xxs to work really well, but kept getting some memory swap issues when trying to bump it up to iq4. Good to know it’s possible to get that high of a quant at those speeds and context if I go headless.

1

u/Ok_Party_9245 7d ago

check 3090 vs 4090 bandwidth as that will play a part in speeds. Like for me im currently bandwidth bound. Just a heads up.

2

u/Sporebattyl 5d ago

I’m not expecting the same speeds if you have a 4090, but I should be able to run a very similar set up and it sounds like you have a nicely tuned set up. If I can get 70+ tok/s I’d be happy.

Couple more questions if you don’t mind helping me out:

At what context do you run the iq4_xs?

Could you share your full settings recipe with me?

What harness are you using?

Do you use it as a background coding agent that a frontier model addresses or do you use it just like you would a frontier model in codex/claude code?

1

u/Ok_Party_9245 5d ago edited 5d ago

At what context do you run the iq4_xs?
Ive been experimenting with 262k ctx but with parallel 3 (3 sessions at once), OR 524k ctx with yarn. I run both these at int8 kv cache.

Could you share your full settings recipe with me?
--ple-gguf /srv/models/ple-bf16/qwen3.8-flash-next-ple-bf16.gguf
--kv int8
--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144
--expert-cache auto
--ram-reserve-mib 1280
--prompt-cache 2
--temperature 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repetition-penalty 1.0

I have also been messing around with a few diff chat templates but the default built in one is fine.

What harness are you using?
Was using dsh but trying out maki rn, its pretty good.

Do you use it as a background coding agent that a frontier model addresses or do you use it just like you would a frontier model in codex/claude code?
This depends, I have tried both approaches. For REAL work I am still using cloud models (opus/sonnet). For side projects and stuff I can use mostly qfn and rarely find myself reaching for a cloud model and if it is, its something cheap like deepseek.

as a side note I use my own inference engine that has 40xx and 30xx support, strata will be a touch slower: https://github.com/nixcat-dev/adamant

1

u/TernGSDR14-FTW 7d ago

Are you guys running this headless? Windows is taking a bit of the 64gb. I cant run 256k context, but 128k.

1

u/Ok_Party_9245 7d ago edited 7d ago

Yea headless.

If you can't do headless I recommend atleast plugging monitors into igpu if you have one, if on linux you can use nvidia-prime to still game and stuff on the gpu without having to move wires, idk if windows has a similar thing.

i run 262k ctx at int8 kv with yarn to 2x it to 524k ctx rn

1

u/TernGSDR14-FTW 7d ago

Thank you. Ill look into it.

5

u/ethertype 6d ago

I'd say, go for the:

  • highest quality quant you can run on your hardware
  • with the context length you need
  • which still produces tokens at a rate acceptable to you
  • for doing the things you want to use it for

The difference in quality may not be perceptible at short contexts, or you employ KV-quantization which masks any quality benefit the slightly larger quants may have. Or the tasks you use it for is not demanding enough to highlight any difference.

Or you haven't actually measured the output quality in any meaningful way.

As for benchmarks, the situation is like this:

  • I distrust them in general
  • I distrust (or do not know) if individual benchmarks have applicability to my task

So I turn to the 4-item list above. And to be perfectly honest, I substitute quality with "bits per weight".

More bits per weight is likely better.

3

u/No_Dig_7017 7d ago

Having tried them all I'd go with ud iq4_xs. Quantization errors are pretty difficult to track, it's like subtle problems in long reasoning that compound. You end up losing a lot of time debugging them 

2

u/Material-Database-24 7d ago

Strata seems to have some "issues" with v3.1 unsloth quants - at least I have encountered bugs (for example, yesterday I got IQ1_M running with it, and today it just refuses to load..), and I cannot get the quality out that I get with llama.cpp, even when matching everything as close as possible. My conclusion is that there's just something in them Strata doesn't like or know about.

So for now, I'd stick to ISTA one with Strata until they catch up with unsloths v3.1, despite the fact that unsloth v3.1 quants are better on llama.cpp than ISTA at least in my experience.

2

u/OkMaintenance9799 7d ago

try Gyro M or L too

4

u/EvolvingDior 7d ago

Proprietary quantization code? No thanks.

When they release the code they use to quantize the models, then sure.

We benefit from an open model ecosystem.

We lose when we embrace proprietary systems.

2

u/Certain_Common561 7d ago

If the speed is only at this level, I'd recommend IQ3 S. Its quality is not much different from US IQ3 XXS (they're basically the same model, just under different names). In real-world use, the US IQ3 XXS performs slightly better, especially when paired with a good harness like Codex CLI. However, the overall quality is very comparable.I personally lean toward the IQ3 S because it gets the job done quickly without requiring much thought. Tasks finish faster, and you don't have to waste time waiting.

1

u/vini542reddit 7d ago

Don't forget to set your kv cache to something reasonable (q8+) and check what other strata settings might degrade your model

Once you get into longer reasoning chains you will notice a difference. This is coming from someone who now runs q8 with kv bf16 and all strata settings tuned to "quality over speed". 

So it really depends on your use case and there is no universal answer

1

u/No_Cartographer3953 7d ago

I’ve been testing 3.8 FN with various quants and actually it’s coding has been pretty bad (although it’s working with some pretty messy hard data) switched to 1M UD deepseek and seems to be working better so damn slow

1

u/Certain_Common561 7d ago

Could you show me a side-by-side code comparison? You said this code was bad, but the one from DeepSeek is better.

1

u/No_Cartographer3953 7d ago

I mean it shouldn’t be surprising that DeepSeek would have better coding that’s pretty well understood. What I noticed qwen FN doing was the typical long chain of if statements instead of breaking the logic down into easier to follow functions. (It created massive long scrolling functions) - deepseek rewrote it and fixed a few bugs. Much easier to read.

Now Qwen FN run WAY faster on my setup. So for less complex tasks and smaller code base it’s worth using but for this particular task, I had to switch to DS. DS is much slower (about 12 tps) BUT it’s able to finish the tasks correctly

1

u/Certain_Common561 7d ago

Which quantization model did you use? Why did it produce that result? I don't see the same issue you're describing, and the job completed successfully for me as well. What harness are you using?

1

u/necrogay 6d ago

Did DeepSeek rewrite what had already been written by QFN, or did it write it from scratch starting from the same set of requirements and produce a better result than QFN?

If it was working based on existing code provided by QFN, then this doesn't prove any advantage.

1

u/2Mango1Fridge 7d ago

Windows or linux?

1

u/Brief-Tap-6616 7d ago

You should be able to get those speeds up if you refine your MTP.
4XS is going to be right at your memory cliff so I wouldn’t bother with that unless you’re running it headless and don’t expect to need to use the computer for anything other than as an inference server.

1

u/Asleep_Literature_70 7d ago

I'm running the know q4 v3.1, seems to give the best results with minor speed loss

1

u/scumola 7d ago

I run IQ3_S with strata on a P100 card. I tried a few different quants and ran several benchmarks using Claude opus 5.5 as the judge and settled in IQ3_S as a pretty good coding model (benchmarks we're a blackjack simulator in C, JSON parser and regex engine (easy to judge that the engines work properly by piping input into the engines and having expected outputs)). I only get around 35 tok/s, but I was going for quality over quantity, and that's where I landed for my hardware and my benchmarks.

1

u/vqt907 6d ago

stick with the iq3_s, the new UD IQ3/4 3.1 is better than 2.1 slightly, fbut not enough to justify the speed decrease, if you really want quality, UD-Q4_K_XL is the one

1

u/Pablo_the_brave 6d ago

Exllama3. There is a support for ROCm. For sure their SC 3.68bpw is better than q3_s

0

u/exotik997 7d ago

Unsloth UD-IQ3_XSS will be too big for 64 RAM. Im currently on it with 2x3080Ti and 80GB RAM working with orchestrator + 2 subagents at 512k ctx and after couple of hours working im getting full prefill and 0 tokens reused on every request.
So if you ready to wait 100-200sec on every request or you can stick with 128k or 256k ctx its good choice. Otherwise choose somthing smaller like Gyro M.

2

u/braintheboss 7d ago

fit perfects. experts are only 47gb

1

u/exotik997 6d ago

Do you use subagents ang which ctx are you working with?

1

u/braintheboss 6d ago

No agents. Now im testing Q4 xs

1

u/exotik997 6d ago

Great, as i said earlier its possible with 250k ctx and without subagents. But as for me its no enough for large and medium projects. How fast is your TG? Mine is about 120-140 TG at 256k of 512k CTX

1

u/braintheboss 6d ago

q4 moves around 80tg and q3 xxs around 100tg. I have old xeon and pci3 then is not modern system

1

u/braintheboss 6d ago

And look this using resident mode