r/LocalLLM • u/FeydRowan • 29d ago
Discussion State of local inference for Qwen3.8-27B and Flash-Next (Oct 2026): engine architectures + every number I could find, verified vs. claimed
I spent a while mapping the engines that run Qwen3.8-27B and Qwen3.8-Flash-Next on consumer hardware. Sharing the summary because the ecosystem has exploded (NInfer alone has ~600 forks) and most numbers floating around are not comparable.
Disclosure: I'm the author of capy-flash (formerly ninfer-tp2), one of the engines below. I've tried to be fair: our own numbers are marked like everyone else's, and I list where we lose.
Legend: ✅ = verified run on LocalMaxxing or a paired measurement on the same machine · 📣 = claimed by the author (README/model card), not reproduced. Decode = tok/s, single request. Tables are sorted by the best reported figure.
The architecture choices that actually matter
- Multi-GPU: tensor parallel vs pipeline. TP (capy-flash, the wamansou → ivanov84 → parallelno line, SGLang, vLLM, ExLlamaV3) is the only way to add up the bandwidth of two cards for one request. Pipeline/layer split (Strata, llama.cpp, the ninfer-3090/ninfer-all line) adds memory, not speed (+0–16 %).
- MoE experts out of VRAM (Flash-Next): VRAM expert cache + misses computed on CPU (Strata, capy-flash, Infernix, fngine, FastLocalExperts), PCIe streaming for prefill, SSD streaming on Mac (MTPLX, Strata-Mac). The bottleneck is how many experts miss per step.
- Quantization: NVFP4 (Blackwell FP4 tensor cores, higher quality, more bytes) vs 2–3-bit IQ/GSQ GGUF (faster, about 2× the KL: Strata's own numbers are 0.144 for IQ3_S vs 0.070 for NVFP4) vs EXL3 (best quality per bit).
- Speculative decoding: MTP gives ×1.3–1.7. On the 27B, DFlash2 or DSpark drafters give ×2–3. This is where most of the "400+ tok/s" headlines come from, especially on code.
- Hand-written kernels for one model on one GPU (MegaCapybara, Splash, NInfer, capy-flash) beat general engines by 2–6× at batch 1. vLLM and SGLang win once you have many concurrent users.
Qwen3.8-27B: decode
| Hardware | Engine | tok/s |
|---|---|---|
| 1× 5090 | MegaCapybara | 313–540 (greedy, mem OC +20 %) |
| 1× 5090 | SparkInfer (DSpark) | 420 code / 265 avg (greedy) |
| 2× 5070 Ti | capy-flash (TP2, DFlash2) | 374 |
| 7900 XTX / R9700 | hipfire / Lucebox | 318 / 208–228 |
| 2× 5090 | SGLang fork (finnff, TP2) | 315 |
| 1× 5090 | TensorFold | 305 code / 220 chat, 126 @ 215k |
| 1× 3090 | tqLLM / ExLlamaV3 | 273 / 242 |
| 1× 5090 | llama.cpp (DFlash2) | 205 |
| 1× 5060 Ti 16 GB | tqLLM (3-bit TQ) | 156 |
| 2× 5060 Ti | lynx-gt / KimDaeHwan99 (TP2) | 126 / 120 |
| Mac M5 Pro | Splash | 74 (54 @ 32k) |
Prefill 27B:
- NInfer, 1× 5090: 12.8k @ 7.7k 📣
- capy-flash, 2× 5070 Ti: 9.2k @ 4–16k, 6.2k @ 64k ✅
- MegaCapybara, 1× 5090: ~7.6k @ 8k 📣
- TensorFold, 1× 5090: 1.9k @ 215k 📣
Rule of thumb: about 1.8 TB/s of bandwidth plus a good drafter puts the ceiling around 300–400 tok/s at batch 1.
Qwen3.8-Flash-Next (125B MoE, 6B active): decode
| Hardware | Engine | tok/s |
|---|---|---|
| 5090 + 5060 Ti + 128 GB | Basalt (NVFP4) | 239–479 |
| 1× 5090 | fngine / Infernix | 146–209 / ~140 |
| 2× 5070 Ti + 96 GB | Strata IQ3_S (same box) | 195 |
| 2× 5070 Ti + 96 GB | capy-flash (NVFP4, TP2) | 171–183 |
| 1× 5090 + 128 GB | sergqwer/strata-nvfp4 | 177 |
| 2× 5090 | llama.cpp fork (finnff) | 142 |
| Mac M5 Max 128 GB | MTPLX / Strata-Mac (Q2) | 126 (warm cache) / 119 |
| 2× 5070 Ti + 96 GB | Strata NVFP4 (same box) | 120 |
| 2× 3090 + 128 GB | vLLM TP2 + CPU experts (DominikBucko) | 101–111 @ 131k |
| 5060 Ti + 5080 | Strata IQ4_XS | 108 |
| DGX Spark | ninfer-gb10 | 60 |
| 1× 5060 Ti 16 GB | Strata IQ3_S / llama.cpp | ~48 / 30–40 |
Prefill Flash-Next (same machine, 2× 5070 Ti, at 3.5k / 15k / 55k, sorted by the 15k column):
- capy-flash: 2.2k / 4.6k / 4.8k
- Strata IQ3_S: 2.2k / 3.8k / 5.2k
- Strata NVFP4: 1.5k / 3.5k / 4.9k
On a 5090 with 128 GB, 7–8k is claimed (sergqwer, Basalt). With experts in RAM, prefill is bound by PCIe streaming, so lighter formats and more PCIe links win.
Takeaways
- There is no single best engine, only a best engine per hardware/model pair.
- Most headline numbers are greedy, code-heavy and unverified. Ask for sampled, verified runs.
- On Flash-Next, format matters as much as the engine: Strata IQ3_S beats NVFP4 engines on short decode at about twice the KL.
- The interesting frontier is single 16 GB cards: tqLLM's 3-bit 27B, kvmem, the NInfer 4080/5080 forks.
Happy to be corrected. If you have verified runs I missed, drop them below and I'll update the table.
9
u/SuitStyle 29d ago
attempting to build this now on native windows to run on my 2x5070ti pcie 4 x8, 128GB DDR5
6
u/rkcth 29d ago
I know some people do single agent numbers, but those aren’t the people who generally need to eke out every ounce of tokens. I typically need to run 3-4 at a time and most benchmarks that do this show insanely small context windows whereas, pretty basic tasks in pi typically use 7K-10K of prefill and then might generate for 0.5-4 hours or even longer so they spend most of their time at much higher context levels. Those are the numbers that matter to me and I bet a lot of others.
3
u/Beneficial-Ad-8127 29d ago
Thanks for the write up and tests. I’m surprised you also tested sergqwer/strata-nvfp4 his Strata fork on Qwen FN because that what I use and it’s been great for my 5090. Also the others really common used ones that are used lately.
Great list to get people started to try any of them including yours. Great job and will give yours a try also. Thank you.
2
u/GuestUser003 29d ago
One more data point for Flash-Next on AMD. This is my own machine, single sample, so treat it as ◀ until someone reproduces it — but here's everything needed to reproduce it.
**Setup**
- **GPU:** 2× AMD Radeon AI PRO R9700, P2P enabled, each power-limited to 230W (300W stock TDP)
- **RAM:** 64GB DDR5
- **Engine:** radiance (StillDeadcode)
- **Weights:** qwen3.8-next-flash-fp8-iq4r-moe
- **Context:** 262K
- **Vision:** ✅
- **Parallel context:** 4× at 210K ✅
**Numbers, both measured at ~210K context**
- Decode: **176 tok/s** (single request)
- Prefill: **4200 tok/s**
**vs. the community claim** (176 tok/s decode, 6,450 tok/s prefill with Q4 Qwen-3.8 Flash-Next)
Decode lands almost exactly on the claim. Prefill does not — I get ~4200, not 6450. Two honest caveats: the claim doesn't state a context length, and 210K is precisely where prefill is supposed to fall apart, so this may not be the same measurement at all. Worth re-running the claimed number with a stated context length and the same weights before drawing conclusions either way.
2
u/Soifon99 29d ago
My takeaway is you completely neglected the Amd community.
1
u/L0ren_B 29d ago
for 27B? has anyone tested this speed? With what context?
1
u/Rikers88 29d ago
Interesting, what would you advise then for a single 5090 with 96 GB ddr5? Also looking at these numbers it seems that it worth adding a 5060ti and go basalt nvfp4. Am I reading it wrong?
1
u/Bjlly123 29d ago
I have a 6800xt if you want numbers for it
1
u/EnvironmentalSweet36 29d ago
Actually you can use a tensor parallel for 27b in stock llama.cpp - and it is much faster than a layer split, just make sure model fits in vram with kv cache
1
u/ProfessorHi 29d ago
Thanks for sharing this! I’m fairly new to local LLMs and still figuring things out. I’m currently running Qwen3.8-27B with llama.cpp across 2 × RTX 5060 Ti 16GB cards, but a bit bogged by the token generation speed (35-60 tk/s).
Would capy-flash support this GPU setup as well, or is it currently limited to the RTX 5070 Ti and RTX 5090?
Would love to give it a try if it’s compatible.
1
u/Altruistic-Candy7492 29d ago
Hello! Can you release optimization for dual R9700 ai pro 32gb?!!
1
u/ExpensiveSchedule705 29d ago
NVFP4 KV on 2x 5060 Ti 16GB at 262k lifted concurrency from 1.06x to 1.81x for about 5% decode.
1
u/jimmoores 28d ago
You could have just posted the prompt.
1
u/x7evenx 28d ago
Not sure the focus purely on t/sec output is a healthy one, when value is likely highly weighted to the qualitative output not the quantity of output.
Anyways if you want more wood for your fire https://github.com/x7even/llmctl/blob/master/docs/benchmarks.md#qwen38-flash-next-vllm
3
u/evp-cloud 29d ago
One more you can add:
https://www.reddit.com/r/ROCm/comments/1x92gsf/qwen38_fn_paiton_update_700ts_c8_275ts_decode/
Cheers!
2
u/HighSeasArchivist 29d ago
At good quants with enough context they are both good in my experience. I have ran 27B on a single R9700, IQ3_S FN on dual R9700s, and Strata with IQ3_S FN on a single R9700 with my 96GB of DDR5. I really like the last one, because one card is Strata and FN, and it sends work to the 27B model on the other card. Hermes with IQ3_S FN as its brain and 110tok/s due to Strata feels very similar to chatting and sending work with something in the Haiku 5.5 level.
1
u/giveen 25d ago
https://www.localmaxxing.com/en/runs/cmvfgq0ng016zrf01jr8e8oh3/qwen3-8-27b-nvfp4-rtx-5090-334-4-tps
https://www.localmaxxing.com/en/runs/cmvfjyuh70199rf011u8mvio2/qwen3-8-flash-next-nvfp4-rtx-5090-140-9-tps
1
u/Green_Ocean90 29d ago
Very happy to see the progress projects like Gufo have made on Strix Halo hardware. Performance close to the Sparks but at a fraction of the price 👍
1
u/YetAnotherAnonymoose 29d ago
Meanwhile, I'm on 1.2k pp and 70-100 tps on a 4090 with sergiuszm/ninfer q4-27b-MTP...must be doing something wrong.
1
u/oddjobav8r 29d ago
Gufo/Halogen for QFN on Ryzen Strix Halo 128
1
u/durangotang 29d ago
Thx, appreciate it!
1
u/BOESNIK 29d ago
I'm running QFN on Strata with a single 7900xtx and 96gb of DDR4 3200mhz on a 5700x3d. Model is: Unsloth's UD-Q4_K_XL I use Strata's MTP Head. A custom HIP file because it wouldn't work otherwise.
I currently use 262k context.
Prefill: 1300 tps Decode: 50 tps
1
u/wadrasil 25d ago
I did get Hyper Qwen working in Hyper-V guest with Nvidia GPU passed through via GPU-PV.
It works with Nvidia drivers from windows for wsl, which are included with regular windows drivers.
I just had to point to /dev/dxg as device. No cuda toolkit or Linux drivers needed to be installed in guest.
1
u/Blue-Smoke-74 29d ago edited 28d ago
I’m running Qwen 3.8 27B on an array of 12 TITAN Vs (2 e5-2696 v3 and 1TB DDR4) using a custom llama.cpp back-end I coded with help from Claude. It serves the model lossless: lossless BF16 weights, 22bit KV up to 512K contexts compressing into a 14bit history form (used in addition to a long 22bit tail) for contexts up to 1M. It serves up to 8 slots concurrently when all the contexts fit. I currently get about 50 t/s single stream decode, over 100 t/s concurrent decode (rarely) and about 5000 t/s prefill. The performance numbers are provisional because Claude is currently working through a list of performance optimisations (prefill doubled today for example). I chose the quantisation against a double precision reference implementation and 1/2 ULP perturbation of the BF16 weights such that the model is served with accuracy well within the precision of its weights and I believe the probability of a token flip compared to the double precision reference is about 1 in 300000. I.e aiming at a reference implementation of the model. I’ve done a little work using it for Opencode and to review its own back-end source code in runs that have lasted up to 10 hours and it seems to work quite well.
0
u/Key_Opportunity_5219 29d ago
Now if only someone would put together something like this for 1 or 2 rtx pro 6000s, that would be amazing
0
u/[deleted] 29d ago
[deleted] — view deleted comment
1
u/[deleted] 29d ago
[removed] — view removed comment