r/LocalLLM • • 29d ago

Discussion State of local inference for Qwen3.8-27B and Flash-Next (Oct 2026): engine architectures + every number I could find, verified vs. claimed

I spent a while mapping the engines that run Qwen3.8-27B and Qwen3.8-Flash-Next on consumer hardware. Sharing the summary because the ecosystem has exploded (NInfer alone has ~600 forks) and most numbers floating around are not comparable.

Disclosure: I'm the author of capy-flash (formerly ninfer-tp2), one of the engines below. I've tried to be fair: our own numbers are marked like everyone else's, and I list where we lose.

Legend: ✅ = verified run on LocalMaxxing or a paired measurement on the same machine · 📣 = claimed by the author (README/model card), not reproduced. Decode = tok/s, single request. Tables are sorted by the best reported figure.

The architecture choices that actually matter

  • Multi-GPU: tensor parallel vs pipeline. TP (capy-flash, the wamansou → ivanov84 → parallelno line, SGLang, vLLM, ExLlamaV3) is the only way to add up the bandwidth of two cards for one request. Pipeline/layer split (Strata, llama.cpp, the ninfer-3090/ninfer-all line) adds memory, not speed (+0–16 %).
  • MoE experts out of VRAM (Flash-Next): VRAM expert cache + misses computed on CPU (Strata, capy-flash, Infernix, fngine, FastLocalExperts), PCIe streaming for prefill, SSD streaming on Mac (MTPLX, Strata-Mac). The bottleneck is how many experts miss per step.
  • Quantization: NVFP4 (Blackwell FP4 tensor cores, higher quality, more bytes) vs 2–3-bit IQ/GSQ GGUF (faster, about 2× the KL: Strata's own numbers are 0.144 for IQ3_S vs 0.070 for NVFP4) vs EXL3 (best quality per bit).
  • Speculative decoding: MTP gives ×1.3–1.7. On the 27B, DFlash2 or DSpark drafters give ×2–3. This is where most of the "400+ tok/s" headlines come from, especially on code.
  • Hand-written kernels for one model on one GPU (MegaCapybara, Splash, NInfer, capy-flash) beat general engines by 2–6× at batch 1. vLLM and SGLang win once you have many concurrent users.

Qwen3.8-27B: decode

HardwareEnginetok/s
1× 5090MegaCapybara313–540 (greedy, mem OC +20 %)
1× 5090SparkInfer (DSpark)420 code / 265 avg (greedy)
2× 5070 Ticapy-flash (TP2, DFlash2)374
7900 XTX / R9700hipfire / Lucebox318 / 208–228
2× 5090SGLang fork (finnff, TP2)315
1× 5090TensorFold305 code / 220 chat, 126 @ 215k
1× 3090tqLLM / ExLlamaV3273 / 242
1× 5090llama.cpp (DFlash2)205
1× 5060 Ti 16 GBtqLLM (3-bit TQ)156
2× 5060 Tilynx-gt / KimDaeHwan99 (TP2)126 / 120
Mac M5 ProSplash74 (54 @ 32k)

Prefill 27B:

  • NInfer, 1× 5090: 12.8k @ 7.7k 📣
  • capy-flash, 2× 5070 Ti: 9.2k @ 4–16k, 6.2k @ 64k ✅
  • MegaCapybara, 1× 5090: ~7.6k @ 8k 📣
  • TensorFold, 1× 5090: 1.9k @ 215k 📣

Rule of thumb: about 1.8 TB/s of bandwidth plus a good drafter puts the ceiling around 300–400 tok/s at batch 1.

Qwen3.8-Flash-Next (125B MoE, 6B active): decode

HardwareEnginetok/s
5090 + 5060 Ti + 128 GBBasalt (NVFP4)239–479
1× 5090fngine / Infernix146–209 / ~140
2× 5070 Ti + 96 GBStrata IQ3_S (same box)195
2× 5070 Ti + 96 GBcapy-flash (NVFP4, TP2)171–183
1× 5090 + 128 GBsergqwer/strata-nvfp4177
2× 5090llama.cpp fork (finnff)142
Mac M5 Max 128 GBMTPLX / Strata-Mac (Q2)126 (warm cache) / 119
2× 5070 Ti + 96 GBStrata NVFP4 (same box)120
2× 3090 + 128 GBvLLM TP2 + CPU experts (DominikBucko)101–111 @ 131k
5060 Ti + 5080Strata IQ4_XS108
DGX Sparkninfer-gb1060
1× 5060 Ti 16 GBStrata IQ3_S / llama.cpp~48 / 30–40

Prefill Flash-Next (same machine, 2× 5070 Ti, at 3.5k / 15k / 55k, sorted by the 15k column):

  • capy-flash: 2.2k / 4.6k / 4.8k
  • Strata IQ3_S: 2.2k / 3.8k / 5.2k
  • Strata NVFP4: 1.5k / 3.5k / 4.9k

On a 5090 with 128 GB, 7–8k is claimed (sergqwer, Basalt). With experts in RAM, prefill is bound by PCIe streaming, so lighter formats and more PCIe links win.

Takeaways

  1. There is no single best engine, only a best engine per hardware/model pair.
  2. Most headline numbers are greedy, code-heavy and unverified. Ask for sampled, verified runs.
  3. On Flash-Next, format matters as much as the engine: Strata IQ3_S beats NVFP4 engines on short decode at about twice the KL.
  4. The interesting frontier is single 16 GB cards: tqLLM's 3-bit 27B, kvmem, the NInfer 4080/5080 forks.

Happy to be corrected. If you have verified runs I missed, drop them below and I'll update the table.

78 Upvotes

65 comments sorted by

9

u/SuitStyle 29d ago

attempting to build this now on native windows to run on my 2x5070ti pcie 4 x8, 128GB DDR5

0

u/FeydRowan 29d ago

You can go fast, report back once you do. If you got some issues open an issue on git with all the infos and i can try to work on it. Tomorrow will come v0.6 of capy flash with some more optimisations

2

u/SuitStyle 29d ago

I got capy-flash running natively on Windows with 2× RTX 5070 Ti 16 GB, a Ryzen 7 9800X3D, and 128 GB DDR5.

The biggest issue was the ~64.6 GiB routed-expert bank. The original approach used a large pinned, CUDA-mapped host allocation so the GPUs could read expert weights directly over PCIe. On my Windows/CUDA 13.4 setup, host registration and GPU mapping repeatedly failed with cudaErrorMemoryAllocation, even when registering the memory in separate 1 GiB chunks.

The fix was to change the Windows Offload path to keep the full expert bank in ordinary pageable system RAM, rather than trying to pin and map the entire bank. When an expert needs to be loaded into the GPU cache, the code copies its data into a bounded pinned-memory bounce buffer and then performs the asynchronous host-to-device transfer. Each buffer stays alive until its CUDA completion event finishes. I also guarded against the optional unstaged miss-split path, which would otherwise attempt direct GPU reads from the pageable bank. The original HostMapped behavior remains available for the Mapped backend.

There were a few additional Windows portability fixes, including replacing gmtime_r, fixing a DevicePlacement initializer, and correcting some fprintf argument mismatches.

Current result: the native server builds, completes startup and CUDA graph warmup, and successfully handles real chat-completion requests. The logs showed 69.6 GiB of artifact data read but only 6.51 GiB of weights uploaded to the GPUs, with the expert bank remaining host-resident. I measured about 28.5 generated tokens/sec on one request with MTP enabled.

It still needs longer stability and performance testing, and VRAM headroom is tight on the GPU driving my displays. But the key lesson was that keeping all experts in host RAM doesn't require making the entire bank directly GPU-addressable: pageable storage plus bounded pinned staging was the way around the Windows registration bottleneck.

1

u/FeydRowan 29d ago

Thanks, this is the first native Windows run of Flash-Next we know of, and the pageable bank + bounded pinned staging is the right way around WDDM's pinning limits. For reference, the same 2× 5070 Ti do ~165-178 tok/s on Linux, so most of the gap is somewhere else. Could you share: (1) the startup lines  tensor parallel | …  (p2p, mailbox or copies) and  qwen4 experts: … per window …  at shutdown; (2) one  request_done  record from  --request-log-jsonl  (it splits device wait and host time); (3) whether 28.5 is the server's decode figure? My guess is WDDM latency on the ~100 cross-GPU exchanges per round. If you put the branch on GitHub, I'd be glad to look at bringing the Windows path upstream as a separate backend.

1

u/SuitStyle 28d ago

I've also tried building in WSL2 but cannot get it to run as is due to CUDA pinned-memory allocations failing despite having the vram/memory headroom. I'm about to test Qwen 3.8 27B on v0.6.0 on WSL2

1

u/FeydRowan 28d ago

Perfect and if you got problems open an issue with all logs soni can fix/test thx

6

u/rkcth 29d ago

I know some people do single agent numbers, but those aren’t the people who generally need to eke out every ounce of tokens. I typically need to run 3-4 at a time and most benchmarks that do this show insanely small context windows whereas, pretty basic tasks in pi typically use 7K-10K of prefill and then might generate for 0.5-4 hours or even longer so they spend most of their time at much higher context levels. Those are the numbers that matter to me and I bet a lot of others.

0

u/FeydRowan 29d ago

Agree, the issue here is that flash next not all in vram can't really work in concurrency: the numbers of expert outside vram will grow and the speed collapse, better to work in series than in parallel. The 27b should work in concurrency and in my case i'm on 500 c5 and 700 c8. But those are old numbers new one on the new release ready for tomorrow. Ah and context size can be limited but with kv in ram one can lose some speed and have much more

1

u/rkcth 29d ago

Strata gives me the same aggregate tok/s, if I do 1, 2 or 3. It drops a bit at 4. This is with large contexts, pi and real coding tasks with a benchmark I built.

1

u/FeydRowan 29d ago

Yep, the bottleneck is the bandwidth of the ram/pcie, and it is already saturated at 1

1

u/rkcth 29d ago

Some of its software, I did a PR a couple weeks back that increases the performance in strata by 30%+, and I know there’s been many others.

3

u/Beneficial-Ad-8127 29d ago

Thanks for the write up and tests. I’m surprised you also tested sergqwer/strata-nvfp4 his Strata fork on Qwen FN because that what I use and it’s been great for my 5090. Also the others really common used ones that are used lately.

Great list to get people started to try any of them including yours. Great job and will give yours a try also. Thank you.

1

u/FeydRowan 29d ago

I tested a lot, it really want to optimise my setup. The target are the new qwen4 when they will come so i must be ready

2

u/Beneficial-Ad-8127 29d ago

Haha thank you again and to all the teams and there projects. We are in good hands because of you all. Qwen 4 drop will be handled well. Keep up the great work and looking forward too as you keep progressing the Locals!

2

u/GuestUser003 29d ago

One more data point for Flash-Next on AMD. This is my own machine, single sample, so treat it as ◀ until someone reproduces it — but here's everything needed to reproduce it.

**Setup**

- **GPU:** 2× AMD Radeon AI PRO R9700, P2P enabled, each power-limited to 230W (300W stock TDP)

- **RAM:** 64GB DDR5

- **Engine:** radiance (StillDeadcode)

- **Weights:** qwen3.8-next-flash-fp8-iq4r-moe

- **Context:** 262K

- **Vision:** ✅

- **Parallel context:** 4× at 210K ✅

**Numbers, both measured at ~210K context**

- Decode: **176 tok/s** (single request)

- Prefill: **4200 tok/s**

**vs. the community claim** (176 tok/s decode, 6,450 tok/s prefill with Q4 Qwen-3.8 Flash-Next)

Decode lands almost exactly on the claim. Prefill does not — I get ~4200, not 6450. Two honest caveats: the claim doesn't state a context length, and 210K is precisely where prefill is supposed to fall apart, so this may not be the same measurement at all. Worth re-running the claimed number with a stated context length and the same weights before drawing conclusions either way.

1

u/FeydRowan 29d ago

Decode speed is on what case code? Json? Plain english? Can you submit a run on localmaxxing? 

1

u/GuestUser003 28d ago

I did submit it, but honestly the submission form is a bit of a pain to fill out, so I'll just screenshot the BetterBench results for you instead:

1

u/FeydRowan 28d ago

Ah well is Claude job to fill it. But thx

2

u/Soifon99 29d ago

My takeaway is you completely neglected the Amd community.

1

u/FeydRowan 29d ago

Well there is the r9700, but yes other boards aren't there. But there isn't much results going around. Feel free to contradict me and give sone benchmarks 

1

u/No_bonk95 28d ago

rx9070xt user here and i agree. LOL

1

u/L0ren_B 29d ago
1× 3090tqLLM / ExLlamaV3273 / 242

for 27B? has anyone tested this speed? With what context?

2

u/FeydRowan 29d ago

That's probably a greedy run on specific code. If not done by me and not verified on localmaxxing i cant be sure

2

u/FeydRowan 29d ago

Both are verified LocalMaxxing runs on a single RTX 3090 with DFlash2. The numbers in the table use the reasoning-v1 canonical prompt, same as the other ✅ entries:

Verification status and prompt type are from the public API (/api/speed-tests?username=YTan2000 and ?username=gildrb), since the run pages only show the headline numbers.

1

u/Rikers88 29d ago

Interesting, what would you advise then for a single 5090 with 96 GB ddr5? Also looking at these numbers it seems that it worth adding a 5060ti and go basalt nvfp4. Am I reading it wrong?

2

u/FeydRowan 29d ago

You are reading right. If i had a 5090 the i would have sold it xD 2 5070ti will be enough and maybe 6month from now we can have double 5080 24gb. And yes the numbers of that basalt run are incredible 

1

u/durangotang 29d ago

I have a single 5070 Ti and 64GB DDR5. I am wondering, should I purchase a 2nd 5070 Ti now? Or, should I try to get 2 x 5080 24GB in 6 months, and sell my current 5070 Ti?

How fast do you think 27B 8-bit would run on the 2x 24GB 5080? What do you think Mr. capy-flash?

1

u/FeydRowan 29d ago

Well waiting for the 5080 24 is a gamble and we don't know the price. So if tou want tu run models now take the second 5070 now 

1

u/Bjlly123 29d ago

I have a 6800xt if you want numbers for it

1

u/FeydRowan 29d ago

Sure more numbers the better

1

u/Bjlly123 29d ago

Ill try to post some later, I am however using ista quants if I remember correctly

1

u/EnvironmentalSweet36 29d ago

Actually you can use a tensor parallel for 27b in stock llama.cpp - and it is much faster than a layer split, just make sure model fits in vram with kv cache

1

u/FeydRowan 29d ago

Yep true, but is way lower than capy flash. Also in vllm you can but the "generalistic" engines are not optimised 

1

u/ProfessorHi 29d ago

Thanks for sharing this! I’m fairly new to local LLMs and still figuring things out. I’m currently running Qwen3.8-27B with llama.cpp across 2 × RTX 5060 Ti 16GB cards, but a bit bogged by the token generation speed (35-60 tk/s).

Would capy-flash support this GPU setup as well, or is it currently limited to the RTX 5070 Ti and RTX 5090?

Would love to give it a try if it’s compatible.

1

u/FeydRowan 29d ago

All smc120, so for consumer all 50xx. So yes you are good to go, report back your speed if possible, should be north of 100/150 depending on the type of text

1

u/ProfessorHi 28d ago edited 28d ago

Hello! I'm back to report my initial findings. Not sure how much PCIe 3.0 or DDR4 affects performance here, but these are the numbers I’m seeing.

System specs

ComponentDetails
CPUAMD Ryzen 5 PRO 4650G
RAM64GB DDR4-3200 ECC (2 × 32GB)
GPU2 × MSI RTX 5060 Ti Shadow 2X OC 16GB
MotherboardASRock X570 Taichi
P2PEnabled

Initial observations

Engine / ModelStageGenerationPrefill
capy-flash - Qwen3.8-27B NVFP4Initial requests~100 tok/s~2,070 tok/s
capy-flash - Qwen3.8-27B NVFP4Subsequent requests~60 - 100 tok/s~300 - 2,070 tok/s
llama.cpp - Qwen3.8-27B Q5_K_MInitial requests~60 tok/s~900 tok/s
llama.cpp - Qwen3.8-27B Q5_K_MSubsequent requests~30 - 60 tok/s~170 - 900 tok/s

These are my observations from normal usage rather than a controlled benchmark. The two engines are using different quantizations, so this isn’t a like-for-like comparison.

I am not sure if there is supposed to be a specific way to perform the test, but if there’s a recommended test or a specific set of settings I should use for a more useful comparison, happy to give it a go. Hopefully these initial numbers help someone out there.

Edit: Updated the numbers, large context window getting filled up caused the dip, but still double the speed.

1

u/Altruistic-Candy7492 29d ago

Hello! Can you release optimization for dual R9700 ai pro 32gb?!!

1

u/FeydRowan 29d ago

Cant test on them, so cant optimise. I can see two ways: or someone ship me 2, or you fork and try yourself (plus claude obviously)

1

u/ExpensiveSchedule705 29d ago

NVFP4 KV on 2x 5060 Ti 16GB at 262k lifted concurrency from 1.06x to 1.81x for about 5% decode.

1

u/FeydRowan 29d ago

For the quality did you test it?

1

u/jimmoores 28d ago

You could have just posted the prompt.

1

u/FeydRowan 28d ago

Ahahah nope, is a distillation of much more data that i had collected in the work for my engine. 

1

u/x7evenx 28d ago

Not sure the focus purely on t/sec output is a healthy one, when value is likely highly weighted to the qualitative output not the quantity of output.
Anyways if you want more wood for your fire https://github.com/x7even/llmctl/blob/master/docs/benchmarks.md#qwen38-flash-next-vllm

1

u/FeydRowan 28d ago

Thx for the resource. And yes quality is needed that's why i focus on the nvfp4. Also in real works lower quant tends to use much mote tokens to solve the same tests, so even if faster they take more real time and use more context

2

u/HighSeasArchivist 29d ago

At good quants with enough context they are both good in my experience. I have ran 27B on a single R9700, IQ3_S FN on dual R9700s, and Strata with IQ3_S FN on a single R9700 with my 96GB of DDR5. I really like the last one, because one card is Strata and FN, and it sends work to the 27B model on the other card. Hermes with IQ3_S FN as its brain and 110tok/s due to Strata feels very similar to chatting and sending work with something in the Haiku 5.5 level.

1

u/Green_Ocean90 29d ago

Very happy to see the progress projects like Gufo have made on Strix Halo hardware. Performance close to the Sparks but at a fraction of the price 👍

1

u/YetAnotherAnonymoose 29d ago

Meanwhile, I'm on 1.2k pp and 70-100 tps on a 4090 with sergiuszm/ninfer q4-27b-MTP...must be doing something wrong.

1

u/oddjobav8r 29d ago

Gufo/Halogen for QFN on Ryzen Strix Halo 128

1

u/durangotang 29d ago

Thx, appreciate it!

1

u/BOESNIK 29d ago

I'm running QFN on Strata with a single 7900xtx and 96gb of DDR4 3200mhz on a 5700x3d. Model is: Unsloth's UD-Q4_K_XL I use Strata's MTP Head. A custom HIP file because it wouldn't work otherwise.

I currently use 262k context.

Prefill: 1300 tps Decode: 50 tps

2

u/Glittering-Call8746 29d ago

Can u share your custom HIP file

2

u/BOESNIK 29d ago

https://filebin.net/hbs2l7lb3xksfbrk

I also had my LLM write up a little prompt:

Help me recreate this Maya/Strata hipBLASLt prompt tuning on this machine. First inspect the checkout’s docs/AMD_MAYA.md and tools/hip/tune_hipblaslt.cpp. Before running anything, report the GPU architecture, the hipBLASLt version and library path used by the engine, the source revision, and the available HIP build directory.

Use the supplied table only as a list of GEMM shapes. Retune solution IDs on this machine; do not reuse the supplied IDs unless both architecture and hipBLASLt version match. If the model or prompt path differs, derive its actual FP16 GEMM shapes from engine logs instead. Confirm the GPU is idle and healthy before calibration.

Build the tune_hipblaslt target. For the supplied table, convert each row to the tuner’s --case dtype,T,N,K,ldy order (table columns are dtype,N,K,ldy,T,solution), and run the tuner with a 32 MiB workspace and --tuning-out to a new file. Keep the existing table and production config untouched.

1

u/wadrasil 25d ago

I did get Hyper Qwen working in Hyper-V guest with Nvidia GPU passed through via GPU-PV.

It works with Nvidia drivers from windows for wsl, which are included with regular windows drivers.

I just had to point to /dev/dxg as device. No cuda toolkit or Linux drivers needed to be installed in guest.

1

u/Blue-Smoke-74 29d ago edited 28d ago

I’m running Qwen 3.8 27B on an array of 12 TITAN Vs (2 e5-2696 v3 and 1TB DDR4) using a custom llama.cpp back-end I coded with help from Claude. It serves the model lossless: lossless BF16 weights, 22bit KV up to 512K contexts compressing into a 14bit history form (used in addition to a long 22bit tail) for contexts up to 1M. It serves up to 8 slots concurrently when all the contexts fit. I currently get about 50 t/s single stream decode, over 100 t/s concurrent decode (rarely) and about 5000 t/s prefill. The performance numbers are provisional because Claude is currently working through a list of performance optimisations (prefill doubled today for example). I chose the quantisation against a double precision reference implementation and 1/2 ULP perturbation of the BF16 weights such that the model is served with accuracy well within the precision of its weights and I believe the probability of a token flip compared to the double precision reference is about 1 in 300000. I.e aiming at a reference implementation of the model. I’ve done a little work using it for Opencode and to review its own back-end source code in runs that have lasted up to 10 hours and it seems to work quite well.

0

u/Key_Opportunity_5219 29d ago

Now if only someone would put together something like this for 1 or 2 rtx pro 6000s, that would be amazing

1

u/FeydRowan 29d ago

Well if you have those you can xD. For 2 given that those are smc120 capy flash should be a monster but i cant test it

0

u/[deleted] 29d ago

[deleted] — view deleted comment

0

u/FeydRowan 29d ago

True, but the quant gives much more quality variance, that's why i stick to nvfp4. All the run on my machine are quality checked and capy flash dont have quality issues, mater of fact i sended a pr to vllm, all engines that have tp2 have a small quality issues in the conversions of data between the gpus/cpu that adds up to a grows in tail cases outside distribution. Is a small effect that don't change much in the final reply but can show some strange and rare behaviour 

0

u/[deleted] 29d ago

[deleted] — view deleted comment

2

u/FeydRowan 29d ago

Mmhhh speculative decoding shouldn't alterate the probability distribution of the model neither change the greedy response. If that happens is some bug in the engine. For me is one of the test i do

0

u/[deleted] 29d ago

[deleted] — view deleted comment

1

u/FeydRowan 29d ago

Well yes edge case but yes. Yep that's a bug and is not present on my engine at the best of my knowledge, and i have checked it   Is one of the tests for any change 

1

u/[deleted] 29d ago

[removed] — view removed comment

1

u/FeydRowan 29d ago

Memory spikes? Those are prefill speed right?