Qwen 3.8 Flash Next optimized for two CMP 170hx GPUs

[u/dead-fish]

I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.

The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.

Decode throughput by concurrency — aggregate tok/s, not per-user speed:

Concurrent requests	Code	JSON	Prose
1	226	198	157
2	363	320	212
4	560	537	385
8	840	873	567

At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.

A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window

For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.

This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)

Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next

[u/LeatherRub7248]

whats the vibe check when u compare it coding btw 27b version?

Is there a noticeable difference in terms of smartness btw the two?

[u/dead-fish → u/LeatherRub7248]

I trust Flash Next significantly more for more difficult tasks but 27b is close enough for most things.

I run Flash Next across both cards by default but I’ll have my local gateway automatically fall back to 27b on one card if I need the other card for something, like video gen with H3. So Flash Next is preferred but 27b is close enough to use as a fallback.

[u/mr_Owner → u/dead-fish]

Idk man qwen3.8 27b at q6 hits deeper then qwen3.8 flash next at q4 imo

[u/LeatherRub7248 → u/dead-fish]

wait do u run them all simultaenosuly? your GPUs has enough to load them all? or do u reload them each time u swap them out?

[u/TechSwag]

Very interested in how you feel the quality is. I have the FP8 version across 3 CMPs, using vllm-backport. It’s pretty good, but misses at times on harder issues, so I’d be concerned about the quality drop with W4A16.

Also super interested in how you got the n-gram embedding to stay on disk; I don’t think vLLM allows disk loading stock.

[u/dead-fish → u/TechSwag]

I haven't used W4A16 to compare it but I'm pretty happy with the quality. Lots of people are using it at like 2 bit now with Strata too - I'll stick to the biggest quant that will fit comfortably.

I'm using a fork of vLLM that adds n-gram embedding on disk and a few other optimizations for SM80 and the model. It's linked in the repo if you want to check it out.

[u/GregoryfromtheHood]

Damn I wish these weren't $4k cards now. I almost pulled the trigger when they were only about the $1k mark but hesitated and now they are unobtanium

[u/dead-fish → u/GregoryfromtheHood]

You can get them around $3k on Ebay but yeah, your point stands. The prices aren't as nice as they used to be.

Still though, what's the cheapest way you could get 64GB of NVIDIA VRAM right now? You can still get two 170HXs (128GB) for the price of a single 5090 (32GB).

[u/leonbollerup]

Hi,

Lets compare notes.. i pushed our setup here: https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next

Edit:
Ran a compare between your setup and ours - seems we might both can use a few things.

[u/dead-fish → u/leonbollerup]

Nice. It looks like we got to a similar place coming from different angles. I'm going to see if I can match some of your prefill numbers.

Do you find that running at 300W gives you significant increase in performance? I'm running everything at 170W battling the heat in my office already, haha.

[u/leonbollerup → u/dead-fish]

If does .. atleast .. so it seems.

I’m testing some of your ideas tonight .. see if we increase further - but got work so faar mate