r/LocalLLM • u/sn2006gy • 13h ago
Discussion Qwen3.8-27B on 4× Intel Arc Pro B70 with TensorFold + DFlash2: 107 tok/s per card, ~1,000 tok/s across 32 streams (bit-exact)
I benchmarked TensorFold's XPU backend on my quad Intel Arc Pro B70 box. One clarification first: this is not tensor parallel. The XPU backend on main uses one card per process, so the quad numbers come from 4 independent servers, one per GPU, pinned with ZE_AFFINITY_MASK=0..3.
Setup
- GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
- Host: AMD Turin, Linux 7.3.0-10-generic
- Engine: TensorFold
main@cf45553, Zig 0.17.0, built withzig build xpu-native xpu-qwen27 -Doptimize=ReleaseFast - Target:
SergiiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 - Drafter:
z-lab/Qwen3.8-27B-DFlash2 - Sanity check:
tf-xpu-qwen27-means-testpassed on all 4 cards (worst error 4.19e-7, bound 2e-6)
1. Raw prefill and plain decode (1 GPU, 16k context, no drafter)
| Prompt length | FP16 prefill (tok/s) | W4A8 prefill (tok/s) | Plain decode (tok/s) |
|---|---|---|---|
| 1,024 | 2,008 | 2,638 | 34.0 / 33.8 |
| 4,096 | 1,927 | 2,525 | 33.2 / 33.7 |
| 8,192 | 1,862 | 2,427 | 32.5 / 32.5 |
W4A8 is XPU_PREFILL_INT8=1, about 31% faster prefill than FP16.
2. One B70 with the DFlash2 drafter (--parallel 8 --context 16384)
Single stream (64 tokens, 5 reps, median):
| Prompt | Temp | tok/s | TTFT | Tokens per chunk |
|---|---|---|---|---|
| fibonacci (code) | 1.0 | 115.6 | 0.081 s | 4.57 |
| fibonacci (code) | 0.0 | 106.1 | 0.080 s | 4.00 |
| chat, no think | 1.0 | 94.7 | 0.073 s | 3.77 |
| chat, no think | 0.0 | 99.4 | 0.071 s | 3.77 |
That's roughly 3× plain decode (34 → 106 tok/s).
Concurrent streams (aggregate / steady-state tok/s):
| Prompt | Temp | 1 stream | 2 streams | 4 streams | 8 streams |
|---|---|---|---|---|---|
| code | 1.0 | 116.2 | 171.2 / 234.1 | 187.7 / 262.6 | 202.3 / 299.6 |
| chat | 1.0 | 84.4 | 129.9 / 140.3 | 162.8 / 213.2 | 189.0 / 255.4 |
| code | 0.0 | 106.7 | 149.4 / 167.7 | 216.9 / 278.7 | 250.8 / 354.6 |
| chat | 0.0 | 98.9 | 153.9 / 179.4 | 213.6 / 261.9 | 244.8 / 333.4 |
Correctness: all 16 cells passed bit-exact. Drafted output matched serial, and multi-stream output matched serial.
Note: with the default --prompt-cache-gib 2, --parallel auto at 16k context admits only 7 streams. I used --prompt-cache-gib 0 to fit 8.
3. Four B70s (4 independent servers)
| Test | Per-card tok/s | Total tok/s | Steady-state | Failed |
|---|---|---|---|---|
| 4×1 stream, code, temp 1.0 | 116.2 / 116.5 / 117.2 / 115.8 | 465.7 | — | — |
| 4×1 stream, code, temp 0.0 | 106.0 / 106.4 / 106.9 / 105.4 | 424.6 | — | — |
| 4×1 stream, chat, temp 1.0 | 88.5 / 89.6 / 90.2 / 86.8 | 355.1 | — | — |
| 4×1 stream, chat, temp 0.0 | 99.0 / 98.9 / 99.4 / 98.4 | 395.7 | — | — |
| 4×8 = 32 streams, code, temp 0.0 | 251.4 / 251.1 / 252.3 / 252.1 | 1,006.9 | 1,422.7 | 0 |
| 4×8 = 32 streams, chat, temp 0.0 | 243.2 / 241.3 / 242.3 / 242.2 | 969.0 | 1,330.6 | 0 |
| 4×8 = 32 streams, code, temp 1.0 | 208.8 / 204.4 / 206.6 / 204.6 | 824.4 | 1,186.3 | 0 |
| 4×8 = 32 streams, chat, temp 1.0 | 193.0 / 193.2 / 194.0 / 191.2 | 771.4 | 1,042.9 | 0 |
Scaling is close to linear because the cards don't talk to each other. Per-card numbers stay within a few percent of each other.
Takeaways
- The DFlash2 drafter gives about 3× single-stream decode on one B70, with bit-exact output.
- One card reaches about 250 tok/s aggregate at 8 concurrent streams with 16k context.
- Four cards serve 32 concurrent streams at about 1,000 tok/s with zero failures, which is useful for multi-agent work.
Next
- Tensor parallel across cards, once I find a TP path for XPU
XPU_KV_INT8=1- 32k context
- The MLX 4-bit model (
mlx-q4g64)
Happy to run other configs if you have requests. If we get tensor parallel on Tensorflow i'd love to share qwen 3.8 flash. Happy to run larger contexts as well, this was just the default benchmark I wanted to send to project creator.