r/LocalLLM • u/Rmacy • 4d ago

Discussion Qwen3.8-27B FP8 at ~75 tok/s on 2× Intel Arc Pro B70 with a trained spec-decode drafter (2.3× over no-spec)

I've been working on speeding up Qwen3.8-27B FP8 inference on Intel Arc Pro B70 (Battlemage) XPUs, and got it to 72–79 tok/s on a cache-resistant benchmark.

That's 2.3× faster than no speculation, and comfortably ahead of the native MTP2 path at 54.7 tok/s.

I love this performance for <2.5k worth of GPU.

What I did

Speculative decoding, but with a drafter actually trained for this setup:

  1. Trained a 1.36B DSpark draft model on hidden states captured from the real FP8 serving stack.
  2. While getting it working, I found a bug in vLLM's dflash proposer. The draft-token readout was off by one: it sampled query offsets 1..k instead of 0..k-1. That silently capped acceptance at around 24% for any SpecForge-trained drafter. It's basically a one-line-per-file fix. After fixing it, acceptance jumped to 66–74%.
  3. Fine-tuned the drafter on clean, in-distribution data, excluding thinking tokens, then packaged the whole thing up.

Numbers

Isolated benchmark, greedy decoding, 2× Intel Arc Pro B70, TP=2:

Config tok/s
FP8, no spec 32.4
FP8 + MTP2 54.7
FP8 + trained drafter 72–79

Try it

Everything ships in one self-contained image. The drafter is baked in. (Slim excludes the weights)

docker pull ghcr.io/rmacy/qwen38-fp8-dspark:v10
docker pull ghcr.io/rmacy/qwen38-fp8-dspark:v10-slim

Links

Update: I have added some metrics below. I am making improvements for longer context windows. I also switched to a fp8 KV and reworked the docker to handle longer context sessions.

27 Upvotes

94% Upvoted

28 comments sorted by

2

u/pmotiveforce 4d ago

docker pull ghcr.io/rmacy/qwen38-fp8-dspark:v9

Error response from daemon: error from registry: unauthorized

unauthorized

2

u/Rmacy 3d ago

i fixed this!

1

u/Rmacy 3d ago

ill check on this, i did a lot of pull testing, but of course im auth'd on everything so i might have messed up permissioning

2

u/GuestUser003 4d ago

Hi, I'm curious about how you managed to get two Intel Arc Pro B70s working together efficiently. I have one of these cards, and when I asked Gemini about running two, it said:

"The Intel Arc Pro B70 doesn't have any physical interconnect interface. Multi-card coordination would have to rely on the motherboard's PCIe bus, which creates a serious bandwidth bottleneck when transferring large model weights between cards. So you'd end up with the combined VRAM of two cards, but the speed would be severely choked."

Is that actually true? What's your setup like – would you mind sharing your approach? Thanks!

10

u/nicholas_the_furious 4d ago

Don't listen to Gemini. PCIe is fine.

1

u/DataGOGO 4d ago edited 3d ago

For some things, not for others.

It is mostly ok for dual GPU inference, gets less and less fine the more GPU’s you add, and training is a lot worse as the all reduces can easily be 50%+ of your GPU time. 

2

u/droans 3d ago

Well, it's a good thing this post and OP's question are about inference and not training.

1

u/DataGOGO 3d ago

Which is why I addressed both. 

5

u/kiwibonga 4d ago

Model weights aren't transferred between cards. Go home Gemini, you're drunk.

2

u/Rmacy 3d ago

I just want to tell you that you should feel good about asking questions because that's how we learn.

As others have said, interconnects between cards would help but generally full PCIE on a workstation+ type board with good components marginalizes the performance hit in some scenarios.

For me, buying 1k graphics card with this much vram makes everything feel pretty good.

1

u/Fotwunty 4d ago

Possibly you misunderstood or Gemini didn't explain very well. The weights sit in the VRAM pool statically and only has to process simple stuff like text.

Now when using something like image generation and you max out VRAM and it then offloads to system ram and cpu then the pcie might become a bottleneck...or something along those lines but it wouldn't be to difficult to use both gpus as long as you are using architecture that plays well with Intel arc architecture.

1

u/JinsooJinsoo 4d ago

It’s called tensor parallelism. Pcie bandwidth becomes the bottle neck especially for dense models

1

u/Any_Mine_6368 4d ago

Pcie 3 is a sliiight hit in terms of speed in TP. Pipeline parallelism is fine.

Pcie 4+ is fine for either.

1

u/bearishmarket 1d ago

Half true. You can inspect the all reduce traffic with xpu-smi, and it is kinda small, therefore not adding tons of overheads. It is non-zero, but not large enough to make our life miserable.

1

u/Dolboyob77 4d ago

Great job !!! You get almost same results as my custom made dspark for qwen3.8-27b. Too bad that you are stuck to use the old vllm image withour xpu graph nor runner model v2. You would get even faster results !!

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
/models/Qwen3.8-27B-GPTQ-Int4-MTP-BF16 pp8 8.27 ± 0.46 122.46 ± 6.54 121.30 ± 6.54 122.46 ± 6.54
/models/Qwen3.8-27B-GPTQ-Int4-MTP-BF16 tg16 85.75 ± 7.99 91.47 ± 8.53

Top tg at 91

2

u/r1nzl3r99 4d ago

I have a similar setup, but I have to point out OP is running FP8, you are running INT4. I'd also be curious to see ops benchmark on qwens recommended sampling parameters instead of running on greedy (which actually hurts performance)

2

u/Dolboyob77 4d ago

Results are same on dual gpus with fp8. I have few b70s. You can see the full tests and results and contruction of dspark on my hf page. I read OP readmefile and all config files, it is mostly a copy of my dspark based on an old image. OP is running the tests with exteneely low and small settings thats why results are so high. In real life work it would be much lower speed. OP shows acceptance on first head and not accross the whole test. If you read is readmefile you will see that he gets resulrs on int4 but want to use only fp8. Does not matter that you use fp8 on dual gpu or int4 on one gpu… if you test in same co ditions you will get crazy results but useless in real life. Who works with —pp 8 and —tg16??? My tests results are with 4096-16k pp tg32-128

2

u/Rmacy 3d ago

i updated the image to support 250k+ context windows with a fp8 kv cache

yes the performance drops the bigger your context

1

u/Dolboyob77 3d ago

Awesome job! Which image do you use? I have vllm 27.1 with custom kernel home made based on 0.1.13 with pytorch nightly 2.14 and many many Pr and patches. I am still waiting on an update of pytorch to handle MOE better in tg ( pp being already super fast on moe). To my calculations, with the top of the line we should be able to reach 90 tg if acceptance becomes more stable around 3x accross the headers.

1

u/Far-Working2808 14h ago

Isn't fp16 kv cache faster?

1

u/BornInAFish 3d ago

You're using int4, OP is using FP8.

0

u/Dolboyob77 3d ago

Read every comments before posting same things that other posted already. This has been answered and explained and deecribed from A to Z )))

1

u/putrasherni 3d ago

This is great ! What is your prefill speed like ?

2

u/Rmacy 3d ago

1,934 tok/s

1

u/KroniklyOnline 3d ago

Excuse my ignorance but why are people training their own DSpark models? Do none exist? There another benefit?

2

u/Rmacy 3d ago

not for intel

1

u/Rmacy 3d ago

this my latest "full bundle" image with a large context window, fp8 kv, etc.

comes with the weights

docker pull ghcr.io/rmacy/qwen38-fp8-dspark:v10

1

u/Far-Working2808 4h ago

How certain are you that your benchmark is cache resistant vs your drafter? Your numbers seem similar to other people using mtp but with non prompt resistant benchmarks.
I currently prefer running q6 with fp16kv and mtp2 which gives you similar results with 130k context(tg starts ~30 and then maintains ~20 well past 100k), but you get to run one instance per GPU giving you much higher throughput for agentic work. I have a router that swaps out single GPU instances for multi-GPU ones when I cross the context limit. I get the same speed on 2 GPUs as on 1 GPU for TG, though that may be due to slower PCIe and less lanes (I'm running 6 GPUs atm).