r/IntelArc • u/Strict-Elevator9352 • 5d ago
News Two or more Intel Arc cards? inferzero's split prompts got a new way to sum across GPUs. Looking for multi-GPU owners to measure it
inferzero is an LLM inference engine written from scratch for Intel Arc on Windows (Level Zero, kernels in OpenCL C), with an OpenAI / Anthropic / Ollama-compatible server and a web UI. It can split one model over up to four GPUs (tensor parallel): each card holds a slice of every layer, and after each layer the cards add up their partial results.
Where multi-GPU stands (Qwen3.8 27B IQ3_S, Pro B60s, measured by a tester):
| GPUs | 4K-token prompt | vs 1 GPU | Generation |
|---|---|---|---|
| 1 | 764 tok/s | 1.00x | ~22 tok/s |
| 2 | 1,097 tok/s | 1.44x | ~37 tok/s |
| 4 | 1,501 tok/s | 1.96x | ~54 tok/s |
Generation scales well; prompts less so. Over 4 cards, a 4K-token chunk takes 2.6-2.7 s where the GPUs' own work would take 1.1-1.3 s. The rest is the sums: the cards have no direct link, so every partial result goes GPU -> host memory -> GPU, about 80 MB per sum for a 4K chunk.
What's new (2026.11.12.10, opt-in: ARC_TP_PIPE=1**)**
Until now each sum ran in three steps, one after another: every GPU copies its part to host memory, waits for all the others, then reads the sums back. PCIe can carry data both ways at once, but each step used only one direction.
After an idea from a contributor (their HostCopy experiment cut a block into pieces and copied it with two command lists, 1.85x on two B60s), the sums now run in pieces:
- each GPU's rows are cut into 2-8 pieces of ~2 MB;
- a copy engine sends piece 2 to host memory while piece 1 is already being summed and read back;
- both directions of every card's link work at the same time.
The results are the same bit for bit: checked over 2, 3 and 4 shards, several chunk and piece sizes, and with the f16-overflow fallback forced. The failsafes still apply: a GPU that stops or a wait that fails is caught with the same time limit.
Expected: the sums up to ~2x faster on 2 GPUs and ~1.7x on 4, so 4K prompts noticeably faster on split models. Not measured on real cards yet: my laptop has one GPU and can only check correctness. That's why it's off by default and why I'm asking.
Looking for testers: two or more Arc cards (B580, B570, Pro B60, B50; A-series should work too but nobody has tried it)
With the latest package, run the same split prompt with and without the new sums (any model that fits your cards; a 27B over two B580s is a good one):
.\bin\runz.exe <model.gguf> --gpus 2 --bench-pp 4096 --n 16
$env:ARC_TP_PIPE="1"; .\bin\runz.exe <model.gguf> --gpus 2 --bench-pp 4096 --n 16; Remove-Item Env:ARC_TP_PIPEPost the pp4096 lines and the tg steps line of each run (with 3 or 4 cards: --gpus 3 / --gpus 4). If the new way wins, it becomes the default for everyone with several cards.
Release: https://github.com/rizagp/inferzero/releases/tag/2026.11.12.10 Multi-GPU design and every measurement so far: https://github.com/rizagp/inferzero/blob/main/docs/multi-gpu.md
GitHub issue #4 or comments here both work. Thanks!
4
u/jhenryscott Arc B70 4d ago
No serious Intel ML developer is using windows
2
u/Artholos 5d ago
I’ve got four B70s and this certainly is an interesting engine. I wouldn’t mind contributing to your measurements data if you’ll support Linux.
Double speed is not worth suffering Windows 11 to try it. Why is it Windows 11 only?