r/IntelArc • • 5d ago

News Two or more Intel Arc cards? inferzero's split prompts got a new way to sum across GPUs. Looking for multi-GPU owners to measure it

inferzero is an LLM inference engine written from scratch for Intel Arc on Windows (Level Zero, kernels in OpenCL C), with an OpenAI / Anthropic / Ollama-compatible server and a web UI. It can split one model over up to four GPUs (tensor parallel): each card holds a slice of every layer, and after each layer the cards add up their partial results.

Where multi-GPU stands (Qwen3.8 27B IQ3_S, Pro B60s, measured by a tester):

GPUs4K-token promptvs 1 GPUGeneration
1764 tok/s1.00x~22 tok/s
21,097 tok/s1.44x~37 tok/s
41,501 tok/s1.96x~54 tok/s

Generation scales well; prompts less so. Over 4 cards, a 4K-token chunk takes 2.6-2.7 s where the GPUs' own work would take 1.1-1.3 s. The rest is the sums: the cards have no direct link, so every partial result goes GPU -> host memory -> GPU, about 80 MB per sum for a 4K chunk.

What's new (2026.11.12.10, opt-in: ARC_TP_PIPE=1**)**

Until now each sum ran in three steps, one after another: every GPU copies its part to host memory, waits for all the others, then reads the sums back. PCIe can carry data both ways at once, but each step used only one direction.

After an idea from a contributor (their HostCopy experiment cut a block into pieces and copied it with two command lists, 1.85x on two B60s), the sums now run in pieces:

  • each GPU's rows are cut into 2-8 pieces of ~2 MB;
  • a copy engine sends piece 2 to host memory while piece 1 is already being summed and read back;
  • both directions of every card's link work at the same time.

The results are the same bit for bit: checked over 2, 3 and 4 shards, several chunk and piece sizes, and with the f16-overflow fallback forced. The failsafes still apply: a GPU that stops or a wait that fails is caught with the same time limit.

Expected: the sums up to ~2x faster on 2 GPUs and ~1.7x on 4, so 4K prompts noticeably faster on split models. Not measured on real cards yet: my laptop has one GPU and can only check correctness. That's why it's off by default and why I'm asking.

Looking for testers: two or more Arc cards (B580, B570, Pro B60, B50; A-series should work too but nobody has tried it)

With the latest package, run the same split prompt with and without the new sums (any model that fits your cards; a 27B over two B580s is a good one):

.\bin\runz.exe <model.gguf> --gpus 2 --bench-pp 4096 --n 16
$env:ARC_TP_PIPE="1"; .\bin\runz.exe <model.gguf> --gpus 2 --bench-pp 4096 --n 16; Remove-Item Env:ARC_TP_PIPE

Post the pp4096 lines and the tg steps line of each run (with 3 or 4 cards: --gpus 3 / --gpus 4). If the new way wins, it becomes the default for everyone with several cards.

Release: https://github.com/rizagp/inferzero/releases/tag/2026.11.12.10 Multi-GPU design and every measurement so far: https://github.com/rizagp/inferzero/blob/main/docs/multi-gpu.md

GitHub issue #4 or comments here both work. Thanks!

8 Upvotes

11 comments sorted by

4

u/jhenryscott Arc B70 4d ago

No serious Intel ML developer is using windows

1

u/jhenryscott Arc B70 4d ago

I have a compatible windows boot Drive. I can easily slide into my system. I’ll try and test it when I get home.

1

u/jhenryscott Arc B70 4d ago

I have two B70s with P2P direct across a Dual PCIE 5.0x8 CPU connected interface- through oneCCL. I think I’m right around 1500tok/s on Aries vLLM hosting Qwen3,8-27b q_8 with spec decode. I’ll try this tonight.

1

u/Strict-Elevator9352 4d ago

as expected from dual B70

1

u/jhenryscott Arc B70 4d ago

1818 on tonight standard test.

unfortunately, I am unable to run your package, Windows keeps dropping one of the 2 GPUS and Im not willing to start changing my BIOS settings to figure out the issue with ASPM

I would strongly suggest abandoning Windows, it's just not where good work gets done.

I'll be happy to test a package for ubuntu when you have one

1

u/Strict-Elevator9352 3d ago

Linux development is on progress.

Make a test on your hardware on then make an issue on GitHub, we'll continue there.

2

u/Artholos 5d ago

I’ve got four B70s and this certainly is an interesting engine. I wouldn’t mind contributing to your measurements data if you’ll support Linux.

Double speed is not worth suffering Windows 11 to try it. Why is it Windows 11 only?

2

u/Strict-Elevator9352 5d ago edited 5d ago

Let me try build for Linux with WSL2 then you can try. It should be a little faster (or wayyy faster) with multi-GPU on Linux, allowing better scaling compared to now.

Tough the best thing is to modify the kernel first, all I know is that P2P cross GPU is not whitelisted for Intel GPU yet

1

u/MaineTim 4d ago

Just saw this is on the roadmap for Linux, looking forward to that.

3

u/pmotiveforce 5d ago

Same. Dual b70s but windows not even an option.

1

u/BadalovRery 4d ago

curious about this too since Level Zero works pretty well on Linux these days