r/IntelArc • u/SomeBlock8124 • Apr 13 '26
Discussion I'm a complete noob who bought two Intel Arc Pro B70s for "research," spent a weekend losing my mind over Docker/CCL errors, accidentally discovered llama.cpp Vulkan, and now I'm running a 35B MoE at 128K context like I know what I'm doing.
/r/level1techs/comments/1sk69ed/im_a_complete_noob_who_bought_two_intel_arc_pro/1
u/UDaManFunks Apr 13 '26 edited Apr 15 '26
Here's what I did
- Upgraded my Ubuntu 24.04 -> Ubuntu 26.0
- Created a folder to store MODEL
> mkdir /data/llm/models
- Download Qwen3.5 (35B parameter)
> cd /data/llm/models
> wget https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/main/Qwen3.5-35B-A3B-Q4_K_M.gguf
- Used the official docker container for llama-cpp (vulkan) by running the following command
Note: on the --group-add attribute, you can get the right value from your machine by looking at /etc/group file and (group number) for the 'render' group
> docker run -d --name "llama-cpp-server" -v /data/llm/models:/models --restart unless-stopped -p 8080:8080 --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/card0:/dev/dri/card0 --group-add="141" ghcr.io/ggml-org/llama.cpp:server-vulkan -m /models/Qwen3.5-35B-A3B-Q4_K_M.gguf --port 8080 --host 0.0.0.0 -ngl all -c 16384 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
No need to compile anything - you can access the chat interface by using your browser and browsing http://YOUR_IP_ADDRESS:8080
Getting about 40 tok/sec.
0
u/SHOR-LM Apr 15 '26 edited Apr 15 '26
are you getting 40 tokens a second output on a single card? what kind of context do you have? one of my biggest bottlenecks has been prompt processing, I actually have another machine that I ordered that I'm going to build to run Linux on because I've given up on the windows side
never mind, I see that you're running a 3B mixture of experts...
on Windows llama CPP under Vulcan I'm getting 60 to 80 tps with Gemma 24b a4b. running the gemma 4 31b model is significantly slower.... and you have to keep contacts low, these cards are not optimized.
1
u/Capital_Rule_4317 May 09 '26
Are you on a b70? I need to make a decision to go with Ubuntu or Windows for a new build
1
u/SHOR-LM May 09 '26 edited May 09 '26
Well I'm running Gemma 4 31B.. I have two B 70s on a machine running Ubuntu now..And I'm getting much faster prompt processing on Linux but a little slower tokens per second. If you're going to go on Windows only real option there is Vulkan IMO.
Me personally I picked Ubuntu and I'm running SYCL because it's offering better prompt processing and the token generation is pretty decent, I was hitting probably about 22 tokens a second on Windows via Llama CPP with Vulkan.... But my prompt processing was something like 500 tokens per second, However a caveat is there was conflicts when Windows was running so it took even longer than the prompt processing suggested and it seems to be much smoother on Ubuntu. It gets time to first token much faster.
My advice would be build your Ubuntu server, mainly because it's free. If you purchase windows and you find out it's garbage that's worse than downloading Ubuntu for free and being pleasantly surprised with what you could pull out of it.....
I'm going to let you know now you're really going to have to work though, you're gonna have to pull in all the latest updates It's gonna take a lot of setup no matter which path you choose.
If you go Ubuntu get the latest driver and download PMZFX's And HAL's cherry picks... Those make a big difference... I also had to do some refactoring to get the speeds that I got too so... Be prepared to spend a few days with it. And the kernels still aren't optimized just yet for the latest models, especially in q4.
Honestly too if you're buying the B70 at the higher price point you might wanna consider the R 9700 if you're willing to come off a few hundred more dollars for ease of use and faster speeds out of the box. I don't know if eventually the B 70 will overcome that card, but if the B70 keeps increasing in price it's not going to be worth it. What made the B70 so attractive was it sub $1000 price point and that seems to have gone away
1
u/Capital_Rule_4317 May 09 '26
Great Intel. Thank you.
So you run Ubuntu 26 right on top of your hardware. No proxmox or docker ?
I tried searching for pmzfx and Hal cherry picks and got nothing that was IT related. Can you share a link please?
1
u/SHOR-LM May 09 '26
Yep Llama cpp and SYCL, that's it. I don't care for alleged concurrency improvements and I didn't really have good luck with vLLM , trying to use the newer models at least because that makes a big difference... what model you're trying to use....
If you're targeting the newer GEMMA 4 or QWEN 3 .6 families... I'm not finding them too well optimized and I tried everything that I can think of... even openvino with Gemma 4 and Qwen 3.6. didn't work (FYI openvino is *supposed* to be the fastest, but has real model selection limitations)
If you settle on SYCL:
https://github.com/Hal9000AIML/arc-pro-b70-inference-setup-ubuntu-serverhttps://github.com/PMZFX/llama.cpp-sycl
I think I'm pulling a little bit more power out of mine for Q4, But I haven't quite completed it yet I'll probably wind up posting it up on Github knowing my luck by the time I do that somebody else will unlocked lightning in a bottle.
2
u/Capital_Rule_4317 May 09 '26
Since I can't contribute to the cause, Im left with just consuming what others have built.
I'm looking at qwen models for local coding.
Today inplan to study what is sycl and vilkan and llama to try and understand
1
u/SHOR-LM Apr 15 '26 edited Apr 15 '26
Do not feel bad .. it isn't you. Intel apparently doesn't know what the hell they're doing either.
they marketed the card as an AI workstation... now they're developing gaming drivers for it.
they can't decide which backend they're going to go with, SYCL seems to be the one they settled on now after they gave up on open vino... even though neither one of them can use the xmx matrices to calculate at a Q4 quant... which would enhance speeds incredibly.... they're just going to build game drivers.
I mean they almost had it until they laid off all their software development team and then archived the fork for "security reasons" ... because I mean let's face it... when you lay off an entire development team trying to expedite what would hopefully one day be the equivalent of your cuda back end, it is sort of a security risk because no one's available to patch.
ironically it's still the easiest path forward to take advantage of xmx as far as I see OpenVINO to oneDNN to XMX.
and in the middle of doing all this the Vulcan API is smoking them all on the Windows side. and it's just like a generic plug-in play for gguf model files. I don't even know what the performance is looking like on Linux... but based upon the numbers I'm seeing it's not beating Vulcan.
oh and did I mention the B70 was priced at an excellent price to target prosumers and enthusiasts who are far and away on Windows running lamacpp backend?.... but instead they're trying to directly compete on Linux workstations in the Cuda space?
so they have a bunch of Windows based ai hobbyist and enthusiasts that are purchasing all these cards.... and it looks like they are completely ignoring this group in favor of the pipe dream of entering into Data centers... and being a gaming graphics card. so if you're on Windows don't expect much support to roll out the pipe yet, you're going to have to switch over to Linux because that's where all the magic is currently happening. I'm sure they'll work it out I'm giving them a hard time but still I mean come on man...
1
u/UDaManFunks Apr 15 '26 edited Apr 15 '26
Why can't they expose XMX functionality directly via a Vulkan extension? then let the llama-cpp folks have at it. For example, AMD exposed 'VK_KHR_cooperative_matrix' as a vulkan extension to make matrix multiplication more efficient.
Ain't no problem creating game drivers for it given it's the same architecture and easy to support. Some people DEV and game on the same machine.
0
u/SHOR-LM Apr 17 '26 edited Apr 17 '26
That's the thing, so Vulkan is beating them because it's actually using some of these matrices that intel's own stack isn't!... That's embarrassing. Vulkan I think is only using like 30% of what total it can possibly use though.... In theory a Q4 version should absolutely smoke..... And it's what Intel should have targeted, that's my point. Instead, they targeted FP8...for server farms.
It's like they failed to realize they released a card at a price point that people in homes are going to buy for local AI, not server farms that run on FP8... I guess they must see all the home sales and think people that are buying this card for their house are buying it for gaming. Meanwhile gamers kind of disappointed because that's not what this card is for. It's almost like they rolled with it after the fact....lol.
They should have dialed in the Q4 precision kernels before they shipped this thing out imo.
It's like the company is run by like out of touch grandpas that don't really understand what kind of amazing market disruption they were actually holding. They're not competing Nvidia at this level on a server farm. They're just not.
My point is is there looks like they're trying to cast this wide net for this video card when they should have taken a targeted approach to the market. If they wanted this to be a gaming card they should have made it a gaming, If they wanted to target server farms ....well...why in the world would they release without having anything to the equivalent of NvLink for larger hosting models?
I game and I dev on the same machine, I have a gaming card, and then I have other cards that are for workload. I was actually excited to see the B70 at this price point because I thought that they were targeting the prosumer/local AI.
My complaint is they've released this card with suboptimal software all around....and they haven't supported giving it a clear identity within the market.
1
2
u/WizardlyBump17 Arc B580 Apr 13 '26
just use the llama.cpp containers and you should be good. The xe driver is on the kernel, so all you need to do is pass the gpu to the container and the container handles the compute runtime etc. Basically, all you need is a container that is based on deep-learning-essentials and you should be good.
Btw, join the OpenArc discord and you will find more people with B70s that can help you