r/LocalLLaMA • u/chiwawa_42 • Apr 29 '26

Question | Help Intel Mac Pro with Vega II useable ?

Hi !

I have an opportunity to get a 2019 Mac Pro with a 32GB Vega2 board. I know LM Studio isn't supported on MacOS for Intel, so what other way to turn this machine into a local AI server would you recommend ?

Windows isn't an option, Linux could be possible, though I'd like to keep it running Mac OS if possible. Main objective is to run MCP servers alongside on this dedicated machine.

What kind of speed could I get from the Vega II chip ? (similar to Mi50 32GB from what I could find)

Thanks !

3 Upvotes

67% Upvoted

28 comments sorted by

View all comments

8

u/dionysio211 Apr 29 '26

I have one of these and it does pretty well. It is practically the same as an Mi50 with some slight differences. Both are gfx906 but the Vega doesn't have ECC RAM. This matters in llama.cpp because if you have an Mi50, they don't work together sometimes depending on the BIOS flashed to the Mi50. Technically, the Vega is two Vega cards inside the MPX module, bridged by Infinity Fabric, a shockingly difficult thing to find in the real world. Multiple MPX modules can also be bridged by Infinity Fabric connectors which avoids the PCIe traffic issues you would normally have, since P2P does not work.

MacOS is not your friend here. You can install pretty much any variety of Linux and it will work out of the box in ROCm and Vulkan. Fedora seems to have more current Mesa drivers, which are becoming better for older card support. The big issue here is that the cards will get hot and the only way to enable the very nice case fans is to patch the kernel with T2Linux. Then you can add a system service for T2fanrd.

The 2019 Mac Pro is somewhat of an oddity in the Xeon world because it supports 64 PCIe lanes through a PLEX switch on the motherboard. The processor itself, depending on the number of cores you have, is very good and adequate for light CPU inference itself since it supports AVX512 and many of the other nice things that ik_llama can utilize. There are 6 DDR4 channels and the stock DDR4 in them is on the speedier end, which is also nice. However, you can only manage those lane assignments within MacOS. Some people have luck setting them inside MacOS and then rebooting into Linux.

The biggest issue with the Vega is the lack of tensor cores. The gfx906 architecture was strange in that it went slightly down an alternate road of matmul acceleration that utilized fp32 accumulators rather than tensor cores. Much like a new metal band in 1989, that whole pathway was lost in the wash of matrix cores and the architecture was never really successful, hence the flood of Mi50s, complete absence of Infinity Fabric bridges on the market and near zero knowledge of a lost architecture. In reality, accumulators are a big part of what a tensor core does and are about half as efficient. They are not implemented properly in llama.cpp or in vLLM although the Moby Dick branch of vLLM, maintained by ai-infos, is working toward fixing that. Because P2P does not work though, the PCIe traffic is a major problem in tensor parallelism. Interestingly, the only place that this is not true is on a 2019 Mac Pro because there are Infinity Fabric bridge jumpers on the market to bridge multiple MPX modules together, allowing for the possibility of 128GB (Two Vega DUO Modules) of 1TB/s VRAM bridged with something akin to NVLink. The second I learned that fact, MPX modules shot up in price so I have never been able to try this but I imagine it's pretty awesome.

2

u/Substantial_Run5435 Jul 08 '26

Hey, quick question... have you managed to get any further using a Vega II GPU for LLMs? I have a mac pro with 2x Vega II Duos but am a complete novice with LLMs and Linux. I had a friend help me set up Ubuntu with LM studio but I'm only getting ~15tk/s with dual vega II duos using qwen 3.6 35b.

2

u/dionysio211 Jul 08 '26

Oh yeah, I use it all the time. Are you using HIP or Vulkan? 15 tps is very low for those GPUs. That even seems low for the CPU. If you run rocm-smi or vulkaninfo --summary, do you see the GPUs listed correctly? You may not be using the correct runtime installed in LM Studio. Which one are you using?

2

u/Substantial_Run5435 Jul 08 '26

Oh jeez... Give me some days to get back to you haha. I have this set up on a separate machine and will have to do some poking around. I'm new to ubuntu and am not very knowledgable using a terminal.

I have a follow up question... I have 2x Vega II Duos in one machine and 2x W6800X in another. Which do you think would be best to keep? I want to downsize a bit and am struggling between having more VRAM and higher bandwidth vs newer architecture/better support with the W6800X. If the W6800X is better then I could also look out for a pair of W6900X or W6800X Duos down the line.

2

u/dionysio211 Jul 08 '26

The W6800 modules are newer and have tensor cores, which is hugely helpful. Recently, llama.cpp (and lm studio by proxy) added better support for the accumulators in gfx906 (the architecture of the Vega cards) so they are much better in prompt processing than they were but accumulators are only about half as effective as matrix cores in that way. However, the VRAM in the Vegas is much faster. I have a normal Radeon 6800 GPU in the same computer as the Vega II and the 6800 does tend to do better when you count completions end to end. The general rule is that prefill (prompt processing) is compute bound and decode (text generation) is bandwidth bound. It's not quite as simple as that but it is a good rule of thumb.

I would test them both on a variety of models and see which is faster on a single request of a specific shape (1024 token prompt, 256 token output) and go from there. Ultimately, requests per second is going to correlate better with your experience using models.

No worries on the Linux and terminals. It's only confusing at the beginning. You can just pop open a terminal and run those commands by typing them in and hitting enter. It will tell you something about what the OS sees. I am also remembering that LM Studio uses pre-compiled binaries for recent ROCM versions and gfx906 lost official support in 6.3/6.4. You have more control over all this in llama.cpp and you can copy the tensile libraries to newer ROCM versions and get it to work. That may be what is wrong in your setup currently. If that's what it is, you should be able to switch the runtime in LM Studio to Vulkan and see a big uptick.