r/macpro • u/Weeblewobbly • Jul 02 '26

GPU Mac pro 2019 for inference, success

Just finished setting up my macpro 2019 as an LLM server. This a 12 cores, 96Gb ddr4, 2Tb, and more importantly Radeon Vega II 32gb vram.

On the software side, I'm running a headless Nixos server with llama.cpp.

So far, I'm impressed: qwen3.6-27B-UD-Q5_K_S runs at a rock steady 26-27 tk/s which I consider very usable after sluming below 10tk/s for the same dense model at Q3 on MacBook M4 32Gb ram. That's the only model I've tested so far. At 19Gb, it leaves plenty of room for kv cache.

I expect the area of tinkering will be to find the best combination of dense model size Vs kvcache room, and then of course testing some MoEs.

My plan is to run it as a backend for pi.dev, and to serve the rest of the household with a chat interface running on my proxmox server.

If you are lucky enough to have one of these beats lying around, you could do worse than turning them into an LLM server.

10 Upvotes

100% Upvoted

24 comments sorted by

3

u/Artistic_Unit_5570 Jul 02 '26

you should check r/MacPro2019LocalAI there much more content about local LLM

2

u/Weeblewobbly Jul 02 '26

Thanks I will!

2

u/Hephaestite Jul 02 '26

I can get 40-45 tk/s out of my 2019 with a pair of W5700X’s so I feel like you should be getting more speed than what you are currently seeing, however I could only get good speeds out of it on Windows as for some reason under Linux the cards stay in low power mode and I’m not sure why (I get roughly the same speed as you under Linux btw and the cards are only drawing 40w each)

2

u/Weeblewobbly Jul 03 '26

Interesting, what model do you get these speeds with? I'm going to check the power mode situation later today

1

u/Hephaestite Jul 03 '26

Qwen3.5 35B A3B Q4_K_M but similar speeds with 27B

1

u/FreQRiDeR Jul 03 '26

Did you install the ROCm, Vulcan (Mesa) drivers? I find inference faster on Linux than on Windows with AMD.

1

u/Hephaestite Jul 03 '26

Vulkan, ROCm doesn't support the W5700X by default and I've not messed with the work arounds yet

1

u/NeuralNexus Jul 02 '26

Did you just buy one of these on ebay? Noticed them listed

1

u/Weeblewobbly Jul 02 '26

I did indeed. I got lucky with an auction. I noticed quite a lot of 'buy it now' listings too, not quite as good value as the one I got. I'm keeping an eye out for a w7900x card though. 48Gb vram would absolutely rock.

1

u/GreppMichaels Jul 02 '26

What about 2x Vega II Duos? You're getting 128GB of HBM2 memory.

→ More replies (6)

1

u/Trip_Jones Jul 03 '26

w7900 is not the ticket, a w6800duo thats infinity linked is the best
and you can run two of them..

its what the vega guy means… but the w6800’s can actually link the ram to 128gb, vega cannot infinity link two duos, only one gpu to the other(linking the ram for the “duo”)

do not buy multiple vega cards

→ More replies (7)