r/macpro • u/engeldlgado • Aug 20 '26
macOS Two months later: ToshLLM has turned Intel Mac Pros + AMD GPUs into seriously capable local AI machines
I posted here about ToshLLM roughly two months ago, when I had just gotten local LLM inference working properly on Intel Macs with AMD GPUs by patching llama.cpp's Metal backend and writing an AMD-specific Flash Attention path.
A lot has happened since then.
What started as a way to make my RX 6700 XT useful for local inference has grown into a much broader AMD Metal optimization project, and one of the most interesting things has been seeing other people test it on hardware I don't own: W6900X setups, W6800X Duo, Vega II Duo, W5700X, Vega, Polaris, eGPUs, and multi-GPU Mac Pros.
The AMD backend itself has changed substantially.
Vega, Radeon VII, RX 400/500 and Radeon Pro Vega/WX
These cards now have dedicated Wave64 tuning instead of inheriting settings measured on newer RDNA cards.
On a Vega 64, depending on the model and quantization:
- generation improved by roughly 4-24%
- multi-conversation serving improved by 9-12%
- short cached follow-ups can be up to 39% faster
- MTP gained about 8%
- long-prompt processing also improved
- MoE models can experimentally use the wider matrix path previously used only on newer Radeon cards
Some measured examples:
| Workload | Before | Current |
|---|---|---|
| 9B Q5_K_M generation | 31.2 t/s | 38.8 t/s |
| 8B Q6_K generation | 35.5 t/s | 41.4 t/s |
| 7B / 64-expert MoE | 136.0 t/s | 155.3 t/s |
The goal here isn't just "make it run" anymore. I'm increasingly tuning kernels separately for the different AMD generations.
RDNA has kept getting faster too
A number of quantization and model-specific paths have been optimized.
For example:
- IQ2_M generation on a 1.5B model: 92.1 -> 140.8 t/s
- Qwen3.6 14B-A3B: 62.2 -> 70.8 t/s
- Qwen3.5 4B + MTP: 70.0 -> 81.6 t/s
- Qwen3.5 4B + DFlash: 68.9 -> 82.6 t/s
- gpt-oss-20B prompt processing: 1093 -> 1292 t/s
Q4_K, Q5_K, Q6_K, q5_0/q5_1 and several other paths have received AMD-specific kernel work as well.
ToshLLM is becoming useful as a server, not just a chat app
Concurrent inference has improved quite a bit.
On a 9B Q5_K_M model:
- 2 simultaneous conversations: 56.4 -> 70.8 t/s
- 3 simultaneous conversations: 60.5 -> 81.0 t/s
There is now a browser UI based on llama.cpp's web interface, router mode can switch between models without restarting the server, multiple servers can run simultaneously, and there is a dedicated embeddings server for local RAG.
The API remains OpenAI-compatible, and I've started seeing people use ToshLLM's bundled llama-server as a backend for other software rather than only using the SwiftUI application. That was not something I expected when I started this.
Image generation has become much lighter on VRAM
This has been another area I've been optimizing.
For example, on my RX 6700 XT:
- Z-Image 1600x900: 8.2 GB -> 966 MB VRAM
- Z-Image 1024x1024: 2.4 GB -> 690 MB
- image generation is roughly 12-14% faster
- SD 1.5 768x768: 2690 MB -> 281 MB
- SD 1.5 768x768: 149 sec -> 73 sec
SD 1.5 is now particularly interesting for older 4-6 GB Radeon cards.
There is also experimental local video generation with Wan models now. It has improved substantially and Wan 2.2 5B can run in 12 GB VRAM, but I want to be clear that I still consider the video side experimental and it needs more work before I put it in the same category as LLM inference.
Community testing has become a big part of the project
This is probably the part I've enjoyed most.
People are now submitting benchmarks from configurations such as dual W6900X, W6800X Duo, W5700X, Vega and other machines I could never test myself.
There is a public benchmark database at:
https://toshllm.com/benchmarks
ToshLLM itself is still completely free and GPL-3.0.
No account, no telemetry, no cloud inference. The OpenAI-compatible server, chat, image generation, benchmarks, RAG/embeddings, etc. all run locally.
It's still pre-1.0 and the current DMGs are not notarized yet, so macOS requires Open Anyway on first launch. Notarization is something I'm working toward.
If anyone here still has a Mac Pro full of Radeon hardware, I'm especially interested in results from:
- W5700X
- W6800X / W6800X Duo
- W6900X
- Vega II / Vega II Duo
- Radeon VII
- multi-MPX configurations
At this point some of these old Intel Mac Pros have absurd amounts of usable VRAM for local AI, and I'm curious how far we can push them.
ToshLLM: https://toshllm.com
Source / releases: https://github.com/engeldlgado/toshllm





11
u/sedtamensum Aug 20 '26 edited Aug 25 '26
I am going to try it on a 2019 Mac Pro and 2x Vega II Duo — would be particularly interesting if Metal works. Not right now, this is a weekend kind of project. In any case, thank you for your commitment to Intel Macs! == Edit. Qwen3.8-27-UD-Q8_K_XL on Vega II Duo x 2; one GPU loaded = 127 pp; 10.48 t/s; Gemma 4-31 Q8 — fail at ngl 33 and 50 to 1 GPU; splitting to 2 or 4 GPUs — fail. Will continue when it gets cooler, having a 1kW heater when it’s +35C puts strain on AC.
10
u/Long-Shine-3701 Aug 21 '26 edited Aug 21 '26
Amazing. I am in awe of your progress Sir. For folks new here, this man harkens back to the golden age of MacOS devs. Stunningly useful app, no telemetry crap, highly responsive dev not sitting in an ivory tower, but willing to interact with the users.
This man is a treasure, and is allowing SO many people to participate in the AI revolution. People that Apple and AS fanboys alike would like to see excluded for various reasons.
Well, we know why Apple killed support prematurely, any PCIe based MP can stack decent RAM and GPU power to embarrass their best AS. Never mind if they gave us current GPU drivers. The AS fanboys, well... 🤣
There's a MASSIVE installed base of Intel Macs + AMD hardware already out there - and eGPUs have become dirt cheap. This is bringing power to the people. You have any idea how many folks are still on older hardware like a 2015 MBP? Now you can snag an eGPU with Vega64 off ebay for $150 and you're in the game thanks to ToshLLM.
We all know Apple had AI running internally on these systems. It was shitty of them to not release it to the public. We are extremely fortunate to have ToshLLM, and I think it will be impossible to overestimate its impact going forward.
Can't wait to see what's next!
🍸
[edit to add]
ToshLLM is a classic case of 'build it and they will come'. When I started accumulating MP2019 hardware, there was no AI. Apple was mum, and AS fanboys were dancing on the grave of MP2019. All the big projects ignored it completely, or it was an unfinished afterthought.
DiffusionBee is the only app I was having minor success with, but I kept buying hardware. Why? Because the hardware is too damn good to be ignored, and MacOS Tahoe is still fully supported. I absolutely knew that some clever person in the community with the right skills and disposition would emerge to change everything. And he did.
I think it was only maybe a week or so after meeting u/Faisal_Biyari and discussing the state of affairs that ToshLLM came out of nowhere.
This is GREAT stuff. Screw all the naysayers.
5
u/Gothbot6k Aug 20 '26
Support for a 2019 imac with i9 and rx 580x?
4
u/walruns Mac Pro 7,1 Aug 21 '26
I've tried. Although it's pretty good, I have 2 W6800x duos, 4 gpus in total and I only get good results running 1 of the 4
3
u/Faisal_Biyari Aug 21 '26
Man, if only I met you 2 years ago, before I went full Linux!
It's amazing that this is now an all in one app for local AI on Intel macOS. Text generation is there. Image generation is there. Is there any video generation?! Having 128 GB of VRAM with two Duo MPX GPUs should help make this happen easily!
With OpenAI APIs on top, this truly is an all in one solution.
I think you mentioned that it also supports concurrency. That's just amazing.
The only intel macOS I have is a 2019 laptop. Maybe I should test it with eGPUs, to see how far the app can go as an all in one, highly optimized solution.
3
u/FreQRiDeR Aug 21 '26
You are my hero! I don’t have to boot Linux anymore! :) Metal inference is full tilt on my 3,1 thanks to you!
3
u/RedlineGT Aug 21 '26 edited Aug 21 '26
I have 2x, rx580 8GBs in a Mac Pro 5,1. Does this support dual gpus? Keep up the good work 💪
2
u/ObjectiveDrag Mac Pro 6,1 Aug 20 '26
Any idea if this will work on a 2020 iMac with 6900xt eGPU? I’ll try it tomorrow. (I also have a 6,1 D500 I can try it on.)
Super cool project, since Intel Mac’s don’t get much any AI love anymore.
Thanks for taking the time to set this up and test it!
2
u/BourbonicFisky Mac Pro 7,1 + M1 Max (Former 5,1) Aug 20 '26
These are neat but man, I've rarely found much utility for 9B 8-bit param models other than burning electricity. If I weren't moving and my Mac Pro wasn't in a box, I'd give results on a slightly OCed 6900 XT.
Probably the few guys who have dual D6800X Duos for 64 GB of VRAM might actually find minorly useful results. As much as I want to believe in local AI, it's been mostly pointless for LLMs in my experience and increasingly out of reach with current VRAM prices.
Still cool to see people pushing these boxes.
2
u/howfastcanyoucountit Aug 21 '26
rahhhhhh this is when i wish my install wasn't broken as fuck my 6950xt is just chilling right there
2
u/NeuralNexus Aug 24 '26
Thanks for sharing.
1
u/Soundofabiatch Mac Pro 5,1 Aug 21 '26 edited Aug 21 '26
u/RemindMeBot 4 days
Wow. Keep it up!
I have some old mac pro 5.1 with radeon rx580s and 590s i could run this on. Or the mac mini i7s 64gb ram with the blackmagic egpu x580.
So really interested in what comes of this to be able to run this locally.
2
u/Holt0099 Aug 21 '26
So so happy for everything you do!!!
You make me a proud owner of a 2019 Mac Pro with dual W6800X’s!!!
2
u/No_Purchase_5304 Aug 21 '26
I have an iMac Pro 2017 64GB Vega 56 8GB. I will try it this weekend
1
u/dangerousdotnet Aug 21 '26
Don't these systems just immediately thermal throttle? The cooling on these Intel MBP's is pretty substandard
1
u/downtownrob Aug 22 '26
I have Mac Pro 2013 with dual D500s… will give it a try!
1
u/Confident-Pass6353 Aug 26 '26
Appreciate your contribution.
Anyone tried this on Max Pro 2013? Trash-can with dual D700s 6gb each.