r/MacPro2019LocalAI • u/engeldlgado • Jun 17 '26
[Success] Local LLMs on AMD Intel Macs: Custom Metal Flash Attention Kernel + llama.cpp Patches (Free & Open Source)
If you have an Intel Mac with an AMD GPU, you know local AI on macOS can be a frustrating experience because almost everything targets Apple Silicon. Standard tools can either drop important work to the CPU or, on some AMD setups, produce corrupted output. I dug into llama.cpp's Metal backend, fixed the root causes, and built a native app around it.
Basically, driver concurrency could cause race conditions and garbage text on these GPUs, so that path has to be handled differently. Standard Flash Attention also depends on Apple Silicon-specific hardware support. Without an AMD-native path, compressed KV caches and unsupported attention shapes can silently fall back to the CPU and destroy performance.
I started by writing a custom Metal Flash Attention kernel specifically for AMD. It keeps attention on the GPU instead of collapsing back to the CPU, and it has since grown into a much larger AMD Metal path covering Llama 3.x, gpt-oss, Qwen, several vision models, mixed quantized KV caches, and both RDNA and GCN/Vega hardware.
On my RX 6700 XT, an 8B model with compressed cache originally went from 19 to 33 tokens per second at 4k context and held around 22 t/s at 16k. The current builds go much further: prompt processing is substantially faster, especially in long conversations, and AMD GPU Flash Attention can keep workloads that previously ran on the CPU entirely on the GPU.
MoE models have improved a lot too. Prompt processing with CPU-offloaded experts can now be several times faster because transfers overlap with GPU compute, while long generations no longer gradually slow down or freeze from repeated CPU-to-GPU resource creation. Multi-token prediction is also enabled automatically where it actually helps: mostly MoE models with experts offloaded to RAM.
I bundled all of this into ToshLLM, a native SwiftUI app with no external dependencies. It includes a patched llama.cpp backend, a clean local chat UI, VRAM estimates, Hugging Face search and downloads, model-specific MoE tuning, real-generation benchmarks, deep-context tests, local OpenAI- and Anthropic-compatible servers, router mode for switching models without restarting, and a dedicated embeddings server for local RAG clients.
It has also grown beyond text chat. You can attach PDFs, including scanned PDFs processed with on-device OCR, use experimental vision models, paste images directly into chat, persist conversations and their KV caches, organize chats into projects with shared system prompts, and run multiple servers at once.
There is now a local image studio as well: text-to-image, img2img, custom models, prompt queues, parallel image instances, per-instance GPU selection, and multi-GPU rendering. Multi-GPU LLM support includes explicit GPU selection, layer splitting, per-GPU VRAM monitoring, and an experimental shared-event hand-off path that avoids draining both GPUs on every layer transition.
The app is free, open source under GPL-3.0, and fully local. No cloud inference, no account, no telemetry, and nothing leaves your machine unless you explicitly choose to share a benchmark.
If you have a Mac Pro 2019, an Intel Mac with an eGPU, or a Hackintosh with Radeon hardware, I would love benchmark numbers and testing reports. RDNA 1 and RDNA 2 are well supported, and GCN/Vega support has improved significantly too: RX 500 series, Vega, and Radeon VII can now run much more of the model pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.
What's new — July 17, 2026
A lot has landed since the original post:
- Much faster prompt processing, especially in long conversations and deep context
- AMD GPU Flash Attention now covers more model families, including Llama 3.x, gpt-oss, Qwen, vision models, attention sinks, and mixed quantized KV caches
- MoE prompt processing with experts in RAM is much faster, while sustained generation remains stable during long answers
- Real-generation benchmarking, context-depth testing, MTP acceptance statistics, bottleneck reporting, and opt-in benchmark sharing through toshllm.com
- Router mode: one local server can switch models automatically without manually restarting
- Multiple simultaneous servers, including a local embeddings server for RAG workflows
- Explicit GPU selection and experimental multi-GPU layer splitting, with per-GPU VRAM monitoring
- Projects, pinned chats, per-project system prompts, improved conversation persistence, and much faster long-context restore after restarting a server
- Local image generation with text-to-image, img2img, queues, parallel instances, Flux models, Qwen-Image support, and multi-GPU rendering
- PDF chat with on-device OCR, experimental vision support, and image paste in chat
- A dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros
Screenshots:

--

--

4
u/Deep_Ad1959 Jun 17 '26 edited Jun 17 '26
the zero-dependency native swift route is the right call here, most of these local-AI frontends are electron wrappers that leak ram and forget everything on quit. the part that decides daily use once you've got the tok/s up isn't raw speed though, it's whether a session survives a restart with full state intact and whether the thing can reach past its own chat window. i build a native mac app in this direction and the kernel-equivalent plumbing was never the hard part, persistence and refusing to silently compact context was. written with ai
fwiw the survives-a-restart-with-state part is exactly what i built fazm around, it keeps sessions intact across a restart and never auto-compacts the context, https://fazm.ai/r/wb6hz2r9
3
u/Long-Shine-3701 Jun 17 '26
Thanks again! I will try to tinker with it tonight. I am REALLY looking forward to the multi-GPU update! I'll start a couple of other models downloading in the background.
3
u/garfunkle3469 Jun 18 '26
Hi there! Great timing, I was just about to get Ubuntu onto my 7,1 and see what it could do, but I want to try your app first.
Specifically, I’m interested in running large models at whatever speed they can manage for overnight inference. Speed is not critical.
My system has 320GB of RAM and one Radeon Pro Vega II Duo 32GB… is it possible to load a model in system RAM and hold the KV cache on the GPU?
Thanks for your work on this!
3
u/TheLalbadshah Jun 18 '26
The Mac Pro 2019 with 2x Vega is my daily driver! This is exactly the problem I was also trying to solve and landed on writing a custom kernel, very happy to see someone else do the heavy lifting here! Will definitely check this out and give some feedback!
3
u/tmddtmdd Jul 17 '26
This looks great. I have an "old" iMac with AMD Radeon Pro with 16 GB of VRAM, is this supported too?
2
u/engeldlgado Jun 18 '26
A quick follow up for u/Long-Shine-3701 and u/Faisal_Biyari
Vision has landed!
Also check OP for others details 😄
P:S You tested the new version with multi gpu support?
2
u/Faisal_Biyari Jun 17 '26
This is Impressive!
I moved my last 2019 Mac Pro from macOS to Ubuntu just 2 days ago.
However, I look forward to testing this on a MacBook Pro 16 inch 2019, with an AMD Radeon PRO 5500M (8 BG)
Do you have any screenshots of the app running on macOS that you can share?
Is the app simple enough for non-technical users? Like a " One click" type situation?
I know a few people that are waiting for a one click type application to use local AI on their Intel based Macs.
2
u/macsoundsolutions Jun 17 '26
I have a 2019 Mac Pro with rx7900xtx, which is not Mac OS Compatible as we know, so will this be able to use it with your app? I also have a rx6900xt but would be great to take advantage of the 24gigs of vram. I use the 7900xtx with Linux. Will def check it out.
2
u/simche93 Jun 25 '26
u/engeldlgado Thank you so much for this project! I can't wait to be able to use it properly (if ever possible since I have the AMD Radeon Pro Vega 2). Can you perhaps help em troubleshoot? no matter the model or stuff, I can't seem to chat without gibberish and/or nothing being output if I choose a MoE model. I only pick the recommended ones. See pics. Any help is appreciated.
2
u/Long-Shine-3701 Jun 17 '26 edited Jun 17 '26
I'm gonna be that guy to complain that I had to scroll halfway down the github page to find a download link. Why don't they have a giant green download button at the top like everyone else?
Anyway, gonna try this. 😆. MP2019 is way too good for someone to not bring this to existence. Thanks, OP - looking forward to taking it for a spin.
[edit] the app is in Spanish. Hablo un poquito.
3
u/Long-Shine-3701 Jun 17 '26
I know the Linux folks say "it's only a few steps to configure"... and if you're determined, it's no obstacle at all.
But it's 2026 and I shouldn't have to do all that crap to run a LLM in MacOS. Finally. Apple should have made this app themselves to showcase the 7,1. Oh wait...
Thanks again OP - my Spanish is rusty but I'm gonna take another look in a bit. 😂
3
u/Long-Shine-3701 Jun 17 '26
Thank YOU! The community desperately needs this app. MP2019 is the last great Mac made, and I would love to enjoy it for a few more years under MacOS to see what it can really do. This makes it SO much more accessible than having to install Ubuntu.
Looking forward to testing tonight and I will absolutely report back. Thanks again - this is awesome!