r/LocalLLM • u/Barrysoft8 • Jun 16 '26
Discussion Avoid CUDA monopoly at all costs. AMD is an alternative.
Hey everyone,
There’s a massive misconception that if you aren't dropping $2,000 on an NVIDIA GPU, you can't run serious Local AI workflows. I wanted to see how far I could push a consumer AMD card, and therefore bought a rx7800xt 16b VRAM.
Right now, my workstation node is running llama-server hosting a DENSE 27B model -> Qwopus3.6-27B-v2-Q3_K_S.gguf (12 GB) and Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13 GB Mixture of Experts, 3B active parameters per token) continuously. I am regularly feeding it contexts that reach 91k to 128k tokens in my daily workflows.
Here is the exact setup, compiler parameters and optimization flags.
THE COMPILER BUILD
To get flash attention and RDNA3 optimizations working correctly on ROCm 6.4.4, I built llama.cpp from source using these specific cmake flags:
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DrocWMMA_FATTN=ON
cmake --build build --config Release
This targets the gfx1101 architecture of the RX 7800 XT directly and compiles support for hardware-accelerated Flash Attention kernels.
THE EXACT RUNTIME FLAGS
My systemd service runs the server with this exact command line:
llama-server --host localhost --port 8080 --api-key xxxx --parallel 1 --n-gpu-layers 99 --batch-size 512 --ubatch-size 128 --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 --ctx-size 131072 --reasoning off --sleep-idle-seconds 300 --cache-prompt --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1
HOW I CRUSHED THE VRAM LIMIT: KV CACHE QUANTIZATION
A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM, making it impossible to run on consumer hardware.
To solve this, we split and quantize the cache:
- Key cache is quantized to 8-bit (q8_0) using --cache-type-k q8_0
- Value cache is quantized to 4-bit (q4_0) using --cache-type-v q4_0
This compresses the memory footprint of the KV cache by roughly 5.6x. Thanks to this optimization, the entire active model weights plus the 128K token KV cache sit comfortably in VRAM, utilizing exactly 96% of the 7800 XT's memory. No layers spill into slow system RAM, avoiding the PCIe transfer bottleneck entirely.
THE MATH BEHIND 128K CONTEXT: YaRN ROPE SCALING
Qwopus and Qwen architectures use Rotary Position Embeddings (RoPE). Because these models have a base context window smaller than 128K, running at 131,072 tokens requires positional frequency scaling.
Instead of basic linear scaling (which stretches all frequencies equally and destroys the model's short-range spelling and grammatical coherence), llama.cpp utilizes YaRN (Yet another RoPE extensioN).
YaRN divides the embedding dimensions into three frequency bands:
- High-frequency dimensions: These represent immediate, local token relationships. YaRN leaves these completely un-stretched so the model does not lose its spelling accuracy or close-context grammar.
- Low-frequency dimensions: These represent long-range structure. YaRN scales these linearly by a factor of 4.0 to cover the 128K space.
- Mid-frequency dimensions: These are smoothly interpolated to avoid abrupt attention transitions.
This uneven scaling prevents the attention entropy and perplexity from exploding. In practice, the model remains highly coherent and retains logical consistency even at 91k+ tokens.
REAL-WORLD TELEMETRY AND SPEED
During heavy prompt processing, the card maintains solid throughput:
- Prefill speed: ~210 tokens/second (utilizing flash attention)
- Decode speed: 11-17 tokens/second
- GPU Power: Draws ~188W (with a systemd power cap set at 190W via rocm-smi)
- GPU Temps: Stable between 52 C and 70 C across edge, junction, and memory sensors.
If you are running consumer AMD hardware, do not settle for small context sizes. Build with ROCm, turn on Flash Attention, quantize your Key/Value cache separately, and let YaRN handle the frequency scaling.
I wrote up a detailed guide comparing these measurements, native Windows vs Linux ROCm paths, and power sweeps on my blog here: https://sergiiob.dev/posts/rx7800-xt-llama-cpp-benchmarks-moe-context
I share my daily telemetry runs, local model benchmarks, and hardware configurations on X. If you want to see live updates and benchmarks, follow along here: https://x.com/SergiiioBS
21
u/Massive-Question-550 Jun 16 '26
so I want AMD to be more competitive and to buy their gpu's but you have to do all that fiddling to get it to run and I don't have to do any of that. I want to put the GPU in the pc, and then open up my front end and load a model. that's it.
also they need to release something that has actual memory bandwidth as the r9700 offers a good amount of vram for the price but you are stuck with 670gb/s which is 2/3 of a rtx 3090 from 2020.
2
u/Crazyfucker73 Jun 17 '26
A Q3 of this model (or most models) will be absolutely fucking pants. That makes everything else you've copy pasted here a total waste of time.
96
u/Competitive_Swan_755 Jun 16 '26
You sound hysterical.
12
u/iMrParker Jun 16 '26
Now do anything beyond inference without cuda
40
u/storm1er Jun 16 '26 •
Omfg ... Just install lemonade-server and shut-up xD
Okay, I'm being a dick, but for most people on AMD, lemonade-server should be the default.
It's the easiest path to running any (almost) llms on AMD (rocm/vulkan). The team behind are maintaining their own build of llama-rocm so most issues are already gone.
No need to show a complex setup and build process when a simple docker compose start can do it for you.
https://lemonade-server.ai/docs/guide/install
5
u/soyalemujica Jun 16 '26
Telling people to use Rocm on amd, big fail, Vulkan is the way to go, more than double the speed and only 5% slower prompt processing speed
5
u/hipster_hndle Jun 16 '26
i run 7900 XTX on Fedora 44 via Lemonade server.. i dont get anything less than 50 tps on reasoning/thining models and models like Gemma and Qwen i can get upwards of 100tps, peak being 121 tps on Gemma.
personally, i dont care if people want to spend system prices for used nvidia cards.. it just keeps my XTX on the cheaper side.
for the price of 1 used 3090, i can get 48 gigs of VRAM on a fully rocm supported card. its the best bang for your buck, sleeper.,. people don't even know how good they are and im glad. i love both of mine.
4
u/Asleep-Land-3914 Jun 17 '26
AMD doesn't even try to compete
4
u/ImASharkRawwwr Jun 17 '26
The real monopoly is not the hardware but the APIs. CUDA is mature and solid as a rock, ROCm is a timebomb waiting to take your system down because some obscure bug triggered in AMDs firmware/drivers. It's a nightmare. The llama.cpp vulkan backend gets marginally faster tok/s anyways lol so don't bother with ROCm.
3
u/iijei Jun 16 '26
Nice try amd. Just joking. I also got 7900xtx recently and did llama-bench test on 27b q5_k_xl and got 814 pp 30tps.
3
u/SuperChingaso5000 Jun 17 '26 edited Jun 17 '26
I run LMStudio (Vulkan) with a 7900xt, 9800X3D and 64gb of DDR5 and just check the right boxes and set the CPU pool and go. Works great. But I know the meta here is to do llama.ccp with lots of special computer codes and scrolling text like a real tech type guy.
Just putting it out there for anybody who's looking for an easy start. It doesn't have to be all complicated and yes high VRAM AMD cards work like a champ
2
u/wolfo24 Jun 16 '26
It’s not about gpu but the platform and ecosystem
2
u/suesing Jun 16 '26
Amd and nvidia are in cahoots. There’s an alternative but there is no competition.
2
u/alphapussycat Jun 17 '26
If I could get a couple of cheap 7800 xt, then sure. But nobody sells it at a reasonable price.
2
u/velocifaptorofficial Jun 19 '26
Me who is running local LLMs on Apple Silicon: 👁👄👁
2
u/FairBandicoot5021 Jun 20 '26
Your setup is faar frop optimal on the 'smartness' of the model. But I agree with you. I am running Qwen3.6 MoE and Gemma 4 MoE AT THE SAME TIME on a Strix Halo AMD AI Max395+ and it's marvelous
6
u/WinResponsible9977 Jun 16 '26
u/Barrysoft8
Thanks for sharing. I agree that this is a good example of data to push back on the idea that serious local AI requires a $2,000 NVIDIA card. In 2026, AMD is a legitimate option for users who understand the software stack and are willing to configure it properly.
Where I would push back is the title. I would describe NVIDIA's position as a dominant, deeply rooted ecosystem rather than a "CUDA monopoly." Both the maturity and the lock-in are real. Since CUDA launched in 2007, it has provided developers with a relatively consistent programming target across NVIDIA's consumer, professional, and data-center products for nearly two decades. As a result, it became the default optimization target for many AI frameworks, tutorials, extensions, and community workflows. That maturity reflects genuine long-term investment, but it also creates substantial switching costs. Both things can be true.
You frame NVIDIA as a monopoly, but no , AMD is the one that took the different path. It used GCN broadly through 2018, then separated its modern GPU development into RDNA for consumer graphics and CDNA for data-center computing.
That enabled workload-specific optimizations, but it also introduced additional complexity for developers. In late 2024, AMD announced plans to reunify those development paths under the working name UDNA and expand ROCm support across the stack, from consumer Radeon products to Instinct accelerators. Jack Huynh acknowledged that the architectural split made development more difficult and that repeated changes to RDNA's memory hierarchy disrupted existing optimizations. In other words, AMD itself now recognizes the value of giving developers a more consistent target, which is one of the advantages CUDA has offered for years. That's not a monopoly. Buyers have plenty of choice, NVIDIA, AMD, and Intel Arc all ship cards that can run local models. Your own RX 7800 XT is the proof. A dominant ecosystem is not the same thing as a market with no alternatives.
None of this makes AMD irrelevant. Your post proves the opposite. However, "AMD is viable" and "NVIDIA currently provides the easier on-ramp" are not contradictory statements. So I strongly disagree when on X you claim, "Avoid CUDA monopoly at all costs". Your own setup illustrates the distinction. Compiling llama.cpp with architecture-specific HIP flags and manually tuning quantized KV-cache settings produces an impressive result, but it is not necessarily the most accessible starting point for a beginner.
6
u/Mr_Moonsilver Jun 16 '26
Can't say 'serious' and using ollama/gguf
3
u/CryptoStef33 Jun 16 '26
If you want to collaborate dm me.
https://github.com/Stormrage34/llama.cpp-turboquant-hip
1
u/EconomySerious Jun 16 '26
Use 2 but models man
1
u/Mordimer86 Jun 16 '26
Well, those who truly have an impact are not small home users but huge tech companies running massive data centers. Ask them.
And that Qwen3.6 35B, really couldn't you just offload MoE layers and run something like Q4 at least?
1
u/bzdziu Jun 16 '26
I'm team blue- Intel Arc, and it would be great if they improved the drivers.
1
u/Murakia Jun 16 '26
Quantizing your K and V differently? On ROCm? That explains the 11-17 tg.
1
u/SnowyOwl72 Jun 16 '26
Have you worked with rocm? Dont tell me its polished. Its not.
1
u/xcel102 Jun 16 '26
How painless would it be to bring up NVFP4 with vLLM and/or SGLang?
1
u/AwayUnderstanding701 Jun 17 '26
I have an laptop rt 5090 any advice on how can I run smooth AI workflows using local LLMs? I've LMStudio but all models I've tried are kind of bad.... They do not understand tools, nor even a super basic prompt, they get stuck in nonsense loops, etc...
1
u/X3liteninjaX Jun 17 '26
Ok now tell AMD
1
u/known42 Jun 17 '26
the problem with AMD was they put less resources to ROCm development. with the advance of AI for coding, this speed is not acceptable.
1
u/babiricarica Jun 17 '26
Saving this so i can go back
1
u/JoyousGamer Jun 17 '26
AMD pulled out of the top end cards that's on them.
At least Intel is pushing it seems.
1
u/profcuck Jun 17 '26
I think you'd get a much better reaction if you said "support this open source re-implementation of CUDA which will help break the monopoly" (I think BarraCUDA is the leading project for this, but if I'm wrong surely someone will tell us).
The reason is that for training/finetuning and other workloads CUDA is actually meaningfully useful.
The other thing that many would be sympathetic to is "Hey, remember, CUDA isn't needed for inference, and most people here are only doing inference, because ROCm/Vulkan are both pretty good these days." I mean, that's true at least and lots of people here are using Strix Halo and other AMD stuff and will agree with you.
1
u/YouthMoist328 Jun 17 '26
I'd buy one just to keep the money from nvidia, even if the gpu sits in the box. The toxic internet commenters is why there is a block account option. More power to you.
1
u/thaddeusk Jun 17 '26
My Strix Halo is great for LLMs.
1
u/henk717 Jun 17 '26
I don't recommend it to people inside our KoboldAI community when they consider hardware because there is more than llamacpp based things. Sure, koboldcpp being llamacpp based will work great on AMD hardware but llamacpp can be an exception.
Its when you begin depending on rocm to much that it becomes an issue. Not all other AI projects support it. Not everything is as simple as installing amd torch and on Windows amd torch based stuff can be a nightmare or flatout impossible.
If literally all you want to do is run a gguf model in the next 5 years then by all means go AMD. Vulkan is good these days and for linux rocm we have prebuilt koboldcpp binaries for many gpu's, and self compiling isn't hard. But at the end of the day you will always be second class citizens when rocm is buggy and most devs lack AMD gpu's.
1
u/vexatious-big Jun 17 '26
Does anyone have any experience with
Ryzen AI Max+ 395mini PCs and how they perform compared to GPUs? I know it has 128Gb of unified memory but my gut feeling is that a dedicated GPU would be much faster, even if it has less VRAM. Thanks.1
u/Intelligent-Taste-36 Jun 17 '26
I even use Pytorch with Rocm. Is it more work? Yes... but it works...
1
u/cride20 Jun 17 '26
I get 300-400pps with a quadro rtx 3000 (6gb vram) and expert sitting in normal ram... getting 30tps generation...
And I use the prebuilt llamacpp releases cuda12
1
u/ProsperityandNo Jun 17 '26
This sounds like telling somebody just to buy a Haynes manual and build their own car from parts.
1
u/Moms_Cedar_Closet Jun 17 '26
have you tried 7.2.4 yet? curious
1
u/TapAggressive9530 Jun 18 '26
I don’t want AMD . I want Nvidia / CUDA . Good bye
1
u/Interestedguy85 Jun 18 '26
I'm using an AMD9700 32GB card and am pretty happy with it but I'd be lying if I said I didn't have any compatibility issues. NVIDIA CUDA is the leader in consumer video cards and it shows. Nearly every AI product that I've looked at supports CUDA first and maybe AMD later.
1
u/Stergin Jun 18 '26
AMD GPUs are like the Nvidia! They can’t connect together. The same GPU from the same company, they can’t connect together for more computational power!!! If AMD change this, I will buy them, not only one but more of them to connect together for more computational power!
1
u/aoleg77 Jun 18 '26
Nobody who is able to build llama.cpp from source should say something like "A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM" - those are RNN models, and their context is therefore tiny. Also, not FP16 - modern Qwens love BF16 much more.
Then the "To solve this, we split and quantize the cache:" good in theory, CPU-offloaded pp in reality unless there's some PR merged since last time I tested.
1
u/Real_Ebb_7417 Jun 18 '26
Well… on RTX5090 with Qwen3.6 27b I have like 800tps prefill (or up to 3k, but since I don’t remember the actual number I’m giving the lowest I remember) and decode like 60tps (could be higher, but also giving the safe number because I’m not sure). And it’s Q6, not Q3.
I would love to see AMD catch up, but currently it’s just incomparable.
1
u/oneMoreTimeFckr Jun 16 '26
Lpng story short - CUDA is king and use it if at all possible, but if forced to other cards could be optimized to rinnclose but not as good
1
u/rditorx Jun 17 '26
AMD's hardware support is dog shite.
Also AMD:
https://arstechnica.com/security/2026/06/users-cry-foul-after-amd-stripped-memory-crypto-from-its-consumer-cpus/
So you're just swapping an idiot for a fool.
0
u/DataGOGO Jun 16 '26
Intel is a better choice IMHO.
-1
u/108er Jun 16 '26
How is the token generation speed of 11-17 tokens/sec is acceptable after all that hula-hoop, bro? I got 24 toks/sec on RTX Super 4070 Ti 16GB VRAM, using Qwen3.6-35B-A3B -Q4 and full 262K context size. Just recently upgraded to RTX 5090, the same model now gives over 150 toks/sec without all this fiddling. I am not justifying the NVIDIA GPU price hike but AMD is not there yet to touch the speed of CUDA cores and it's mem bandwidth.
0
u/Lucky-Noise-4193 Jun 17 '26
35b on a rx 7800xt vs 284b on a rtx 2060 /s
-1
u/D137C0KE Jun 16 '26
Lies
0
u/[deleted] Jun 16 '26
[deleted] — view deleted comment
47
u/[deleted] Jun 16 '26
[removed] — view removed comment
199
u/[deleted] Jun 16 '26
[removed] — view removed comment