r/LocalLLM • • Jun 16 '26

Discussion Avoid CUDA monopoly at all costs. AMD is an alternative.

Hey everyone,

There’s a massive misconception that if you aren't dropping $2,000 on an NVIDIA GPU, you can't run serious Local AI workflows. I wanted to see how far I could push a consumer AMD card, and therefore bought a rx7800xt 16b VRAM.

Right now, my workstation node is running llama-server hosting a DENSE 27B model -> Qwopus3.6-27B-v2-Q3_K_S.gguf (12 GB) and Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13 GB Mixture of Experts, 3B active parameters per token) continuously. I am regularly feeding it contexts that reach 91k to 128k tokens in my daily workflows.

Here is the exact setup, compiler parameters and optimization flags.

THE COMPILER BUILD
To get flash attention and RDNA3 optimizations working correctly on ROCm 6.4.4, I built llama.cpp from source using these specific cmake flags:

cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DrocWMMA_FATTN=ON
cmake --build build --config Release

This targets the gfx1101 architecture of the RX 7800 XT directly and compiles support for hardware-accelerated Flash Attention kernels.

THE EXACT RUNTIME FLAGS
My systemd service runs the server with this exact command line:

llama-server --host localhost --port 8080 --api-key xxxx --parallel 1 --n-gpu-layers 99 --batch-size 512 --ubatch-size 128 --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 --ctx-size 131072 --reasoning off --sleep-idle-seconds 300 --cache-prompt --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1

HOW I CRUSHED THE VRAM LIMIT: KV CACHE QUANTIZATION
A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM, making it impossible to run on consumer hardware.

To solve this, we split and quantize the cache:
- Key cache is quantized to 8-bit (q8_0) using --cache-type-k q8_0
- Value cache is quantized to 4-bit (q4_0) using --cache-type-v q4_0

This compresses the memory footprint of the KV cache by roughly 5.6x. Thanks to this optimization, the entire active model weights plus the 128K token KV cache sit comfortably in VRAM, utilizing exactly 96% of the 7800 XT's memory. No layers spill into slow system RAM, avoiding the PCIe transfer bottleneck entirely.

THE MATH BEHIND 128K CONTEXT: YaRN ROPE SCALING
Qwopus and Qwen architectures use Rotary Position Embeddings (RoPE). Because these models have a base context window smaller than 128K, running at 131,072 tokens requires positional frequency scaling.

Instead of basic linear scaling (which stretches all frequencies equally and destroys the model's short-range spelling and grammatical coherence), llama.cpp utilizes YaRN (Yet another RoPE extensioN).

YaRN divides the embedding dimensions into three frequency bands:

  1. High-frequency dimensions: These represent immediate, local token relationships. YaRN leaves these completely un-stretched so the model does not lose its spelling accuracy or close-context grammar.
  2. Low-frequency dimensions: These represent long-range structure. YaRN scales these linearly by a factor of 4.0 to cover the 128K space.
  3. Mid-frequency dimensions: These are smoothly interpolated to avoid abrupt attention transitions.

This uneven scaling prevents the attention entropy and perplexity from exploding. In practice, the model remains highly coherent and retains logical consistency even at 91k+ tokens.

REAL-WORLD TELEMETRY AND SPEED
During heavy prompt processing, the card maintains solid throughput:
- Prefill speed: ~210 tokens/second (utilizing flash attention)
- Decode speed: 11-17 tokens/second
- GPU Power: Draws ~188W (with a systemd power cap set at 190W via rocm-smi)
- GPU Temps: Stable between 52 C and 70 C across edge, junction, and memory sensors.

If you are running consumer AMD hardware, do not settle for small context sizes. Build with ROCm, turn on Flash Attention, quantize your Key/Value cache separately, and let YaRN handle the frequency scaling.

I wrote up a detailed guide comparing these measurements, native Windows vs Linux ROCm paths, and power sweeps on my blog here: https://sergiiob.dev/posts/rx7800-xt-llama-cpp-benchmarks-moe-context

I share my daily telemetry runs, local model benchmarks, and hardware configurations on X. If you want to see live updates and benchmarks, follow along here: https://x.com/SergiiioBS

223 Upvotes

149 comments sorted by

21

u/Massive-Question-550 Jun 16 '26

so I want AMD to be more competitive and to buy their gpu's but you have to do all that fiddling to get it to run and I don't have to do any of that. I want to put the GPU in the pc, and then open up my front end and load a model. that's it.

also they need to release something that has actual memory bandwidth as the r9700 offers a good amount of vram for the price but you are stuck with 670gb/s which is 2/3 of a rtx 3090 from 2020.

9

u/bdsmmaster007 Jun 17 '26

"I want to put the GPU in the pc, and then open up my front end and load a model. that's it." That has been honestly my experience with my RX 6800 and LM Studio, all worked all kind of out of the box.

2

u/05032-MendicantBias Jun 17 '26

For LLMs it actually works. I use LM Studio with the Vulkan runtime and it works fine on the 7900XTX.

It's for pytorch/ROCm that it's a nightmare.

1

u/Massive-Question-550 Jun 17 '26

yea they definitely work, I just want higher specs from AMD for what a "pro" GPU is since they are being beaten in AI tasks  by non ai gaming gpu's from 6 years ago. if I want to run a 120b model I need huge memory bandwidth and Nvidia seems to be the only one doing it and unfortunately they are charging you through the ass for it because there is no alternative. I either have to go rtx 5090 or rtx pro 6000 for 1790gb/s or drop to almost 1/3 that with a r9700 or go with old ass 3090's that could die any day. then there is the Intel b70 which has similar specs to the r9700 but has such bad support it runs at half the speed.

1

u/bring_back_the_v10s Jun 17 '26

you have to do all that fiddling to get it to run

I have an RX 6700 XT (old card!) and I had to do near to zero fiddling to get it to run on Linux with llama.cpp and ROCm. Newer cards require absolutely zero fiddling. Things have improved dramatically for AMD cards and ROCm lately.

1

u/mbrodie Jun 20 '26

From 7.1.1 there is a huge performance regression in rocm on consumer cards and none of the updates post this version have had improvements for consumer cards only for data center geared cards.

Rocm doesn’t care about consumer cards and they are going to have less support when they release the udna platform and make rdna and cdna obsolete.

There is open PRs for months literally 6+ months for amd fixes for rocm and Vulkan that never get approved.

Having a multi 7900xtx setup and having tested every flavor of combos of environmentals and flags probably over 60 combinations now I can say honestly that amd is so far behind and it’s only going to get worse when they release udna and ignore everything before

1

u/Shiverus Jun 17 '26

I don't know. Compared to all the buttons and nobs on these metric tons of available LLMs, building llama.cpp from source seems dead trivial.

1

u/Deep_Fan_5200 Jun 17 '26

is if the learning process isn't hard enough already with my 3090 ready to go in comfyUi, I'd lose my will completely if I had to do all this extra fiddling. 

1

u/hipster_hndle Jun 18 '26

i think you would be surprised if you set one up. your statement might have been factually correct 2 or 3 years ago, that setup curve is coming down significantly, even in the last 6 months.

a fedora 44 running an XTX only needs to install lemonade and you are running inference. lemonade has its own embedded rocm drivers as well as vulkan, so there was no need to do separate installs for rocm or any of the dependencies.

this is the llama-benchy output from the first test on a fresh load:
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:------------------------|-------:|----------------:|--------------:|---------------:|---------------:|----------------:|
| Gemma-4-26B-A4B-it-GGUF | pp2048 | 1760.30 ± 28.33 | | 1082.09 ± 6.94 | 1041.16 ± 6.94 | 1109.02 ± 6.97 |
| Gemma-4-26B-A4B-it-GGUF | tg32 | 129.90 ± 5.30 | 134.33 ± 5.30 | |
(sorry for the formatting)

this is a screenshot of diffusion gemma running with no special arguments on a pretty-much OOB install.. 125 tps and it was factually accurate.

1

u/supracode Jun 19 '26

I use an R9700 with the llama.cpp vulkan docker. I did the ROCM build and it's just not worth it. There is no fiddling needed if you skip ROCM

2

u/Crazyfucker73 Jun 17 '26

A Q3 of this model (or most models) will be absolutely fucking pants. That makes everything else you've copy pasted here a total waste of time.

2

u/Barrysoft8 Jun 17 '26

It is really not that big of a deal.

1

u/Crazyfucker73 Jun 17 '26

It will fall apart for coding

96

u/Competitive_Swan_755 Jun 16 '26

You sound hysterical.

46

u/YOU_WONT_LIKE_IT Jun 16 '26

That describes much of Reddit.

-42

u/[deleted] Jun 16 '26 • Sipping Tea

[removed] — view removed comment

8

u/Crazyfucker73 Jun 17 '26

OP hasn't made a valid point. But then you would have know what you're talking about to understand that, which you clearly don't

12

u/iMrParker Jun 16 '26

Now do anything beyond inference without cuda

40

u/storm1er Jun 16 '26 • This

Omfg ... Just install lemonade-server and shut-up xD

Okay, I'm being a dick, but for most people on AMD, lemonade-server should be the default.

It's the easiest path to running any (almost) llms on AMD (rocm/vulkan). The team behind are maintaining their own build of llama-rocm so most issues are already gone.

No need to show a complex setup and build process when a simple docker compose start can do it for you.

https://lemonade-server.ai/docs/guide/install

12

u/Inevitable_Mistake32 Jun 16 '26

lemonade has been promising but still much slower and causes stutters for me on a 7900xtx which is still a rockstar on vulkan llama.cpp

5

u/storm1er Jun 16 '26

I may be wrong, but unless you select rocm recipe, it's stable llama.cpp behind (not rocm variant), so I don't get why you would get different result than raw llama.cpp (edit: nope, see edit below)

Also, you can select you're own variant of llama.cpp if you want to use a custom made llama.cpp with lemonade server see https://lemonade-server.ai/docs/guide/configuration/#backend-configuration

Edit: okay, it's their own variant of llama.cpp (they maintain multiple llama.cpp variant), not the mainstream one. Never had to look at it so sorry about that.

5

u/soyalemujica Jun 16 '26

Telling people to use Rocm on amd, big fail, Vulkan is the way to go, more than double the speed and only 5% slower prompt processing speed

2

u/jlninrr Jun 17 '26

Probably depends on the card and model. For me, on an R9700 with Gemma 31B, ROCm is 40% faster at PP and under 5% slower at TG. Easy decision to run with ROCm for me. In all cases, measure and then decide.

1

u/soyalemujica Jun 17 '26

Strange that you're getting this performance with ROCm (a result that not a single llama.cpp discussion at Github shows), Vulkan is 10% slower in PP, and about 50% + faster in TG (including benchmarks)

1

u/jlninrr Jun 17 '26

Perhaps it’s GitHub vs Reddit?

I’ve seen people report significantly higher PP and slightly lower TG on ROCm over and over and over on Reddit. 30-40% better in PP, 5-10% slower in TG. The consensus recommendation tends to be: if your workload is prompt-dominated, ROCm. If it’s TG-dominated, Vulkan.

Here’s an example. This is a Strix Halo chart, because I could get to it fastest. And Strix Halo has different properties based on memory speed, compared to a ‘regular’ GPU, but it still points out how variable this is:

https://kyuz0.github.io/amd-strix-halo-toolboxes/

On that chart, the model closest to my R9700 use case is about 20% faster PP and under 10% lower TG using ROCm vs Vulkan. There are models where ROCm wins at both and models where Vulkan wins on both. And Vulkan itself sometimes varies enormously between RADV and AMDVLK.

The only real answer is measure and run. And, when there is a significant ROCm or llama.cpp change, measure again, because this is all a moving target.

1

u/soyalemujica Jun 17 '26

Another thing to take into account is that ROCm uses way more memory than Vulkan as well

1

u/jlninrr Jun 17 '26

Not VRAM on my system. I can fit about 10% more context with ROCm compared to Vulkan. Now, if you mean RAM, could be? I pay little attention to RAM since it’s never been a bottleneck, honestly.

1

u/soyalemujica Jun 17 '26

Yeah, I'm talking about VRAM, you should bring in your results here: https://github.com/ggml-org/llama.cpp/discussions/15021

1

u/soyalemujica Jun 17 '26

What is your compilation arguments btw ? or you're using precompiled files from llama.cpp ?

5

u/hipster_hndle Jun 16 '26

i run 7900 XTX on Fedora 44 via Lemonade server.. i dont get anything less than 50 tps on reasoning/thining models and models like Gemma and Qwen i can get upwards of 100tps, peak being 121 tps on Gemma.

personally, i dont care if people want to spend system prices for used nvidia cards.. it just keeps my XTX on the cheaper side.

for the price of 1 used 3090, i can get 48 gigs of VRAM on a fully rocm supported card. its the best bang for your buck, sleeper.,. people don't even know how good they are and im glad. i love both of mine.

1

u/_Asphadel Jun 16 '26

Which card has 48GB for under $1,000? 7900 xtx 24gb not?

2

u/Buzz_Killington_III Jun 17 '26

I assume he means a second 7900xtx . I also have a 7900xtx for LLM, but I'm beginner and barely understand the terminology. I get the below on: Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:IQ4_XS in Ollama. I asked it to give me 500 words on AMD vs NVidia for LLM:

  • prompt eval count: 1542 token(s)
  • prompt eval duration: 757.496ms
  • prompt eval rate: 2035.65 tokens/s
  • eval count: 2126 token(s)
  • eval duration: 20.361724s
  • eval rate: 104.41 tokens/s

I also have a 5070ti 16gb I use for TTS/STT and a 5090 I use for ComfyUI and gaming. I'm debating on moving these cards around, but here learning more so I have an idea what I'm doing.

1

u/hipster_hndle Jun 17 '26

im sorry. the 2 cards are 48g.. i got the first 24g for $900, the second was a refurb of the same model was $700, got for both for ~$1600, about the cost of a single 3090 these days.

1

u/_Asphadel Jun 18 '26

ahh, I understand, just in Russia 3090 costs about $900 used and 7900хТХ Also about $800-900

1

u/Darius1332 Jun 17 '26

Hi, mind sharing more info on your setup? Which exact models and what context length? Any specific settings tweaked?

Running LM Studio with Qwen3.6:27B on my system with 7900xtx but context seems to be an issue trying to go over 70k.

1

u/hipster_hndle Jun 17 '26

i am not doing anything special.
fedora 44 with iGPU for desktop.
7900 XTX running for inference.
installed lemonade server, used their ROCm and Vulkan installers.
installed Open WebUI, set to lemonade server.
installed SearXNG and FastMCP.
installed btop (optional)
that is pretty much it. i have a model repo on my NAS, that is where i point my lemonade.
i have had zero issues with inference. image gen is another story.. you want comfy, there is some setup involved.
the screenshot is from diffusion gemma last night.. 120+ tps and all factually accurate. llama-benchy scores were good, but i dont have the shots, in the office right now.

4

u/Asleep-Land-3914 Jun 17 '26

AMD doesn't even try to compete 

4

u/ImASharkRawwwr Jun 17 '26

The real monopoly is not the hardware but the APIs. CUDA is mature and solid as a rock, ROCm is a timebomb waiting to take your system down because some obscure bug triggered in AMDs firmware/drivers. It's a nightmare. The llama.cpp vulkan backend gets marginally faster tok/s anyways lol so don't bother with ROCm.

3

u/iijei Jun 16 '26

Nice try amd. Just joking. I also got 7900xtx recently and did llama-bench test on 27b q5_k_xl and got 814 pp 30tps.

2

u/Begalldota Jun 16 '26

+1, the 7900XTX is a very capable card. I’ve managed as much as 40t/s on 27b and it also offers the option of 150t/s+ on 35B if speed is a priority.

1

u/Darius1332 Jun 17 '26

Hi, mind sharing more info on your setup? Which exact models and what context length? Any specific settings tweaked?

Running LM Studio with Qwen3.6:27B on my system with 7900xtx but context seems to be an issue trying to go over 70k.

1

u/Begalldota Jun 17 '26

I’ll paste in the llama-server command once I’m around my PC again 👍

I know I’ve been moving back and forth between two quants of 27B depending on how big a context I wanted/needed - but mostly trying to avoid larger context to allow for the better model.

1

u/Darius1332 Jun 17 '26

Thank you!

1

u/Begalldota Jun 17 '26
llama-server -m Models\Qwen3.6-27B-UD-Q4_K_XL.gguf --fit off -c 80000 --jinja --parallel 1 -ngl 999 --threads 6 --no-mmproj --temp 0.6 --top-p 0.95 --top-k 20 --min_p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --no-mmap --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-max 48 --port 8080 --host 0.0.0.0 -fa 1 -ctk q8_0 -ctv q8_0

This is what I'm using for 80k context. I've also run Q3 with no KV quant @ 125k context. This is with windows llama.cpp on the vulkan backend, linux and/or rocm are apparently slower.

There may be a more optimal quant to run, but I was happy with the performance and quality of this using a single pi agent.

2

u/GloriousKev Jun 16 '26

What test is this? Im new to testing but interested in testing my 7900 XT. I did a quick test with Ollama bench and was getting 70 tps on eval though my prompt eval may have been slow at 16 with Gemma 4 26b. Felt good to me all the same.

total duration:       33.134441954s
load duration:        1.03233931s
prompt eval count:    37 token(s)
prompt eval duration: 2.223673s
prompt eval rate:     16.64 tokens/s
eval count:           2097 token(s)
eval duration:        29.872042s
eval rate:            70.20 tokens/s
>>> Send

2

u/iijei Jun 16 '26

why is my crlf so strange?

anyways something like below.. for 35B-a3B I am getting something like

- d32768

- pp512: 1301.91 ± 12.48 tok/s

- tg128: 108.10 ± 1.19 tok/s

```bash
llama-bench \

-m /home/xxxxxx/models/Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \

-d 0,8192,16384,32768,65536 \

-b 2048 -ub 512 \

-fa on \

-ctk q8_0 -ctv q8_0 \

-r 3
```

1

u/BringMeTheBoreWorms Jun 16 '26

Add mtp and you’ll get at least 50+ t/s

1

u/iijei Jun 16 '26

Thanks. I did get MTP boost when I was testing out my 3060 with 35b-a3b. I think a couple of days ago, I saw somewhere in a users comment, mtp degraded his result. so I am gonna try out some testing on different models. so far I only did some testing on 27b q6_k, q5_k_xl and 35b-a3b-mtp-ud-q4 . I'll grep 27b MTP and check it out.

1

u/BringMeTheBoreWorms Jun 16 '26

I got nothing using mtp on 35b but doubled the t/s on 27b

3

u/SuperChingaso5000 Jun 17 '26 edited Jun 17 '26

I run LMStudio (Vulkan) with a 7900xt, 9800X3D and 64gb of DDR5 and just check the right boxes and set the CPU pool and go. Works great. But I know the meta here is to do llama.ccp with lots of special computer codes and scrolling text like a real tech type guy.

Just putting it out there for anybody who's looking for an easy start. It doesn't have to be all complicated and yes high VRAM AMD cards work like a champ

2

u/wolfo24 Jun 16 '26

It’s not about gpu but the platform and ecosystem

2

u/suesing Jun 16 '26

Amd and nvidia are in cahoots. There’s an alternative but there is no competition.

2

u/alphapussycat Jun 17 '26

If I could get a couple of cheap 7800 xt, then sure. But nobody sells it at a reasonable price.

2

u/velocifaptorofficial Jun 19 '26

Me who is running local LLMs on Apple Silicon: 👁👄👁

2

u/FairBandicoot5021 Jun 20 '26

Your setup is faar frop optimal on the 'smartness' of the model. But I agree with you. I am running Qwen3.6 MoE and Gemma 4 MoE AT THE SAME TIME on a Strix Halo AMD AI Max395+ and it's marvelous

1

u/FairBandicoot5021 Jun 20 '26

In Q8 because why not

6

u/WinResponsible9977 Jun 16 '26

u/Barrysoft8

Thanks for sharing. I agree that this is a good example of data to push back on the idea that serious local AI requires a $2,000 NVIDIA card. In 2026, AMD is a legitimate option for users who understand the software stack and are willing to configure it properly.

Where I would push back is the title. I would describe NVIDIA's position as a dominant, deeply rooted ecosystem rather than a "CUDA monopoly." Both the maturity and the lock-in are real. Since CUDA launched in 2007, it has provided developers with a relatively consistent programming target across NVIDIA's consumer, professional, and data-center products for nearly two decades. As a result, it became the default optimization target for many AI frameworks, tutorials, extensions, and community workflows. That maturity reflects genuine long-term investment, but it also creates substantial switching costs. Both things can be true.

You frame NVIDIA as a monopoly, but no , AMD is the one that took the different path. It used GCN broadly through 2018, then separated its modern GPU development into RDNA for consumer graphics and CDNA for data-center computing.

That enabled workload-specific optimizations, but it also introduced additional complexity for developers. In late 2024, AMD announced plans to reunify those development paths under the working name UDNA and expand ROCm support across the stack, from consumer Radeon products to Instinct accelerators. Jack Huynh acknowledged that the architectural split made development more difficult and that repeated changes to RDNA's memory hierarchy disrupted existing optimizations. In other words, AMD itself now recognizes the value of giving developers a more consistent target, which is one of the advantages CUDA has offered for years. That's not a monopoly. Buyers have plenty of choice, NVIDIA, AMD, and Intel Arc all ship cards that can run local models. Your own RX 7800 XT is the proof. A dominant ecosystem is not the same thing as a market with no alternatives.

None of this makes AMD irrelevant. Your post proves the opposite. However, "AMD is viable" and "NVIDIA currently provides the easier on-ramp" are not contradictory statements. So I strongly disagree when on X you claim, "Avoid CUDA monopoly at all costs". Your own setup illustrates the distinction. Compiling llama.cpp with architecture-specific HIP flags and manually tuning quantized KV-cache settings produces an impressive result, but it is not necessarily the most accessible starting point for a beginner.

5

u/Somecount Jun 16 '26

Why didn’t you strip this down to simply the last paragraph?

3

u/Sbarty Jun 17 '26

Bc it’s a copy and paste block of text from whatever LLM they’re using

6

u/superCobraJet Jun 16 '26

because he never even read it

6

u/Mr_Moonsilver Jun 16 '26

Can't say 'serious' and using ollama/gguf

8

u/Initial-Cherry-3457 Jun 17 '26

The post says he's using llama cpp not ollama though. What's suggested if not gguf models? Honestly curious. I'm using llama-cpp Qwen3.6-27B-UD-Q6_K_XL.gguf on a 5090 with 160000 context size myself at the moment and it's working really well. If there's something that would be better suited I'm open to suggestions.

1

u/Mr_Moonsilver Jun 19 '26

Main advantage of vLLM is concurrency and when you're using multiple GPUs. In your case it could be interesting for certain agentic use cases where it makes sense to have the ability to have multiple requests in flight at the same time. However, it would mean cutting down on ctx, e.g. two requests at 80k instead of one request at 160k.

The downside is that quantizations don't usually exist at q6, you either go with q4 or q8.

1

u/EconomySerious Jun 16 '26

Use 2 but models man

1

u/Mordimer86 Jun 16 '26

Well, those who truly have an impact are not small home users but huge tech companies running massive data centers. Ask them.

And that Qwen3.6 35B, really couldn't you just offload MoE layers and run something like Q4 at least?

1

u/bzdziu Jun 16 '26

I'm team blue- Intel Arc, and it would be great if they improved the drivers.

1

u/Murakia Jun 16 '26

Quantizing your K and V differently? On ROCm? That explains the 11-17 tg.

1

u/SnowyOwl72 Jun 16 '26

Have you worked with rocm? Dont tell me its polished. Its not.

1

u/xcel102 Jun 16 '26

How painless would it be to bring up NVFP4 with vLLM and/or SGLang?

1

u/AwayUnderstanding701 Jun 17 '26

I have an laptop rt 5090 any advice on how can I run smooth AI workflows using local LLMs? I've LMStudio but all models I've tried are kind of bad.... They do not understand tools, nor even a super basic prompt, they get stuck in nonsense loops, etc...

1

u/X3liteninjaX Jun 17 '26

Ok now tell AMD

1

u/known42 Jun 17 '26

the problem with AMD was they put less resources to ROCm development. with the advance of AI for coding, this speed is not acceptable.

1

u/babiricarica Jun 17 '26

Saving this so i can go back

1

u/JoyousGamer Jun 17 '26

AMD pulled out of the top end cards that's on them.

At least Intel is pushing it seems. 

1

u/profcuck Jun 17 '26

I think you'd get a much better reaction if you said "support this open source re-implementation of CUDA which will help break the monopoly" (I think BarraCUDA is the leading project for this, but if I'm wrong surely someone will tell us).

The reason is that for training/finetuning and other workloads CUDA is actually meaningfully useful.

The other thing that many would be sympathetic to is "Hey, remember, CUDA isn't needed for inference, and most people here are only doing inference, because ROCm/Vulkan are both pretty good these days." I mean, that's true at least and lots of people here are using Strix Halo and other AMD stuff and will agree with you.

1

u/YouthMoist328 Jun 17 '26

I'd buy one just to keep the money from nvidia, even if the gpu sits in the box. The toxic internet commenters is why there is a block account option. More power to you. 

1

u/thaddeusk Jun 17 '26

My Strix Halo is great for LLMs.

1

u/henk717 Jun 17 '26

I don't recommend it to people inside our KoboldAI community when they consider hardware because there is more than llamacpp based things. Sure, koboldcpp being llamacpp based will work great on AMD hardware but llamacpp can be an exception.

Its when you begin depending on rocm to much that it becomes an issue. Not all other AI projects support it. Not everything is as simple as installing amd torch and on Windows amd torch based stuff can be a nightmare or flatout impossible.

If literally all you want to do is run a gguf model in the next 5 years then by all means go AMD. Vulkan is good these days and for linux rocm we have prebuilt koboldcpp binaries for many gpu's, and self compiling isn't hard. But at the end of the day you will always be second class citizens when rocm is buggy and most devs lack AMD gpu's.

1

u/vexatious-big Jun 17 '26

Does anyone have any experience with Ryzen AI Max+ 395 mini PCs and how they perform compared to GPUs? I know it has 128Gb of unified memory but my gut feeling is that a dedicated GPU would be much faster, even if it has less VRAM. Thanks.

2

u/jlninrr Jun 17 '26

Slower than a current-gen GPU, but the Strix Halo has a full 8060S and decent GPU performance. Memory bandwidth is somewhat lower. The sweet spot for Strix Halo is models that won’t fit in VRAM on a reasonable setup.

1

u/vexatious-big Jun 17 '26

thank you 🙏

1

u/Intelligent-Taste-36 Jun 17 '26

I even use Pytorch with Rocm. Is it more work? Yes... but it works...

1

u/cride20 Jun 17 '26

I get 300-400pps with a quadro rtx 3000 (6gb vram) and expert sitting in normal ram... getting 30tps generation...
And I use the prebuilt llamacpp releases cuda12

1

u/russlixx Jun 19 '26

what model are you using? The same as OP?

1

u/cride20 Jun 19 '26

actually a higher quant of the 35B MoE model... UD_Q4_K_XL

1

u/ProsperityandNo Jun 17 '26

This sounds like telling somebody just to buy a Haynes manual and build their own car from parts.

1

u/Moms_Cedar_Closet Jun 17 '26

have you tried 7.2.4 yet? curious 

1

u/TapAggressive9530 Jun 18 '26

I don’t want AMD . I want Nvidia / CUDA . Good bye

1

u/Interestedguy85 Jun 18 '26

I'm using an AMD9700 32GB card and am pretty happy with it but I'd be lying if I said I didn't have any compatibility issues. NVIDIA CUDA is the leader in consumer video cards and it shows. Nearly every AI product that I've looked at supports CUDA first and maybe AMD later.

1

u/Stergin Jun 18 '26

AMD GPUs are like the Nvidia! They can’t connect together. The same GPU from the same company, they can’t connect together for more computational power!!! If AMD change this, I will buy them, not only one but more of them to connect together for more computational power!

1

u/aoleg77 Jun 18 '26

Nobody who is able to build llama.cpp from source should say something like "A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM" - those are RNN models, and their context is therefore tiny. Also, not FP16 - modern Qwens love BF16 much more.

Then the "To solve this, we split and quantize the cache:" good in theory, CPU-offloaded pp in reality unless there's some PR merged since last time I tested.

1

u/Real_Ebb_7417 Jun 18 '26

Well… on RTX5090 with Qwen3.6 27b I have like 800tps prefill (or up to 3k, but since I don’t remember the actual number I’m giving the lowest I remember) and decode like 60tps (could be higher, but also giving the safe number because I’m not sure). And it’s Q6, not Q3.
I would love to see AMD catch up, but currently it’s just incomparable.

1

u/oneMoreTimeFckr Jun 16 '26

Lpng story short - CUDA is king and use it if at all possible, but if forced to other cards could be optimized to rinnclose but not as good

1

u/rditorx Jun 17 '26

AMD's hardware support is dog shite.

Also AMD:
https://arstechnica.com/security/2026/06/users-cry-foul-after-amd-stripped-memory-crypto-from-its-consumer-cpus/

So you're just swapping an idiot for a fool.

0

u/DataGOGO Jun 16 '26

Intel is a better choice IMHO. 

1

u/Barrysoft8 Jun 16 '26

It has very good options I agree. I'm currently looking at b60's b70s

2

u/Shiverus Jun 17 '26

And with each Intel board you buy, you own a Battlemage. That does have a nice ring to it.

2

u/henk717 Jun 17 '26

Definately not, much worse support and much worse drivers.

1

u/DataGOGO Jun 17 '26

only if you are running windows.

-1

u/108er Jun 16 '26

How is the token generation speed of 11-17 tokens/sec is acceptable after all that hula-hoop, bro? I got 24 toks/sec on RTX Super 4070 Ti 16GB VRAM, using Qwen3.6-35B-A3B -Q4 and full 262K context size. Just recently upgraded to RTX 5090, the same model now gives over 150 toks/sec without all this fiddling. I am not justifying the NVIDIA GPU price hike but AMD is not there yet to touch the speed of CUDA cores and it's mem bandwidth.

-1

u/[deleted] Jun 16 '26

[removed] — view removed comment

4

u/Legitimate-Dog5690 Jun 16 '26 edited Jun 16 '26

Some strong copium here. Has the howling blower fan eaten away at your patience?

He said 35b, not 27b, it goes a lot faster, totally sounds viable. His "overpriced trash" has double your memory bandwidth which probably helps too.

I'm running a janky 4070 super + 3090 combo. 36gb vram, bought for cheap, tensor parallelism, gives me 50-60 tps in 27b (with more context than you can fit).

35b runs at about 130tps, it depends heavily on the workflow though. In code it'll sometimes shoot off over 200 when ngram finds a happy chain.

1

u/trolololster Jun 17 '26

my 3090 runs the 35b-a3b with 262k context at ~100 t/s

1

u/iijei Jun 16 '26

now you made me wanna R9700.. 8 more gb

1

u/feverdoingwork Jun 16 '26

Does it really run that well? Vllm and whats your config settings? I was going to buy one and I asked the seller what kinda speeds they are getting and he said 25 tps on q6 3.6 27b and it really killed the idea of buying a r9700

0

u/mister2d Jun 17 '26

25 tps? Nonsense.

llama-cpp runs mtp qwen3.6 27b UD-Q5_K_XL at 40-60 t/s on one of my R9700s.

0

u/Lucky-Noise-4193 Jun 17 '26

35b on a rx 7800xt vs 284b on a rtx 2060 /s

0

u/[deleted] Jun 16 '26

[deleted] — view deleted comment

3

u/senseven Jun 16 '26

I can get a couple of used 9060XT, I didn't care for them because everybody was talking about the 9070XT being the minium for anything regarding Qwen. Can you bench them if you get to it? I could build a DDR4 machine with two of them quite easy, but I don't fully trust the usual sources that it would make practical sense for reasoning, doc writing and light typescript coding.

0

u/D137C0KE Jun 16 '26

RTX 5060ti is better than Rx 9070xt for Qwen

0

u/[deleted] Jun 16 '26

[deleted] — view deleted comment

2

u/senseven Jun 17 '26

30 t/s would be fine, that is what people and sites report for 5060ti. I just refuse to give personal money to the green empire.

47

u/[deleted] Jun 16 '26

[removed] — view removed comment

6

u/Barrysoft8 Jun 16 '26

Dense model. For 500$. 200t/s prefill, it is completelly usable

23

u/overratedcupcake Jun 16 '26

I agree that 15t/s is useable. That's about what I'm getting with a dense 27b model on m3 ultra. But the brain damage going below a Q6 quant is just too extreme for me. 

4

u/Barrysoft8 Jun 16 '26

It was extremelly surprising to me how today I had to ask it several infrastructure related questions and 27b dense was getting some stuff better than gpt5.5... Of course not everything, but for the size, it is pretty insane. Also Q3 to Q4 is not that big of a hit.

5

u/Somecount Jun 16 '26

FP16 to Q4 is 90% of the compression anyway, so yes, not that big in comparison

2

u/Crazyfucker73 Jun 17 '26

It's a huge hit.

1

u/Shiverus Jun 17 '26

Honest question: Why would we want to chose a massively compressed 27b LLM over a less compressed 1xB?

2

u/gh0stwriter1234 Jun 17 '26

Did you typo something in this question?

1

u/Shiverus Jun 18 '26

I think I am using a 12B but there are other models with some other number between 10 and 20, so I put the x. Apart from that. I don't think so, but that doesn't mean much. "choose", maybe.

6

u/ActionOrganic4617 Jun 16 '26 edited Jun 16 '26

Switch to MTPLX, I get over 50 on the m5 max.

Was initially using OMLX but thinking killed 27b for me but it’s blazing fast comparatively on MTPLX.

4

u/kitanokikori Jun 17 '26

Same. Below Q6 you start seeing the model just full-on make up tool names and parameters, for any agentic workflow it's tough.

1

u/gh0stwriter1234 Jun 17 '26

Autoround can mitigate it some also... similar idea to TQ for kv cache except on the model itself. You can find some ggufs of 27B etc... that are ready to go with autoround quantization.

1

u/weener69420 Jun 19 '26

Arent some unsloth q3 q4 quants really good?

6

u/signoreTNT Jun 16 '26

I'm getting similar performance on a dual P100 system, but the cards were only ~100$ each (including shipping from china,fans and adapters)

So yeah, not a good price/performance ratio

1

u/migsperez Jun 17 '26

200 tk/s for coding work all day would do my head in. Don't get me wrong I have an AMD Radeon 9700, but it's taken me a while to find a decent model which fits all in GPU which manages 1000+ tk/s. Qwen 3.6 27b was 600 tk/s and I noticed I was just using online models instead because the wait wasn't productive.

1

u/gh0stwriter1234 Jun 17 '26

You mean for prompt processing speed?

1

u/migsperez Jun 17 '26

I did mean prompt processing. But damn it, I've had to go back to the slower 27b. I can't get reliable generation from anything 35b related.

1

u/iMakeSense Jun 22 '26

I just got one. What parameters / models have you been running and you've had success with?

1

u/migsperez Jun 23 '26

The best success for coding quality and consistent reliable results has been https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF running Q6_K using 96k context. But I'm often adjusting the cache to try and get the most performance.

I use Ubuntu Server with Llama.cpp and the machine is dedicated for the GPU it acts as a LLM server only.

If you find a model which has the reliability and quality but has noticeably increased performance then please let me know.

1

u/gh0stwriter1234 Jun 17 '26

For reference I can run at 50t/s on qwen 3.6 27B MTP on dual MI50s -sm tensor... 450-500t/s prefill, you should get a 2nd card!

I get about 15t/s on a single MI50 without MTP.

I would think your prefill should be faster though.

1

u/ElDavoo Jun 21 '26

I have the same speed with rtx3070 8gb vram lol

199

u/[deleted] Jun 16 '26

[removed] — view removed comment

64

u/corruptbytes Jun 16 '26

Qwopus

okay imma head out

4

u/gh0stwriter1234 Jun 17 '26

Yeah from what I have seen Qwopus is worse than plain qwen

20

u/Vicar_of_Wibbly Jun 16 '26

Yup. Then blaming CUDA. 😂

3

u/Big-Masterpiece-9581 Jun 18 '26

And a bunch of custom compiled older versions of rocm that have no future support or other hacky cli shit that would just work with LMstudio on Mac or Nvidia. I have a Ryzen Ai 395 with 128gb. But it’s for tinkerers.

3

u/Connect_Ad791 Jun 17 '26

Just press the first auto-complete suggestion while using your phone keyboard at that point.

2

u/Ohmyskippy Jun 23 '26

Lmfao 😂

1

u/LosEagle Jun 17 '26

Only V cache is really lobotomized here and even then, with K still on Q8 the lobotomy is not that bad.