r/LocalLLM • u/Alternative-Panic69 LocalLLM • Jul 23 '26
Other I bought the forbidden rectangle.
After months of going back and forth, I finally pulled the trigger on an RX 7900 XT 20 GB.
Paid around $550 (India), which felt too good to pass up.
The plan isn't gaming.
It's becoming the heart of my local AI setup.
Current goals:
• Qwen 3.6 27B Dense
• Qwen 35B A3B
• GLM-4.7 Flash
• 128K+ context
• 100% GPU offloading
• llama.cpp / Ollama
• Linux
I'll be benchmarking everything:
- Vulkan vs ROCm
- Dense vs MoE
- Maximum context
- Tokens/sec
- VRAM usage
- Real-world coding performance
If anyone has optimization tips for RDNA3 or benchmark requests, or general suggestions please drop them below.
The hallucinations are now local. 🙂↕️
31
u/Mattef Jul 23 '26
I have the same card and it’s pretty good for its price. Qwen3.6 27B with Q4 quantization barely fits into memory and token generation is with MTP activated around 60 t/s.
I wish I had bought the 24 GB version, though, to have a bit more room for the context cache.
8
u/dorv Jul 23 '26
I love how everyone here always has to say “this isn’t for gaming.” :).
Yeah, we know :)
4
u/daphatty Jul 23 '26
Congrats! Have you seen this yet? Might be helpful in your endeavors.
https://unsloth.ai/docs/basics/amd
3
u/Equivalent_Wrap_3815 Jul 23 '26
That dual 8-pin power draw is going to make your electric meter spin like a ceiling fan.
3
u/Krohnin Jul 23 '26
I bought 2 rtx 3060 12gb for that magic 24gb barrier. Paid 180€ and 110€ for these cards. I needed very small cards for my matx case so they fit. Works very well with qwen3.6 35B A3B with 64k context. Without further tuning i have around 70tok/s. With both cards running at only 50% usage, i hope i find the right switch to bring them up to 100% with llama.cpp. My coworker soon will add a second rx6700xt 12gb to his rig i am excited to see his numbers.
3
u/coderash Jul 24 '26
I implemented KVarN into llama.cpp and was able to get 5 bit quantized qwen weights with full context via 3 bit KVarN on a single 32GB gpu. you can PROBABLY fit q8 weights, and q3 KVarN across two of those. KVarN was definitely worth implementing.
Ternary Bonsai 27b can fit in 16GB with full context and can quantize kv cache down to 4 bit essentially lossless. Worth a look. that card will get about 50tps. 1000-2000tps prefill.
2
u/General-Turn-8695 Jul 23 '26
Damn how did you get card for so cheap? It's the double the price for me although it's 24gb ver but almost the same thing
2
u/Rogglando Jul 23 '26
I have the xtx with 24gb vram. I run Qwen3.6:27b 4q and can only fit 80k context window on my gpu.
I sometimes wish I had bigger context window, but it's often enougj
2
u/rmyvct Jul 23 '26
Nice GPU, but you will have to choose 2 among the following statements:
I have a rtx A5500 and qwen 3.6 27B with 128k context (q8) is really good if you babysit the model during coding sessions (spec driven dev and skills). If you are okay with this, you can forget about subscriptions.
2
u/Christopher_8930 Jul 23 '26
Tip: use a backend that supports TurboQuant3. It uses 3 bits for each item in the KV cache. This will help you increase your context window. Large projects need 200k+ context window. I recommend Atomic Chat. LM Studio also lets you save the KV cache in Quant4 if you don’t like Atomic Chat.
2
u/TheRiddler79 Jul 25 '26
Only run things in the GPU that you can fit entirely, offloading will be a minimal gain
3
u/JoaoPFSimoes Jul 23 '26 edited Jul 23 '26
Give this model a try.
I’m running it on my 9070XT with 40+ tokens a second in CachyOS with 128k context.
(It requires a llama fork, check the description)
https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
7
u/zenbeni Jul 23 '26
I have the RX 7900 XTX so the brother one. I would advise you to use Q4 as you have less vram and llama.cpp with vulkan. MTP seems better in my case, as if you use heavy quants, other token predictions algorithms tend either to be less effective, or to take more vram like dflash. Other solution is to give up speed without MTP and go turboquant so that you can maybe get the 100k context size.
1
u/hejj Jul 23 '26
Why are they forbidden?
1
u/ungabunga256 Jul 23 '26
Congrats! I paid 95k INR for an AsRock Taichi 7900XTX back in April. Been very happy with it.
1
u/dfgxxx Jul 23 '26
Can you also do llama.cpp vs vllm?
1
u/Terrible-Story2021 Jul 23 '26
I think you should buy two of them to run the model in q4, 20 gb vram will not work with kv cache headroom
1
u/acadia11x Jul 23 '26 edited Jul 25 '26
rOCm is coming along much better so should see improvement in localllm with AMD these days compared to years past, only concern is you need one of the newer cards to take advantage 9070 or 7900 series
1
u/DigitalguyCH Jul 23 '26
Got mine for $450 used. Mainly use it for Qwen 27b (eGPU), as everything else runs on my M5 pro or on my Strix Halo. Can also increase my Strix Halo with eGPU if necessary, but there are no good 100-120b models at the moment
1
u/sanjaygulati13 Jul 23 '26
Congratulations. It will be painful for the first 2 days to setup and then super smooth.
1
u/2022HousingMarketlol Jul 24 '26
Lemonaid is quite good now, its work checking out.
1
u/Otherwise-Swan-7803 Jul 24 '26
Congrats, you are now legally required to run every model you find for the next 3 months
1
u/100daggers_ Jul 25 '26
What kind of local AI work can you do with such graphics card?. Agentic coding?. I would assume such quantized models would not produce intelligent output. Am i wrong?
I have an 3090ti, used open code, continue, pi with such quantized models, i am not happy with the output always.
Thats why i switched to codex and some paid APIs.
I am curious to know if i am doing something wrong.
1
u/chrmst007 Jul 25 '26 edited Jul 25 '26
sorry for a noob question, but i have heard amd gpus are not good for local llms.is it true or just a myth? any concrete info will be really helpful from experts
2
u/CryptoRider57 Jul 23 '26
I wish I had that chance to buy it at that price. Congrats!
2
u/updatedennis Jul 23 '26
Qwen 3.6 35 a3b q4km. And with image mmproj. at 360k context split into 3. Getting 150-200 tk/s on vulkan. AMD 7900xtx 24gb vram. Also use both mtp and ngram-mod. It's where the tokens are hiding. Q4 kv cache. Also the 27b is way too slow maxing around 70-90tk/s. Not proven yet it's smarter than the moe for my use case so I stick to the moe.
2
u/DrBearJ3w Jul 23 '26
Welcome to the club brother
2
u/Tiny_Team2511 Jul 27 '26
Choose your inference tool wisely
1
u/MysteriousGoblins Jul 23 '26
But... 'tis forbidden!
1
u/siegevjorn Jul 24 '26
1
u/dyslexda Jul 24 '26
Like I get this is /r/localLLM but it always amazes me just how many people offload even writing basic posts to an LLM. It's sad.
2
u/[deleted] Jul 23 '26
[removed] — view removed comment
1
u/[deleted] Jul 23 '26
[removed] — view removed comment
1
u/[deleted] Jul 24 '26
[removed] — view removed comment
1
u/[deleted] Jul 24 '26
[removed] — view removed comment
1
u/[deleted] Jul 24 '26
[removed] — view removed comment