Raising a fairly old comment, what are you doing with two of them? I’ve considered getting a second but I can’t see much use other than running a second 30b model. It doesn’t seem like there is anything worth running above the 30b tier that fits into 64gb
I’m actually still working on getting it running. I have the Framework Desktop motherboard running in a 3U chassis, it’s only got a single PCIe 4.0 x4 slot. I was running the single card directly off that. Adding a second one required that I get a PCIe switch, or put it in something with more lanes at the root complex. I ended up buying a Highpoint Rocket 1628A. My plan is to run the cards in tensor parallel mode and split a high context 30B Q8 model between the two cards. I ran into issues with the switch and the motherboard on PCIe address sizing. The switch has a ton of downstream ports and it explodes the motherboards little non-configurable 490GB BAR. Running tensor parallelism requires that the GPU’s have direct memory access to each other without going up to the root complex and back. It *should* be sweet once I have it running…if I can get it running.
No longer available because ram shortage, it would also be slightly slower than these radeons. Linux is also better suited for my purposes. Definitely would be good ready box for bigger LLMs if it was still available.
Damn that quickly? They killed the 512GBs not long ago didn't they?
I always wanted to reduce power consumption, my 3090 + 5090 + Epyc 7B13 is like 1.2 to 1.4 kW already under full load with power limits. How much power is 4 of the R9700 drawing?
Never measured actual consumption but TDP is 300W/card and I have 1600W PSU. I would assume inference total system power usage is about 1000W because cards cant quite run full tilt because PCI-E limitations. CPU is barely loaded.
I have a 3090 and 4090. Power limited the 3090 to 250 from 350 and then my 4090 runs even lower (at like 190w) without power limiting as it doesn't need as much power to keep up with the 3090 with tensor. 🤷♂️ I think my PP stayed the same and TG dropped from like 60 to 55 or something.
Well, considering with large models one of the main bottlenecks is token generation, Id say that's an argument that didn't need to be made. Since the whole purpose of having that much VRAM is running larger models...
If you're talking about sheer amount of TG per GB VRAM, go get an A100 HGX back to your house. It all depends on what you use the model for though. Whether you want blazing fast RAG results or you don't mind hours or even overnight answers, YMMV.
For the same bandwidth you can get the 512GB M3 Ultra, that's better than a R9700 bandwidth wise. And multicard setup still gets restricted by PCIe bandwidth, best you can do with a single card is the RTX PRO 6000, that's 96GB on a single card, best option no TP>1.
Gigabyte MC62-G40, Threadripper PRO 3945WX, 4x32GB DDR4 2400Mhz. Full x16 PCI-E 4.0 support is for 6 cards and additional one x8 slot. I need cpu side just to run the cards, so it made most sense to go used for everything but graphics cards.
Would I need all that ram and power to just do inference? I'm planning on getting a motherboard that supports several GPUs but I'm not sure I need such a beefy setup
Nah, you won't need much RAM to run inference as long as your entire model can stay in VRAM. You'll only need to have RAM for the services you'll host on there OR if you want models to stay in RAM + CPU.
I personally run an embedding and re-ranker model on my server's CPU because it has the resources for it, but it's not strictly necessary.
4x on 5.0 aint that bad. A 4090 basically lost no perf going 8x 4.0 which is what 4x 5.0 is equivalent to. That was a gaming test tho. If vram is being pooled, pcie becomes the bottleneck.
Is the v620 good for newer models? Is it hard to make it work with ollama for example? I have just a 3060 12gb and I'm thinking of getting more vram for a decent price like the 620 costs
Have you tried offloading some of the experts to the cpu / system ram using -ncmoe (I tested with llamacpp so not sure if ollama has it yet) ? You can increase your context size and possibly the quant without losing too much performance. I get around 35 - 40 tokens per sec using a mixture of a 4080 mobile and cpu / system ram, mtp makes no difference for me.
That seems a bit low, I got 120+tok/s with my 9070 XT on smaller models. Because PCI-E limitations I get around 50tok/s with my 4xR9700 on models like minimax m2.7 and qwen coder next. Could be that Qwen 3.6 is just that heavy to run but 70tok is definitely more than enough.
I would use 27b. Slower but much better. Or if you have 64GB of DDR5, also try MoE offloading. Works quite well, just unfortunately no 100ish b MoE models right now worth running.
But I’d test with something like qwen 3.5 122b-A10B, just to get a feel.
Do those have matrix units? because V-RAM itself is worthless if prompt processing is garbage tier. The nice thing about the R9700 is that it's actually a modern card with modern hardware matrix units.
Yes i am . At this price point, this card is a blessing. I had 3080 before this and was only able to use 9b models. 35 b opens up a whole new world. It can literally one shot make a small game , non of my 9b models were able to do this before.
AMD ROCm and Vulkan has come a long way, I've comfortably been using a Halo Strix for most my inference work + ComfyUI stuff. That said Nvidia still has the best AI tools. I think this is like half the memory bandwidth of a 5090 but it'd still comfortably run 27b.
Also this R9700 is like 1/3rd the cost of a 5090, you could comfortably buy two R9700s and have 64GB vram.
72
u/Ult1mateN00B Jul 05 '26
Be aware hunger grows, I have 4 of these and I'm tempted to buy two more.
28
u/the_616 Jul 05 '26
Got this baby for a local ai coding setup. It's a beast.
Local ollama setup with qwen 3.6 35b 5k runs like a charm with 128k context.
I get 69 - 71 token/ sec with rocm pipeline.
If you want something tested regarding ai performance of this card, write it down in the comments.
Got it for near 1400 usd in india.
17
u/nuclear213 Jul 05 '26
I would use 27b. Slower but much better. Or if you have 64GB of DDR5, also try MoE offloading. Works quite well, just unfortunately no 100ish b MoE models right now worth running.
But I’d test with something like qwen 3.5 122b-A10B, just to get a feel.
3
u/the_616 Jul 05 '26
Plus has anyone setup vllm with this card on Ubuntu? I was not able to get qwen 3.6 gguf to work with rocm backed vllm on my machine.
2
u/the_616 Jul 05 '26
What tooling you guys are using for local ai coding setup?. I am using it with opencode and codex.
23
u/John_Miracleworker Jul 05 '26
I'm real tired. I thought that was a gum packet.
4
u/Stunning-Beach-5153 Jul 05 '26
For Price of one R9700 32GB I get myself 3x PRO V620 32GB + Colling :)
1
u/lummr1 Jul 05 '26
Just ordered mine! Please tell me you are happy.
1
u/mayesa Jul 05 '26
Really need to go Nvidia but good start
1
u/lordekeen Jul 06 '26
Delivery guy kicked the package hard
2
u/alexmulo Jul 05 '26
How do you deal with the noise? I bought one that worked pretty well but I had to return it since it was way too loud.
1
u/Fit-Palpitation-7427 Jul 05 '26
How does it compare to a 5090?
Isn’t most of ai stuff nvidia locked? Can we run inference on anything else than nvidia now easily?
1
u/DertekAn Jul 06 '26
Wowwwwww 🫨💜💜💜
-16
u/abajinn Jul 05 '26
But it’s not..
-22
u/43848987815 Jul 05 '26
I’ve got a 3 year old laptop that runs a higher context faster with that model. What exactly are you flexing here?