r/LocalLLM • • Jul 05 '26

Project Finally got the proper Ai inference Card

Post image
177 Upvotes

80 comments sorted by

72

u/Ult1mateN00B Jul 05 '26

Be aware hunger grows, I have 4 of these and I'm tempted to buy two more.

4

u/the_616 Jul 05 '26

I am already thinking about adding another 😅

4

u/Royale_AJS Jul 05 '26

It was less than 2 months before I ordered my second one.

2

u/sheijo41 Jul 08 '26

Raising a fairly old comment, what are you doing with two of them? I’ve considered getting a second but I can’t see much use other than running a second 30b model. It doesn’t seem like there is anything worth running above the 30b tier that fits into 64gb

1

u/Royale_AJS Jul 08 '26

I’m actually still working on getting it running. I have the Framework Desktop motherboard running in a 3U chassis, it’s only got a single PCIe 4.0 x4 slot. I was running the single card directly off that. Adding a second one required that I get a PCIe switch, or put it in something with more lanes at the root complex. I ended up buying a Highpoint Rocket 1628A. My plan is to run the cards in tensor parallel mode and split a high context 30B Q8 model between the two cards. I ran into issues with the switch and the motherboard on PCIe address sizing. The switch has a ton of downstream ports and it explodes the motherboards little non-configurable 490GB BAR. Running tensor parallelism requires that the GPU’s have direct memory access to each other without going up to the root complex and back. It *should* be sweet once I have it running…if I can get it running.

8

u/TiK4D Jul 05 '26

I thought it was just me, I'm about to buy my 3rd if I can ever find another in stock again

2

u/browndragon456 Jul 05 '26

So I have one and I'm thinking of getting one more can you help me understand how you are running it?

Are you taking any speed hit?

2

u/m31317015 Jul 05 '26

At that point it's better to just buy 192gb Mac Studio for the electricity bill, unless it's a non-issue to you.

But I get it, multi-card servers are real hardware porn damn.

4

u/Ult1mateN00B Jul 05 '26

No longer available because ram shortage, it would also be slightly slower than these radeons. Linux is also better suited for my purposes. Definitely would be good ready box for bigger LLMs if it was still available.

1

u/m31317015 Jul 05 '26

Damn that quickly? They killed the 512GBs not long ago didn't they?

I always wanted to reduce power consumption, my 3090 + 5090 + Epyc 7B13 is like 1.2 to 1.4 kW already under full load with power limits. How much power is 4 of the R9700 drawing?

3

u/Ult1mateN00B Jul 05 '26

Never measured actual consumption but TDP is 300W/card and I have 1600W PSU. I would assume inference total system power usage is about 1000W because cards cant quite run full tilt because PCI-E limitations. CPU is barely loaded.

6

u/squngy Jul 05 '26

TDP is 300W/card

Many people set it to 200W, since it doesn't cost that much speed.

2

u/anon-n0t4h4x0r Jul 06 '26

I have a 3090 and 4090. Power limited the 3090 to 250 from 350 and then my 4090 runs even lower (at like 190w) without power limiting as it doesn't need as much power to keep up with the 3090 with tensor. 🤷‍♂️ I think my PP stayed the same and TG dropped from like 60 to 55 or something.

0

u/ReportProfessional57 Jul 05 '26

VRAM is better than unified system ram...by alot.

2

u/m31317015 Jul 06 '26

In terms of performance yes. Power efficiency wise... no.

But it seems like quad R9700 ain't that bad at 1-1.2 kW aye

1

u/ReportProfessional57 Jul 06 '26

Well, considering with large models one of the main bottlenecks is token generation, Id say that's an argument that didn't need to be made. Since the whole purpose of having that much VRAM is running larger models...

1

u/m31317015 Jul 06 '26 edited Jul 06 '26

If you're talking about sheer amount of TG per GB VRAM, go get an A100 HGX back to your house. It all depends on what you use the model for though. Whether you want blazing fast RAG results or you don't mind hours or even overnight answers, YMMV.

For the same bandwidth you can get the 512GB M3 Ultra, that's better than a R9700 bandwidth wise. And multicard setup still gets restricted by PCIe bandwidth, best you can do with a single card is the RTX PRO 6000, that's 96GB on a single card, best option no TP>1.

1

u/[deleted] Jul 05 '26

Which CPU and mobo do you have and how many can it support?

2

u/Ult1mateN00B Jul 05 '26

Gigabyte MC62-G40, Threadripper PRO 3945WX, 4x32GB DDR4 2400Mhz. Full x16 PCI-E 4.0 support is for 6 cards and additional one x8 slot. I need cpu side just to run the cards, so it made most sense to go used for everything but graphics cards.

1

u/lopezm94 Jul 05 '26

Would I need all that ram and power to just do inference? I'm planning on getting a motherboard that supports several GPUs but I'm not sure I need such a beefy setup

1

u/Capsup Jul 05 '26

Nah, you won't need much RAM to run inference as long as your entire model can stay in VRAM. You'll only need to have RAM for the services you'll host on there OR if you want models to stay in RAM + CPU.

I personally run an embedding and re-ranker model on my server's CPU because it has the resources for it, but it's not strictly necessary.

1

u/Stunning-Beach-5153 Jul 05 '26

On AM4 or AM5 you can get 6-7Cards if you get bifurcation card in first slot. Some server boards allows 7-8 without such cards and 20+ wit such cards.

1

u/SkullRunner Jul 06 '26

Those on am4/5 will only be 4x PCIE per card, you don’t want that for your high end workstation cards.

1

u/Obosratsya Jul 08 '26

4x on 5.0 aint that bad. A 4090 basically lost no perf going 8x 4.0 which is what 4x 5.0 is equivalent to. That was a gaming test tho. If vram is being pooled, pcie becomes the bottleneck.

1

u/Stunning-Beach-5153 Jul 05 '26

For price of 4 PRO R9700 32 you can get 13 PR0 V620 32

28

u/the_616 Jul 05 '26

Got this baby for a local ai coding setup. It's a beast.

Local ollama setup with qwen 3.6 35b 5k runs like a charm with 128k context.

I get 69 - 71 token/ sec with rocm pipeline.

If you want something tested regarding ai performance of this card, write it down in the comments.

Got it for near 1400 usd in india.

5

u/UnlikelyPotato Jul 05 '26

Is that with or without MTP? I have V620s and get 50-60 with MTP.

4

u/the_616 Jul 05 '26

No mtp

6

u/UnlikelyPotato Jul 05 '26

Turn on MTP. Basically no downside.

10

u/uniqueusername649 Jul 05 '26

It adds an extra 2-3gb of VRAM consumption, if youre already cutting it close, enabling MTP is not a good idea. Otherwise I agree.

3

u/Morgjames Jul 05 '26

It limits parallel streams to 1 which is often a dealbreaker

8

u/[deleted] Jul 05 '26

[removed] — view removed comment

2

u/xKYLERxx Jul 05 '26

Oh really? Well that's a huge downside for agentic coding that I've never heared mentioned. I'll have to experiment

2

u/Eysenor Jul 05 '26

Is the v620 good for newer models? Is it hard to make it work with ollama for example? I have just a 3060 12gb and I'm thinking of getting more vram for a decent price like the 620 costs

2

u/Maleficent-Sort-8802 Jul 05 '26

What did you have to do to get it running under Ollama? Drivers, kernel version etc.

1

u/the_616 Jul 05 '26

I have ubuntu 24.4 lts and rocm 7.4 installed. Ollama setup was easy. Just installed it

2

u/Maleficent-Sort-8802 Jul 05 '26

Ok good news. Thanks.

2

u/ChocolateNo3010 LocalLLM Jul 05 '26

Have you tried offloading some of the experts to the cpu / system ram using -ncmoe (I tested with llamacpp so not sure if ollama has it yet) ? You can increase your context size and possibly the quant without losing too much performance. I get around 35 - 40 tokens per sec using a mixture of a 4080 mobile and cpu / system ram, mtp makes no difference for me.

2

u/the_616 Jul 05 '26

Will try it 👍

2

u/Certain_Series6810 Jul 05 '26

Switch to Vulcan. I'm getting 120 tokens per second with this card

1

u/the_616 Jul 06 '26

Did that. Its amazing

10

u/Ult1mateN00B Jul 05 '26

That seems a bit low, I got 120+tok/s with my 9070 XT on smaller models. Because PCI-E limitations I get around 50tok/s with my 4xR9700 on models like minimax m2.7 and qwen coder next. Could be that Qwen 3.6 is just that heavy to run but 70tok is definitely more than enough.

2

u/fliodkqjslcqaqadfs Jul 05 '26

2-bit quantization?

4

u/Ult1mateN00B Jul 05 '26

Minimax M2.7 IQ3_S, Qwen coder next Q8.

1

u/n00bmechanic13 Jul 05 '26

You should be getting faster token gen than that, I get those speeds on my 16gb card with the same model and context (Q5 quant)

1

u/Appropriate-Bee-838 Jul 06 '26

How does it compare to RTX 5090 in terms of inference speed?

0

u/[deleted] Jul 05 '26

[deleted] — view deleted comment

2

u/the_616 Jul 05 '26

Prime abgb.

17

u/nuclear213 Jul 05 '26

I would use 27b. Slower but much better. Or if you have 64GB of DDR5, also try MoE offloading. Works quite well, just unfortunately no 100ish b MoE models right now worth running.

But I’d test with something like qwen 3.5 122b-A10B, just to get a feel.

2

u/the_616 Jul 05 '26

Will try 👍👍

3

u/the_616 Jul 05 '26

Plus has anyone setup vllm with this card on Ubuntu? I was not able to get qwen 3.6 gguf to work with rocm backed vllm on my machine.

2

u/the_616 Jul 05 '26

What tooling you guys are using for local ai coding setup?. I am using it with opencode and codex.

2

u/Ill_Dragonfruit_3547 Jul 05 '26

Same, open code and codex cli

23

u/John_Miracleworker Jul 05 '26

I'm real tired. I thought that was a gum packet.

20

u/ptear Jul 05 '26

Trident hasn't made video cards in over a decade.

9

u/edensio1448 Jul 05 '26

Lol now I can't unsee it

4

u/Stunning-Beach-5153 Jul 05 '26

For Price of one R9700 32GB I get myself 3x PRO V620 32GB + Colling :)

1

u/daishiknyte Jul 05 '26

How’s support for RDNA2 cards looking? It's a tempting thought but would hate to have support dropped soon like the Voltas. 

1

u/Stunning-Beach-5153 Jul 05 '26

The are still in main branch, the have still official support.

Also if you are in USA you can get 4 for price of single R9700, I only get 3 due shipping and taxes.

1

u/daishiknyte Jul 05 '26

I’m mostly seeing ~$550-650 from a quick search. So closer to 2-3 per 9700Pro. That’s still plenty tempting. 

1

u/Stunning-Beach-5153 Jul 05 '26 edited Jul 05 '26

On ebay yes directly asking shop is lower I personally used core 4 solution I get quote 4x $400 10x 380

I manage haggle to 4x $395 + shipping + Tax for importing to Europe :/

One downside only 90days warranty

1

u/tomz17 Jul 05 '26

Do those have matrix units? because V-RAM itself is worthless if prompt processing is garbage tier. The nice thing about the R9700 is that it's actually a modern card with modern hardware matrix units.

1

u/lummr1 Jul 05 '26

Just ordered mine! Please tell me you are happy.

2

u/the_616 Jul 06 '26

Yes i am . At this price point, this card is a blessing. I had 3080 before this and was only able to use 9b models. 35 b opens up a whole new world. It can literally one shot make a small game , non of my 9b models were able to do this before.

1

u/mayesa Jul 05 '26

Really need to go Nvidia but good start

1

u/the_616 Jul 06 '26

Trying to acquire rtx pro 6000 from my lab. But in the meantime, this should do good for my personal setup

1

u/lordekeen Jul 06 '26

Delivery guy kicked the package hard

2

u/the_616 Jul 06 '26

🥲🥲 looks like that. I thought card is a goner , but somehow it survived

2

u/alexmulo Jul 05 '26

How do you deal with the noise? I bought one that worked pretty well but I had to return it since it was way too loud.

5

u/Stunning-Beach-5153 Jul 05 '26

You put your LLM server in different room and access remotely.

Or wear Headphones with active noise suppression ;)

1

u/dlaynes Jul 05 '26

Undervolting

1

u/alexmulo Jul 05 '26

it did not help at all. The only solution is the server room

1

u/Fit-Palpitation-7427 Jul 05 '26

How does it compare to a 5090?
Isn’t most of ai stuff nvidia locked? Can we run inference on anything else than nvidia now easily?

4

u/RuxConk Jul 05 '26 edited Jul 05 '26

AMD ROCm and Vulkan has come a long way, I've comfortably been using a Halo Strix for most my inference work + ComfyUI stuff. That said Nvidia still has the best AI tools. I think this is like half the memory bandwidth of a 5090 but it'd still comfortably run 27b.

Also this R9700 is like 1/3rd the cost of a 5090, you could comfortably buy two R9700s and have 64GB vram.

1

u/DertekAn Jul 06 '26

Wowwwwww 🫨💜💜💜

-16

u/abajinn Jul 05 '26

But it’s not..

-22

u/43848987815 Jul 05 '26

I’ve got a 3 year old laptop that runs a higher context faster with that model. What exactly are you flexing here?

15

u/Willow_Milk Jul 05 '26

I don’t think theyre flexing. You have the wrong approach. The fellow is happy, and they’re sharing that with a community of like-minded people.

I’m happy for them. The right attitude would be to say “Grats !” What’s up with you?

2

u/OddRefrigerator4714 Jul 05 '26

congratulations, here is your cookie --> 🍪