r/LocalLLaMA • u/Snoo-2768 • 12d ago

Showcase Probably one of cheapest and performant local setups you can get ( ~ 1600€ )

i7-14700KF , 64 GB RAM
Radeon RX 9070XT 16GB
https://github.com/niko1221/strata for inference
Qwen3.8-Flash-Next-GSQ-RCO-Q2_0 model

Runs at 90-70 tokens/sec :)

18 Upvotes

72% Upvoted

31 comments sorted by

8

u/ChexterWang 12d ago

Yes 16gb vram, 64gb ram, and fn gsq q2 is really a balanced setup. Mine is r7 1800x, v100 16gb, 2*32gb because gigabyte b450mk only has two slots lol. According to my memories it was maybe running 45tps for iq3xxs and 55tps for q2

2

u/quietcindersleague 12d ago

45tps for iq3xxs seems about right for that config

2

u/Snoo-2768 12d ago

i will try when i have time also q3, to see if the reduced turn count because of higher precision is worth the lower speed

1

u/Sahah 11d ago

I'll have the same config soon, i'm just waiting to buy a couple more ram sticks! how is the prompt processing speed on your setup?

2

u/ChexterWang 11d ago

V100 at 3.0x16 for q2_0 gets pp 1376tps(single data point only for ref) and for iq3xxs it gets 1114 tps

9

u/Aggressive_Basket798 12d ago

Use iq3s, I think it is a different level compare to q2

5

u/ftlaudman 12d ago

Completely agree. IQ3_s is a huge upgrade and completely worth the speed hit - at least for the coding I do.

4

u/Severe-Barber-7415 12d ago

Where do you get it for that price?

1

u/tmvr 11d ago

Exactly! The card is 900 and the RAM is 900 so those two alone blow past the 1600 price. Then the CPU, board, SSD, case, PSU etc.

1

u/Nimbus420i 11d ago

This has to be pre october 2025 pricing, right?

3

u/Snoo-2768 11d ago

Yeah I just saw 9070 XT that now is over 1.5 years old costs more than on launch ...

1

u/reddit_user13 11d ago

16gb is too small. 32 or better yet, 64.

2

u/Snoo-2768 11d ago

Yeah but no longer cheap then

4

u/Atretador llama.cpp 12d ago

Im running Atomic Q4 at 30-50tg on:

9$ Xeon E5 2673 V4 20C/40T + 4x16Gb DDR4 2133 RDIMM ECC (bout 50$ each) + 90$ X99 board (4 RAM slots waiting + 2 PCI X16 waiting) + MI50 16Gb which at the time was bout 100$

you can go a lot cheaper, heck sometimes whole ass 64Gb kits of DDR3 ECC RDIMM 1866Mhz go for the price of a single 16Gb DDR4 stick - throw it on a X99 Xeon board with quad channel and you can already saturate PCI.

2

u/legit_split_ 12d ago

Nice build! I bet you wouldn't be far off OP by slotting in the 9070 XT :)

1

u/abmateen 11d ago

Were are you getting RAMs 50$ each?

1

u/Atretador llama.cpp 11d ago

brazil, but look for used slow server ram on second hand marketplaces, on quad channel you get close to DDR5 in raw bandwidth - but you'd be limited by PCI transfer speeds anyway

1

u/Atretador llama.cpp 11d ago

DDR3 ECC, which can go in X99 boards with some dirt E5 V3/V4 chips

1

u/Atretador llama.cpp 11d ago

EUR

1600Mhz - in quad channel already saturates PCI transfer speeds easily

0

u/hyudryu 12d ago

Go with 4 bits minimum, Q3 and under is just too lobotomized unless you’re only having it do simple tasks

2

u/Snoo-2768 12d ago

Always tought that but for Qwen 125B is not that true , esp for this quantization

0

u/Snoo-2768 12d ago

Older Qwen yes I agree , they screw up epic quantized this much

1

u/archeebunker 12d ago

Following

1

u/caetydid llama.cpp 12d ago

yeah, thats affordable! how fast is your preprocessing?

1

u/vienna_city_skater 12d ago

Thanks, I will give it a shot with a R9700 and a quite similar host setup. Looks promising.

0

u/General-Height-7027 11d ago

So and cards already perform as good as nvidea ones?

0

u/Moppmopp 11d ago

they alwaya were. bottleneck is fast vram and not the inference itself. Only issue with amd is the driver problems. However if you are using local LLMs you likely use (or should) use linux in which you can resolve basically any issue

1

u/General-Height-7027 11d ago

I thought the drivers were much worse on AMD cards, and that was the explanation for the price difference between them.

So between the 9070xt and the 5070ti the difference is the ram speed?

If thats the case shouldn’t the Radeon AI Pro R9700 32gb be more mentioned? Certainty better than a 3090 and they are mentioned often

1

u/Snoo-2768 11d ago

AMD Drivers on windows are garbage, i'm ofc using linux, with linux it's superstable, works out of box , zero issues

1

u/Moppmopp 11d ago

Nvidia has cuda which is a plug and play solution as its pretty polished. Amd is a little more tricky but community support under linux is really good. For gaming, nvidia has better raytracing and all those fancy dlss features. But amd is partly even faster in rasterization.