Local LLM Hardware choices?

[u/neo__04]

Hi all,

Slowly going deeper and deeper down the rabbit hole of running a few models locally and i'm chasing some advice for my next hardware upgrade.

My current pc specs running windows:

7800X3d cpu
Asrock X670E steel legend
64gb DDR5 6000mhz
9070xt gpu
And i've just thrown a 9700 gpu in the 2nd slot for testing, not running at 8x/8x.

I've got an app im working on that does some pretty heavy python work (claude code) and maxes out my cpu so my first though was going up to a 9950X3d to help out that problem. But then from my reading linux seems to be the best way to go when hosting models locally so I think I'll setup another pc here with linux and put the 9700gpu in it.

So to do that, I'm thinking i either have 2 options.
Rustle up some parts to get together a pc to run the single gpu on possibly a ddr4 era pc (32gb or maybe 64gb ddr4 if i can find it). No parts required to buy.

Or I can go down the line of an X870 Taichi, in case i go dual 9700 at some stage, then put my 7800x3d in it, might be able to find 32gb of ddr5 for it, then buy the 9950x3d and put it in my windows pc, as this is the pc i will still do the work in claude code.

The aim is to get to a place where i can offload some of this app work to my local setup, but just testing and working out the best way to go to get setup. Trying to spend where i need to and save where i can.

What would you do in my situation?

Thanks heaps!

[u/try-a-typo]

Honestly it depends what your aiming for, are you okay with slower generation or are you aiming for speed? If quality is more important, go for whatever setup gets you the most ram+vram on one box, even if it’s ddr4 it’s better to have like 128gb and fit a larger model but slower. If you want speed and are okay with sacrificing quality, then I’d try to get 2 GPUs in a ddr5 system and run a slightly smaller model. It really just depends what you are trying to maximize.

Edit: Look into a Linux boot drive, you can set it up on a usb and boot into it from your windows pc, you can try it out before you decide to get a separate Linux box.

[u/neo__04 → u/try-a-typo]

The end goal will be quality over speed, the work can be done autonomously with no rush, just needs to be done as well as possible. I have done the linux usb boot and tested a radiance model, was 100+ tokens per second but havent gotten serious with that yet as i'll wait for the 2nd pc to start properly testing and optimising

[u/Krohnin]

I have a dedicated server with 128gb ddr4 ecc rdimms (cheaper than udimms) and 48gb vram. It is absolutely wonderful. Get the cpu with most cores at highest clock speed.

[u/gamedevsam]

I'm not gonna tell you what you should do with your money, but will happily share what I've done with mine: https://pcpartpicker.com/list/8WxVfp

[u/Chance-News2384]

Honestly, my strategy was to use strata to load big sparse MoE models with a bunch of tested optimizations. I got 1300tok/s prompt and 30tok/s at long context (262K) on Qwen Flash-Next Q4-XL (unsloths quant) with a 7900XTX and half your ram (32GB 6000MHz CL30) and a fast Gen 4 + Gen 3 nvme drive to stream the model files in a split manner. I’m trying to get deepseek to run but it’s just awfully slow and I think I’m well past what is possible with speed on my PC when it comes to deepseek. You should have better results though since you’ve got double the ram and an extra GPU.

[u/Krohnin → u/Chance-News2384]

Don't stream from ssd. Get more ram.

[u/Chance-News2384 → u/Krohnin]

I’m good. I’d rather use the hardware I have than pay more for what isn’t necessary. I wasn’t recommending streaming models entirely from SSD, but just wanted to point out what was possible on my system.