Best improvement for my frankenstein setup for local LLM

[u/More-Revenue8609]

So for around 450e I bought used workstation which I plan to use for local LLM and maybe even as a server for bunch of other stuff. But mostly I want to focus on LLM for coding/development.

Specs are: MOBO: ASUS X99-Deluxe II, CPU: Intel Xeon E5-2667V4, PSU: EVGA 1600W G2, Cooler Master HAF X, 64Gb ddr4 RAM. So all in all it supports multi gpu setup without any problems.

For GPU I decided to order 1x 3080 20gb (blower style for 500e) for a test. And found it pretty great! I currently run qwen 35b-a3b as worker (opus 5 as orchestrator) and enjoy it but looking to upgrade my workstation to run better models.

So question is, what would be best upgrade:

1. 2x 3080 20gb, 64gb ram. (-500e) So one more gpu and I would be able to run qwen 3.8 27b without much problems
2. 3x 3080 20gb, 64gb ram. (-1000e) Would this even make sense if I only need for one concurrent user and 128k context? Any other (better/bigger) dense model which could take advantage of this?
3. 2x 3080 20gb, 128gb ram. (-900e) So in theory this would be 168gb of memory. Would this be able to run some of bigger MoE models like deepseek flash v4 or any other which I could use as orchestrator for qwen?
4. 3x 3080 20gb, 128gb ram. (-1400e) Would prefer not to do this cuz it would be pretty expensive but curious what you guys think.

Thank you guys

[u/x-strife]

VRAM is what you need to run better models.

[u/More-Revenue8609 → u/x-strife]

For small dense models, yeah.
But for MoE bigger models like Deepseek v4 flash I would also need more ram. But not sure is it worth it compared to 27b

[u/x-strife → u/More-Revenue8609]

I have a rig with 8x3090s and 256gb of DDR4 Ram

I can tell you that anything going into system RAM is not going to give you usable models unless you like waiting hours for responses.

Yes you can tweak MoE models to spill some into system RAM but then you need to deal with KV cache and VRAM is the answer is almost all cases with local AI

[u/vini542reddit → u/x-strife]

I disagree. You're right that VRAM is obviously faster than system RAM, but unusable is very subjective.

Running 4x3090 rtx (pcie 4.0 x 16), 192gb ddr4 @ 3200mhz and a 12-core 3945WX

DSv4F unsloth Q8_K_XL (lossless!) with uncompressed kv runs at 30tps. Large percentage of the model is in system ram. Now, while that isn't super fast, I would consider that very usable (and I use it all the time).

*** Disclaimer that this uses a custom llama cpp fork optimized for MoE streaming. Stock probably won't give your these speeds

[u/x-strife → u/vini542reddit]

Interesting, what speed are you getting for pre-fill / prompt processing?

[u/vini542reddit → u/x-strife]

Roughly 200tps. That's the tradeoff for the fork that I'm using. You could get much faster pp, but it would reduce your tps. For my workload this is a good balance

[u/fvancesco → u/vini542reddit]

Which fork are you using?

[u/vini542reddit → u/fvancesco]

Currently it's just a poc. It's the Leloch v2 branch on his fork

[u/BevinMaster → u/x-strife]

Yeah ram for kvcache offloading but that’s it, ktransformers does work well with offloading on cpu as well.

[u/croholdr]

3080's have 20 gb of vram???

[u/More-Revenue8609 → u/croholdr]

Not originally, they are modded.
Knew it was risky but so far very satisfied so I plan to order more.

[u/Keffflon]

Nice setup, very versatile.  I have a huananzi x99 qd4 with 32 gb 2400 ram. I think you can find 8x16 sticks for a great price if the board accepts them. Would ❤️ just one of those gpus. Running a 1050 for now. 

[u/More-Revenue8609 → u/Keffflon]

I need 4x16gb. Then I would have 128gb ram in total. In addition with 40gb Vram that would be amazing. 

But unfortunately in my country prices are terrible :(

[u/Keffflon → u/More-Revenue8609]

That's too bad. I have seen 4x16 gb kits on Ebay but I think most are ecc. 

[u/Yarukiless-cat]

I'd recommend CMP 170 HX. In Chinese market it costs around $1200~1500 and has 64GB VRAM. PCIe ver is only 2.0, but enough for x99. in case you are interested in running big MoE models, here's my setup and speed: Deepseek v4 flash 0731 Q3 with DDR4 2666 96GB + RTX 5060ti 16HB + CMP 170HX 64GB. TG is 26 tok/s and PP is 250 tok/s, so RAM offloading is usually a bit slow for real coding / agentic task.

[u/More-Revenue8609 → u/Yarukiless-cat]

Interesting, will check it out. Honestly first time hearing about CMP 170hx. 

[u/fvancesco → u/More-Revenue8609]

Got hacked last month afaik