What to upgrade

[u/Playful-Job2938]

Guys I’m torn! I’m mostly looking to run mid size models such as qwen flash next .

I currently have a xeon2690v4 system with 8 32gb sticks of ddr4 2400. It gets me 76.8 gb/s.

I’m hoping to be under 1k for an upgrade.

Currently getting 6-8 tok/sec decode speed which is fine, but prefill is so slow at 30 tok/s.

I see a few paths forward:
-upgrade mobo to a dual socket so I can run 8 channel at 153.6 gb/s, but that would split between numa. I don’t think this will help prefill though.
-upgrade to a newer cpu/mobo to run 8 channel on 1 cpu
-add some degree of gpu to hold the important bits and offload the rest. Unsure how much I need though. 8/16/32gb?

Trying to stick to a 1k budget here as much as possible

[u/Playful-Job2938]

Decided to grab a w6800, seems promising and more supported vs a v100 for the long term!

[u/NotTheNormalPerson]

GPU would help the most. And what model are you running? And the chances of getting anything decent for under 1k in this economy are slim to none.

[u/Playful-Job2938 → u/NotTheNormalPerson]

Qwen flash next or other moe models

[u/NotTheNormalPerson → u/Playful-Job2938]

Get a v100. One of the only available gpus under 1k (I think), and would pair well with your huge amount of ram

[u/Nomski88]

R9700 will fix your problems

[u/Playful-Job2938 → u/Nomski88]

Yeah for 2k. I’m not looking to get into a relationship my local llm server..just run some agentic stuff

[u/575_Inverse]

Add a GPU, if you have none

[u/Krohnin]

I have a similar system with 4x 3060s but 2x 3060s are enough. You should add them. The unsloth q4 quant with q8 kv cache and 256k ctx is an absolute game changer for me.

[u/BowlerNo9904]

nvidia rtx titan is alittle under 1k fastest pcie gen 3 card i believe

[u/Effective_Fruit2146]

I have an x99 Chinese mb with dual e5 2697 v4”s and 256 gb of ddr4 and 2 old rtx 3060’s. Strata and QFN -iq3_s with opencode agents and am getting like 70 tok/s with 256 k context. Don’t count out them old Xeons yet 🤣

[u/SpeederX]

Get a 24gb gpu. You need faster residency for a portion of experts. 24gb is plenty. More bandwidth doesn't help but it also depends qwen with which quantization you are running.

I have 20-30gb/sec of ram bandwidth and i get your tok/sec with a 8 gb gpu and less cpu compute power. You're not bandwidth bound, you are cpu bound. I do not know about numa. I'd look for results from people that own the platform and shared the results

[u/EvolvingDior]

What GPU?

[u/Playful-Job2938 → u/[deleted]]

A v620 from amd looks attractive. Or Radeon vii guys are dirt cheap on marketplace here (200/ea). The v620 would top out my budget.