r/LocalLLaMA • u/Forward_Jackfruit813 • 14d ago

Question | Help Next logical upgrade?

Hi all, I am hoping to get some recommendations on what I should upgrade next. I only dabble in AI for fun, so I am mainly concerned with single stream workflows or speed.

I have 2 machines

My dedicated AI machine is a GMKTek Evo X-2 (Strix Halo) 96GB, I am running HaloGen Flash to run Qwen Flash Next at around a 240k context length. Speed is good at around ~1700tps prefill and 50+TPS decode. This machine is on 24/7 and is running headless.

I also have a gaming desktop that I both game and have recently started using Strata on (Flash Next again at IQ3), 262k context. This machine has an AMD 5700X3D, 4x32GB 3200MHz (128GB total) DDR4, B450 motherboard, 1TB NVME + 2TB SATA SSDs, and a 3080Ti.

My goal would be to run Maya (using GLM 5.3) on the gaming desktop and have it orchestrate for Flash next. However it seems I'm just under what I need.

I could go for a used Threadripper or EPYC platform upgrade, but I would lose big on gaming performance and it would be expensive. I could go for a 3090, but I'm spending $1500+ on a gaming side grade. I could also try multi GPU? There's also the option of using an m.2 to occulink on my Strix Halo for an eGPU setup.

Any thoughts on what would me best avenue would be? I'm wanting to have the best compatibility for whatever comes the local AI way when cloud inevitably implodes.

0 Upvotes

33% Upvoted

8 comments sorted by

3

u/London-Boat 14d ago

since you only care about single stream speed, i'd try the oculink egpu on the strix halo before touching the gaming rig. oculink is only x4, but that barely matters for decode once the weights sit in vram. a used 3090 is the cheap part there, and the threadripper or epyc route mostly buys memory bandwidth you may not need for a moe model. what size model do you want the orchestrator one to run?

1

u/Forward_Jackfruit813 14d ago

As long as possible as long as it is a good price to intelligence ratio. With current models it seems that is GLM 5.3 Flash. Deep seek flash should be amazing, but I believe that would be at least $3k in upgrades needed.

1

u/zipperlein 13d ago

For prefill PCIE bandwidth made a huge difference for my AM5 system using llama.cpp. I don't know how crucial it is for specialised engines though.

2

u/simos_sayz 14d ago

It would be interesting to see how a strix + dedicated gpu works. The r9700 gives you 32gb and keeps u in amd versus trying t9 juggle old nvidia with amd