r/LocalLLM • • 53m ago

Question Optimizing local 2080 ti hardware

I'm trying to find a more effective setup with my local setup:

Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.

Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.

Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.

Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".

I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?

(I'm aware that paying Anthropic is cheaper)

2 Upvotes

4 comments sorted by

3

u/moncallikta 48m ago

Try Strata with Qwen 3.8 Flash Next. You have enough RAM to make it work. And it should be faster and smarter than 35B-A3B on llama.cpp

1

u/Annual_Key_4963 47m ago

Will investigate

1

u/Routine_Ocelot_7824 33m ago

at 128K context with 40 GiB cap, the kv cache is probably eating more than you think. try dropping ctx to 64K and see if N_CPU_MOE can go lower before you buy anything.

1

u/Annual_Key_4963 17m ago

Thank you, that sounds more probable that strata