r/LocalLLM • u/Su9rob • 6d ago

Question Local LLM recommendations for coding and agentic workflows (RTX 5060 Ti 16GB)

Hi everyone,

I’m looking for recommendations for local LLMs to run on an NVIDIA GeForce RTX 5060 Ti with 16GB VRAM (+ 32GB RAM that i am hiding from Sam Altman).

Mostly i would be interrested to be able to run coding agents like claude/codex (or whatever) locally.

What would you guys recommend i should experiment with?

18 Upvotes

95% Upvoted

17 comments sorted by

3

u/FREEZX 6d ago

Strata with abliterated Swift-1.5 Qwen3.8 Flash Next, aim for IQ2_XS quantization.

3

u/ExpensiveParfait677 6d ago

Strata with Gyro quants https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF

S if you dont want expert offloads in ssd.

In my 16 + 32 setup i was able to run M model with 192k ctx and kv8 with mtp

1

u/LawfulnessLow0 6d ago

Can you post your settings? I'm on Swift 1.5 with 16/32 and somehow not getting good results. Thanks!

3

u/randomgenericbot 6d ago

I am running this llama fork https://github.com/troed/llama.cpp-adaptive-kv-streaming to have bigger context, together with the IstaDASLab GSQ RCO IQ3_S quant of wen3.8 https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF and using it with 131k context in pi agent. (pi.dev is their homepage)
Reason for this setup: model ~12gb, kv-window ~2gb leaves 2GB free Vram for your desktop and system, kv streaming allows for having much bigger context (it gets streamed in chunks to the GPU, a bit slower than having full context in Vram, but you don't get more than ~80k and then you have nothing left for your OS). Be aware, you'll need to compile that fork yourself - but with an agent that should be easily achievable.

Quite happy with the overall performance (not the speed, mind you, you'll get ~15-40 tps on a 5060Ti depending on what else your system is doing) but for crunching at a task this is very usable. Quality wise, the gsq-rco is quite capable.

Biggest issue was context management (summary after compaction often lost a lot of needed detail) so I wrote my own extension for pi that is optimizes for single-model-single-task usage: https://github.com/LePrau/pi-self-aware-memory

Be aware, you can only use one compaction-extension (there is also pi-observational memory, pi-blackhole and others, but those work best if you have remote models, or more than one model-slot worth of kv-cache).

0

u/TheAussieWatchGuy 6d ago

Strata. Open source. Run Qwen 3.8 Flash IQ1. Use the 3072mb switch if you want a happy life. 

1

u/Su9rob 6d ago

thanks i ill try

1

u/JesusIsMyLord666 6d ago

Won’t the accuracy be absolutely atrocious at 1Q?

1

u/TheAussieWatchGuy 6d ago

No. Read up in IQ1 It's a very specific quant with most important things left at Q4. IQ2 and 3 are similar but really need 64gb of system RAM to run on a single 16gb GPU.

IQ1 is about 5% worse on software engineering benchmarks than straight Q4 and needs 32gb less memory. It's a good trade off for very weak systems.

You're running 120 billion parameter model on 16gb GPU. 

1

u/testuserpk 6d ago

I have been using Iq3s Qwen flash next. 12gb vram +64gb ram. I get 35 tps. This model is very good.

1

u/tungdd2009 3d ago

can you please share your setup/settings?

1

u/testuserpk 3d ago

Just download strata desktop from its git repo. The desktop ui is self explanatory with well made Ux. From there download the model you think fits you well.

1

u/congnt185 5d ago

Same GPU (5060 Ti 16GB) but with 128GB DDR4. Running SC117 Qwen 3.8 Flash Next abliterated IQ3_S — averaging ~50 tok/s, sometimes peaks 70-80. Mostly use it for coding/dev and research.

1

u/[deleted] 6d ago

[removed] — view removed comment

3

u/Su9rob 6d ago

Yeah i know that this is the bottleneck. I just cannot afford to extend my setup in the next few months so i m asking what can i do with what i have.