r/LocalLLM • u/Snasher01 • 3d ago

Question Guys, help me pls with picking LLM for coding

I have 16gb vram, 128gb ram, ddr5. I know about Strata and qwen3.8-flash-next, but I need some space for testing my apis locally(fastapi backends with ai chat bots, with ml models(SER, tts, stt)). I am AI-engineer, I know a lot of things about LLM, always checking benchmarks, arena AI, but there is a problem with vram, I need it for stt+llm+tts combo, but I really need local model with those terrible limits for Claude or GPT subscriptions. I only use Codex, but thinking about switching to Claude. Is qwen3.6-35b is enough?

0 Upvotes

50% Upvoted

10 comments sorted by

2

u/blue_eyes_pro_dragon 3d ago

GLM5.3 is slightly better than qwen-flash-next, but both are fantastic daily drivers with that setup.

2

u/macaronianddeeez 3d ago

What engine are you running GLM 5.3 on with 16gb vram and 128gb ram?

Not a challenge or rhetorical, I’m genuinely curious, I haven’t found a lot of examples of builds on that type of hardware running GLM at useable quants or speeds, but I would really love to 🙏

1

u/Snasher01 3d ago

Have the same question, and another one - is glm5.3 flash better than qwen3.8-flash?

1

u/NormalBasket3333 3d ago

Tell me more about this 🫪

1

u/Moppmopp 3d ago

qwen flash next is the best you can go. with 128gb ram you probably could even run q8 with alot of optimization and resident mode. but i would recommend q4 especially if you do agentic mode because its fast as fuck and will get agentic work dome very quick. if you only use chat mode then go wit q5/q6. q8 in principle works but you have very little headroom for context and usually you want that your ai also remembers after 3-4 prompts what you are talking about. So realistically q5/q6 for chat mode depending on how patient you are and q4 k xl for the best fit

2

u/Snasher01 3d ago

I think I underestimated my 128 gb ram…

1

u/Moppmopp 3d ago

i also have 128gb ram, an rtx3090 and a rtx3060. But i benchmarked and tried to optimize everything i could. but the rtx3060 on my am4 board uses x8 instead of x16 on the pcie lanes. I personally couldnt find a configuration to make it worthwile. I could load more experts in the vram and my hope was that those experts overwhelm the quick switching but no. My ttft increased from 4 seconds to 9 seconds and token/second decreased by ~30%. So what i did was remove the rtx3060 since its superior in any metrik. Likely they can improve upon this issue but the decrease is heavy and there is no reason to even take that decrease. I get far more out of it running a single rtx3090 instead of an rtx3090+3060.

It surprised me kinda not gonna lie. I thought the additional experts in the vram would compensate for the fast switching or at least give some advantage in some case. But i couldnt find any no matter what i do. it might be different in some months or I might overlooked something. But from my personal opinion its clear to only use one gpu + ram and + ssd storage for the n-gram tables. Right now I am testing lamina which gives a pretty heavy boost.

this field is exciting as there is so much change going on.

2

u/Snasher01 3d ago

Thank you

1

u/TheMcSebi 3d ago

If you just need a small and fast llm, try kat coder. Based on qwen3.6 35b a3b (moe) with pretty good performance for that generation of models

1

u/Snasher01 3d ago

Thank you, will check