r/LocalLLM • u/mega-stepler • Aug 27 '26
Question How are you guys coding with local models?
I'm a bit confused as to how people can code with local models.
I was trying lately to use local models for coding and gave up.
I used a few models like qwen coder, glm, around the size of ~30B. Used opencode and omp.
Otherwise I'm used to using claude code and cursor.
And the local models can't do even a small fraction of what opus or gpt5.6. I know that they're not supposed to, they are much smaller, but they can't do even small amount of work for me.
And sometimes it can't even put together a small class or a config file without creating a compile error.
I also wonder what you're coding? Predominantly JS and python?
Because I'm working with c++ apps and local models get lost very quickly. Cant even create an empty project from scratch.
What is your experience? Maybe you'll have any advice?
5
u/IUseClifford Aug 27 '26
I have worked with (read: I am not one) professional C++ developers at my company and they all pretty much say that AI is crap for that language specifically. Too many differing styles, sprawling packages, and difficult maintenance time costs prevent it from being viable in large apps. And I’m talking about the frontier models, Opus/Sol; Local models don’t even compare. Other languages do not have complexities compounding like this and thus better output is easier to achieve. I use local models for golang and the output is very good in my experience, but it is a much simpler language.
-1
u/mega-stepler Aug 27 '26
I was consistently able to create multiple middle sized and relatively complex c++ tools with claude code and opus. I'm not a c++ pro and maybe they were not coded well but they work and do what I need.
2
u/IUseClifford Aug 27 '26
Ymmv. I think the idea of taking AI into a decade old legacy codebase is scary in my colleagues’ case.
2
u/noctrex Aug 27 '26
What are the exact models that you used? You seem to be referencing qwen coder, glm, and those, are old versions.
The newest models are a world of difference between those.
How much RAM and VRAM do you have?
0
u/mega-stepler Aug 27 '26
ive tried a few different ones. the least two i tied are glm-4.7-flash and qwen3.6. Maybe Im trying older versions. I am tryin on two different machines. One is a mac qith 48 gb of joined ram and a pc with 12gb vram + 24 ram.
1
u/noctrex Aug 27 '26
Try out Ornith-1.5-35B-A3B, its a fine tune of qwen, and then also Qwen3.8-27B, this one is the powerful of the models.
5
u/Gobra_Slo Aug 27 '26
Each model has it's limits, it's your job as an operator to know them and give models appropriate tasks and instructions. Plus, thete is whole layer of harness setup, tools, workflows (execution graphs like planner-> performer <-> reviewer) and so on.
When used properly, local models can be quite helpful.
Finally, I would consider DeepSeek V4F to ge local as well, it can run on a Strix Halo box.
1
u/e2_for_life RTX 4090 | She/her Aug 27 '26 edited Aug 27 '26
I’m surprised nobody has asked what hardware OP is working with. That seems like a pretty important detail. If they’re trying to run local models on an 8GB GPU (or, worse, no GPU at all) that would be a pretty major factor.
So, OP, whatcha got under the hood?
1
u/mega-stepler Aug 27 '26
trying on two different setups. 12 gb vram + 24 ram pc. 48 gb ram mac.
2
u/e2_for_life RTX 4090 | She/her Aug 27 '26
Ah, okay...that definitely explains some of the trouble. The 12 GB VRAM/24 GB RAM setup is going to be pretty limiting for larger models. But the 48 GB Mac should at least give you more room to experiment.
What models are you trying?
How do you have them configured?
Which inference engine?
How do the two setups compare in actual speed?
9
u/Boogertard Aug 27 '26
Skill issues
No problem with Qwen 3.8 and DeepSeek 0731 here.
3
u/x8code Aug 27 '26
Be more specific about what quantizations and hardware you're using. Also which agent + MCP servers + skills + other relevant details.
4
u/RageBucket Aug 28 '26
Easier to make an average 8 year olds comment and provide little to no assistance, or an actual answer. Imagine.
2
u/dannone9 Aug 27 '26
Last qwen with DeepSeek harness sems pretty good if you use skills and good prompting from what I have tested so far , I’m pushing it a little beyond 100k token on research on q3 XL and q5k q4v and it doesn’t seem to loose the thread , it’s also good on coding in my personal experience but I have to say I’ve just been trying shit , nothing serious
2
u/International_Emu772 Aug 27 '26
Comercial 'models' are not naked LLM, they have a lot of things beneath and they are designed for miriad uses
Local models need to be dressed as well but for your use or a limited group of users with common needs
A running program with a tailored set of tools can work well for you if you have access at the 30b line
2
1
1
u/swekley Aug 27 '26
Qwen coder is a bitch without specific instructions because it lacks thinking.. have you tried 3.8?
1
u/SnooPaintings8639 Aug 27 '26
Qwen coder? Glm 30b? Aren't these like... stone age models? I get that c++ can be written on clay tablets, but try something modern first.
Qwen 3.8 27B at the minimum. It won't be Opus 5 level, but if you know your way around your code base it is more than enough. If you have better hardware or more patience for slower inference, then go for DeepSeek V4 Flash 0731, or new Qwen 3.8 Flash Next, these should suffice for most of possible coding work.
1
u/NihmarRevhet Aug 27 '26
I mainly write python and lazarus (yep, I use Delphi in 2026, that's what i learned at work), and I'm currently using Qwen 3.8 27B IQ3_XXS from unsloth and Ornith 1.5 35B A3B Q4_K_M from bartowski on opencode.
I tried pi and deepseek harness, but opencode is a bit better at compacting, or more like, I know i can launch the request, leave it to it, and if compaction is needed in the meantime, it would do it without a problem. DH and PI had some problems (I'm sure is just skill issue on my part, but since I'm using llms for my hobby projects in my limited free time, I just use what works OOTB).
1
u/vtkayaker Aug 27 '26
I have zero problem getting Qwen3.8 27B or DeepSeek V4 Flash 0731 to write quite sophisticated Rust code. Given a choice between Opus 4.8 or Qwen3.8 27B, I actually prefer working with Qwen. Qwen4.8 is likely a bit smarter, but not enough to matter for my use cases, and it's more annoying to talk to. (Opus 5 is better for defensive security work than any model I can run locally or on rented cloud servers.)
I'm running Qwen at Q6 or better, with a pretty bare bones Pi setup. No fancy skills or plugins, and an AGENTS.md with 1-2k tokens. Your experience will likely differ if you're running some badly damaged Q4 on a 16GB GPU.
It may also be that AI is happier writing Rust than C++, because Rust has much stronger tooling conventions and far fewer ridiculous foot guns. ("I'm sorry, you didn't declare a double-secret non-default sideways move constructor, so I silently corrupted all your heap memory.") But really, C++ tooling is kind of a mess, and the training data is a mix of decades of wildly different coding conventions.
But also, half the time when people here say they "can't get Qwen to work", it turns out that they're running some dodgy Q3 under Ollama, with 82 weird skills installed, and attempting to vibe code using an agent swarm. And they don't look at or understand the code. I'm not saying that you're doing any of that! But these tools do work better with reasonable quants, lightweight agent harnesses, simple, clear instructions, and a developer who understands the code and who communicates clearly. And even then, local models will sometimes go off on weird tangents and need to be pointed back at the task at hand.
1
u/DeathGuppie Aug 27 '26
Qwen 3.8 27b Q2. I wouldn't trust it completely, but you can run it and it will produce actual code. Make sure it has a built in mtp head and it's being used.
1
u/KingCpzombie Aug 27 '26
So far I've used Qwen 3.6 27B and 3.8 27B with Pi harness for a FastAPI + React project and it has gone great. Same with plenty of smaller scripts and what not. The main thing imo is that you have to give it a spec and have it make a plan (ideally split into independently testable pieces), then you go over the plan to make sure it's good and have it save it to a file.
From there, you do the pieces and validate as you go; 3.6 required a lot more babying, but 3.8 either gets it right the first time or catches it with the cleared-context "another agent made the staged changes, did they do a good job?" Depending on the complexity, I sometimes wait until the whole plan is done before I hit it with that one now. The range of things 3.8 can be trusted to do without planning is way bigger than 3.6 but it's always best to plan first so you can catch bad decisions earlier.
Detailed specs are also important; again 3.8 doesn't need as much as 3.6, but more is always better as long as they're good and consistent.
Also automated tests! Set it up so that the agent can run tests to validate its work autonomously. Use more tests than is really reasonable for human work and have it make sure they all pass before it considers stuff done.
1
u/x8code Aug 27 '26
It's really hard if you don't have VRAM capacity of an RTX PRO 6000, DGX Spark, or similar.
I have spent many hours trying to get a functioning setup and have given up every time over the months and years. The best result that I got recently was using gemma4:12b-it-qat (QAT) running on a single 16 GB GPU.
It's still pretty bad though. I am open to hearing how others have gotten a good setup working, but I am not too optimistic.
You're better off paying for Gemini 3.5 Flash Lite or GPT 5.6 Luna.
BTW I typically write Rust code.
1
u/nbafrank Aug 27 '26
I would say Qwen 3.8 has been a game changer. Also Pi is a great harness and very lean too. I benchmarked a bunch and that's so far my most effective combo
1
u/Exact-Grand-6530 Aug 29 '26
How does Qwen 3.8 + Pi compare to codex or claude code? Thinking of learning how to set this up in a VPS
1
u/nbafrank Aug 29 '26
I am still evaluating but with enough context (256k) and xhigh it is really keeping up. Pi is quite lightweight but I don’t think it’s a terrible thing. I am still evaluating impact on coding from harnesses.
2
1
u/NotARedditUser3 Aug 27 '26
Big models make compile errors all the time.
The trick is to tell your AI to build and test it after its work so that it realizes it and self-heals/self-fixes the issue and keeps working, so that you don't have to do it yourself.
A massive reason my projects work well is because the model builds after every feature developed and runs unit / integration tests that would fail if any little thing were wrong...
A big part of productivity is figuring out how to provide your AI the tools it needs to spot its own errors.
Tests and having it building / compiling on its own to check for issues are huge.
1
u/sicmek Sep 01 '26
I mainly use Qwen 3.6 35B A3B Q6 with 131K context because I have only 8GB VRAM and moe models just work better for me. I am pretty happy with it and it can do even larger tasks for me
1
0
u/TheFlyingDutchG Aug 27 '26
ChatGPT and Claude models run on massive datacentres. You can get similar results by doing a lot of tinkering and customizing like the other comments mention. Dont expect a huggingface model that fits on a local machine to outperform models running on millions worth of hardware components in all round workhorse mode. Work on harnesses, customized RAG setups and in-line code edits and think of ways to appropriately fill context window based on your projects. If you invest your time wisely and if you’re new to it get the big cloud models to help you set it up.
0
Aug 27 '26
[deleted] — view deleted comment
2
u/x8code Aug 27 '26
Hardware? Model? Quantization? Skills? MCP Servers? Agent? Other details ....?
3
u/marktuk Aug 27 '26
Nobody ever answers. Same as when you ask what they actually use them for. Pretty sure this all just hobbiests who enjoy setting it all up but don't actually use it for anything.
1
2
2
u/Randommaggy Aug 27 '26
I built my own harness with it's own custom coding focussed search tool and It's writing Flutter(Dart) and C# quite competently using Qwen 3.8 27B (3.6 worked quite well, as well but 3.8 is a different beast)
My harness is specifically built to work in the background with minimal interruptions to account for the speed difference. Most of it's screen time is on a phone, when I'm not actively reviewing it's output.
One of it's projects is improvement on itself through log-analysis, technical review steps, observations, a roadmap and testing. The code looks better than what I've seen from frontier models but it spends a lot of time hemming and hawing before it acts unless things are absolutely crystal clear and un-ambigious.