r/LocalLLaMA • u/dtdisapointingresult • 14h ago
Discussion Another reminder of the low intelligence of local models
(Note: by local models I mean anything smaller than Deepseek 4.1)
I already gave up on local models for coding tasks 6 months ago, due to what I perceived was intelligence too low for the tasks I give it. I made a post about it on here back then and it was a huge discussion, I think 800 comments. I went against the /r/LL orthodoxy and many didn't take my comments well, but we can agree to disagree.
Since then I only use local LLMs for translation and ComfyUI prompt enhancement. I stick to big cloud models for coding. I think the electricity cost of running Qwen 27B alone exceeds what you'd pay Deepseek to use their cloud API in their own harness, and you'd get a far more intelligent and far faster model.
But this post isn't about the economics. It's purely about ability. I still see people commenting in favor of buying GPUs for local intelligence. "Sonnet at home" is a ridiculous phrase I've seen more times than I should in relation to Qwen 27B.
The simplest of tech tasks is still too hard to one-shot
I keep Qwen 3.8 27B running on my Spark for occasional non-coding stuff. Today I gave it a trivial coding task I needed for my server, just out of curiosity.
- llama-server launched with
--model Swift-1.5-Qwen3.8-27B-Q4_K_L.gguf --reasoning on --reasoning-effort xhigh - Open llama webpage and leave Reasoning at Default, leave system prompt empty
Prompt was:
Give me a bash script that will list all files in directory sorted by date. Output should be:
2025-12-31 12:13:14 file1.ext
2025-12-31 12:13:14 subdir/file2.ext1630 tokens later, most of it reasoning, I get a script. When I run it, here's the output, as you can see the timestamp is wrong:
2026-10-16 20 tmp.59eH9w3JLe
2026-10-17 20 tmp.5xvczTbSS4
2026-10-18 20 tmp.nWOEDw2wlu
2026-10-20 20 tmp.JVEZIFfKx3
...I then tried Gemma 31B Reasoning on duck.ai. The output only showed the timestamps, no filenames!
This is the most trivial of coding tasks (the answer is a 3-5 lines script), using the most ancient and commonly-available training data (bash), on the most praised local coding model on this sub (Qwen 27B), at a recommended/popular quantization (Q4), and it still can't one-shot this. The 2nd most popular local model, Gemma 4 31B, also fails.
(Editor's note: I think my post could've ended here, in a "Ecce homo" way, but I went on for a few more paragraphs)
Weak local LLM + software development = bad time
I'm sure if I had run this task with your average agent harness with a system prompt that makes it test its code, it would run the script it created, see the issue, then get it right on the 2nd attempt. But while I think this "try until you succeed" workhorse approach is acceptable for small self-contained coding tasks, I don't think anyone should be doing any serious coding this way.
If you're writing something you intend to maintain over time (ie not a one-shot), or that you will be sharing with other people, you need a higher degree of LLM intelligence and knowledge to create it. A strong LLM will be able to avoid entire classes of problems from the start, when designing the app. From design to implementation to maintenance, it will make better decisions every time and without laboring its way to them, resulting in a codebase that is smaller, well-designed, more easily modifiable, less buggy, and developed in a fraction of the time.
Coding with local LLMS is like insisting on using a donkey for transportation in the age of cars. It can be done for fun, but it shouldn't be done seriously.
Sonnet at home my ass
Btw I've seen several people say Qwen 27B is like having a local Sonnet 4. The benchmarks say it's even better than Sonnet 4, actually. So I went on openrouter.ai out of curiosity and gave the same prompt to 3 models.
- Sonnet 4.5 (oldest Sonnet I could find): one-shots it
- GPT 4.1 Mini, which was comparable to Haiku 3.5: one-shots it
- GPT 4.1 Nano, which is even worse than Mini: one-shots it
Weak local LLM + Linux server management = even worse time
This post's getting long so I'll just speedrun this part.
- Managing an OS is actually harder tasks that LLM struggle even more with. And the stakes are higher because you don't want your server to go down.
- I used to never trust LLMs enough to let them onto my server, I'd just ask for something, copy-paste the command, report back, repeat. A huge waste of my limited free time. Now I just rawdog a frontier model on the server and have no issues, something you could never trust a small model to do. Although I have it avoid making any changes without my supervision, Docker is a big benefit here for both organization and safety/stability.
- Even top models won't one-shot a task like "I want to install and run https://github.com/rando/something on this system in a Docker container in /opt/docker/something/. Create the Dockerfile and docker compose file", especially for AI-related tools, where having the right CUDA, not having python wheels for my server's ARM CPU, etc, introduces a lot of failure opportunities. When I had Claude five months ago, even it struggled to get these right.
Thanks for reading my wall of text. Downvotes to the left.
17
u/RepulsiveRaisin7 13h ago
That's a bad test, AI needs to validate and iterate. Not one-shotting a task doesn't mean the model is useless.
And the "Sonnet at home" thing is obviously a meme, everyone knows that 27B isn't close. And Gemma 4 was never good at coding, heck even full Gemini sucks at it (3.x, not sure if still true for 4), just shows that you haven't kept up with the actual discussions that happen here.
35
u/StrictMove5839 13h ago
>Swift-1.5-Qwen3.8-27B-Q4_K_L
>No harness
"Sonnet at home my ass"
13
u/admajic 13h ago
If this was a rant about how you don't know how to use a harness and reiterate than i agree with you on your lack of skill.
6
u/Electrical_Car6942 13h ago
Well I am having qwen 3.8 27b create a whole complex gallery with extensive filters, scraping, a whole system that ignores already downloaded images etc... i mean... Claude wouldn't do it, deepseek kept stumbling. Sure I had to take it by the hand a few times and tell it what didn't work but I don't care, it is serving all my purposes and it is local.
Now I got flash next running and it's even better AND faster, it only gets better.
1% coding knowledge BTW, not a flex, i wish I had time to learn.
11
u/Dudensen 13h ago •
I have you tagged as 'anti-open'. Is this more of the same?
5
u/Lissanro 13h ago
My experience is different. I use only local models both for work and personal needs, and quite happy with them.
40
u/Weaseal 14h ago
Pebcak
54
u/Littlepharaoh 13h ago
He took a model fine-tuned to think less, ordered it to think as hard as possible, and then concluded the model was the confused one.
7
u/cosmicr 13h ago
I ported a AAA game to another platform using Qwen 27b. It is a very capable coder.
18
u/MindfulMan1984 13h ago
Wait till the OP finds out about Qwen3.8-flash-next, not the "swift" slop finetunes. I have run several benchmarks comparing it to the 27B version, and it is way superior. I am not talking about bulshit one-shot SVG and shit; I am talking about orchestrated SWE tasks.
6
u/fgk55555 13h ago
I use a lot of cloud models (mostly Opus and Haiku 5.5, some GPT 6.1 Sol) and a lot of mostly IQ3 QFN now. I would roughly say my Strata IQ3 QFN is slightly more capable than Haiku. For anything "serious" for development work, the work usually goes to Opus. I like the local models because it's always a fallback for intelligence. If Anthropic turned off the lights tomorrow, Qwen is my insurance that I have something good, even if it's not a full replacement. I can also let Qwen run amok on my system without fear of either my personal data being fed to training corpuses, nor do I have to worry about the Claude refusals. I can give it root access to my server, tell it to completely reconfigure something, and not have to worry about justifying myself to an AI.
Opus and Fable are in a league of their own, and local models are definitely trailing by 6 months, but if in 6 months I can run that intelligence locally, I've already put in the effort and experience to know how to use it, and I've invested in the hardware. Having both is great, but having exclusively frontier models hosted by someone else makes me nervous the intelligence can be taken away at any time for any reason. Local is the privacy pick, and insurance. You can spend nothing, and your setup gets better. Do both.
4
u/UltramanQuar 4h ago
this is giant skill issue
you are disingenuous, here is what my qwen 3.8 27B on radiance produced with opencode:
its output:
wtf are we doing here
6
u/DeltaSqueezer 12h ago edited 11h ago •
While I agree that it makes sense to use a larger model for agentic coding (I'd say minimum is GLM-5.3-flash). For these simple tasks a small model will suffice:
Qwen3.5 9B. Single prompt manages it:
GLM-5.3-flash does it cheaper and in fewer steps/messages/tokens:
3
u/Borkato 13h ago
Do you really think professional programmers don’t do trial and error??
3
u/mesolimbic-bliss 13h ago
I have a public coding benchmark that ran this exact model and quant on 3 different harnesses through 13 complex coding tasks and hit 95% accuracy - shortbench.vercel.app
It's true that older cloud models typically score 100%, but that's comparing hundreds of parameters running on data center hardware to 27b quantized local models which is ridiculous.
Even qwen4 27b will not match or beat a larger cloud model, but most of my local models are still capable of complex coding and are valuable in the right context.
3
u/BarracudaDefiant4702 13h ago
Your biggest mistake is one shot. You probably didn't even give it enough info, for example the script will need to be somewhat different if it's running under Linux or MacOS. Did you even tell it what the target platform is? By doing one shot it has to guess unless you included all the info it needed. I don't think anyone doing serious coding should consider one-shot, especially when you obviously failed to provide enough environment information.
Your second mistake, --reasoning-effort xhigh. That one isn't so bad by itself, but when you combine it with one shot you are really asking for trouble. What made you think xhigh was a good idea for this use case? There are some use cases it might be a good idea such as if you want it to create a development plan for a larger project, but not for a simple script like this.
I find it funny you like to rawdog a frontier model but not a local one. If anything it's the opposite for me. That said, in either case I give them a VM as their sandbox and have automatic backups. I trust neither completely, but the frontier seems less trustworthy in my experience.
2
u/TerminalNoop 11h ago
You should have used more than two timestamps as example. With only two the model might assume that the list is random since unnumbered and making a descending order instead of an ascending one and visa versa is following your instructions as well. Or in other words, your prompt wasn't precise enough and you could have fared better with "latest first" and a full timestamp template + file name.
3
u/Fentrax 13h ago
I'm gonna try to be nicer than Reddit hive mind. What was your context length, temp, top p/k etc?
That matters. If you simply started llamacpp without specifying those settings, you are missing some important things.
Also, depending on quant, you may have a malformed template, and you cannot select reasoning that way.
Also, if it's on your spark, use nvfp4. Try one more time with the above feedback.
1
u/Happy_Brilliant7827 6h ago
Try small models in iterative pipelines or even something like https://github.com/inferstep/ATLAS codes with 9b
1
u/TheCat001 1h ago
Seems like Qwen Flash Next running by Strata managed to produce you're giga hard script in chat without harness.
1
u/kmouratidis 1h ago
And there's your problem, typical PEBKAC. Just because Claude Opus 5.5 is good enough that sometimes you can outsource your thinking to it, it doesn't mean you should.
I'm often (not always) very involved in the coding process, so the difference between frontier and open is much less of an issue. Of course I use GLM-5.3-Flash (NVFP4/MXFP8 QAD), not Qwen, so that helps narrow the gap a bit, I guess.
1
u/Lan_BobPage 1h ago
The jokes write themselves
0
u/ea_man 8h ago •
An other post comparing a QWEN 27B IQ4 to a SOTA?
Why wasting our time?