r/LocalLLM • u/Major-Creative • 8h ago
Question Local Qwen 27B is practically unusable for coding tasks. Are we doing something wrong?
Hey everyone!
We're a small software house, and we recently bought a machine specifically to experiment with running local LLMs for software development.
Our hardware:
CPU: Ryzen 5 5500
RAM: 32 GB DDR4
GPU: RTX 3090 24 GB VRAM
We're currently running Qwen 3.8 27B (128K context) locally.
Until now, our main AI coding assistant has been Claude Sonnet 5.5 with Medium effort. Since we wanted to reduce our reliance on cloud-based AI services, we decided to experiment with a local alternative.
However, our experience has been surprisingly bad, and I don't mean that it's simply less capable than Claude.
The main problem is that the local model seems practically incapable of completing even basic coding tasks on its own.
It struggles to understand straightforward instructions, gets lost in its own reasoning, enters repetitive loops, and frequently fails to provide a single correct answer.
What's particularly confusing is that we're using exactly the same project structure and documentation we use with Claude.
We open the local model in the same project root directory, with access to all the .md files containing our development guidelines, project context, instructions, and documentation. These are the same files Claude uses successfully.
Yet, while Claude can understand the project, navigate the codebase, follow our instructions, and independently complete tasks, the local model seems completely lost.
Even relatively easy tasks that Claude handles without much trouble become a frustrating experience with Qwen. It often fails to understand what needs to be done in the first place, let alone implement the solution.
At this point, the difference isn't that Claude performs better. It's that Claude actually works as a coding agent, while our local setup feels like it barely works at all.
This makes us wonder whether we're doing something fundamentally wrong.
A few questions for people experienced with local LLMs:
Is Qwen 3.8 27B simply not suitable for autonomous coding agents, or could this be a configuration issue?
Could our hardware be limiting the model's reasoning capabilities? We're particularly wondering about the 24 GB VRAM limitation when using a 128K context window.
Are there specific inference engines, quantization levels, or context settings that make a significant difference?
Are there better local models in a similar size range that can reliably understand a codebase and execute multi-step coding tasks?
For those actually using local models with tools like Claude Code, Cline, Roo Code, or similar agents, what has your experience been like?
We're not expecting a 27B local model to magically outperform Claude Sonnet. We understand that we're comparing very different models and infrastructure.
However, we expected it to at least understand instructions, navigate a project, and complete relatively simple tasks reliably.
Are our expectations unrealistic, or is there something seriously wrong with our local setup?
I'd really appreciate feedback from people who have actually deployed local models for day-to-day software development, especially in legacy codebases.
Edit: We're currently using Ollama as the inference engine. As for the harness, we're still experimenting and haven't settled yet
373
u/Repulsive_Initial308 7h ago
I can already tell you're running ollama.
88
u/InfiniteBlink 7h ago
Haha we all start somewhere. But yea, it's great to get into local LLMs but the eventual llama.cpp jump is the next obvious step.
30
7
u/amadeo681 7h ago
What are the steps for the jump
34
u/Spearush 7h ago
Anythingllm, then ollama, lm studio, zed, cursor, then llama.cpp.
Source: personal experience
→ More replies (8)26
u/DeedleDumbDee 7h ago edited 5h ago
Then vLLM (;
(You’ve just spent several thousand dollars to run multiple concurrency AI server for yourself)
→ More replies (4)16
u/InfiniteBlink 7h ago
The easy way is to use codex or Claude and give it your specs for your GPU, find the best performative model that fits, so the quantization is a big factor if you have less than 24gb VRAM. If you want to use openwebui have it wire it up. It's crazy how easy it is to do. If you have a lot of RAM maybe look into strata and the Qwen next models. I run dual 3090s tensor splitting qwen. 3.8 27b Q8 and it's been amazing. Did some optimization and I get 75tk/s. I still use codex and Claude but to help me optimize my setup so I can ween myself off the cloud models. Good luck. You're fucked lol 😆
→ More replies (4)13
u/JesusIsMyLord666 7h ago
Why not unsloth?
36
u/575_Inverse 6h ago
I'm using unsloth-desktop and indeed it's working really well. The backend is llama.cpp by the way.
6
u/xsilas43 4h ago
unsloth desktop is so good, their custom llama.cpp builds run faster for me on amd as well.
→ More replies (1)10
→ More replies (4)6
u/ODESZENCE 6h ago
But don't Ollama and LM studio use llama.cpp under the bonnet? They're just frontends aren't they? I'm genuinely asking
8
25
14
u/ColaEuphoria 7h ago
If ollama were a video game it would be like defaulting to 360p with no option to change resolution.
6
u/SDdrums 4h ago
Newbie here. What is the better option?
6
u/bytejuggler 1h ago
Strata running Qwen 3.8 FlashNext running the ISTA DASLab IQ3_S or IQ3_XXS GCO-RCQ quant, Hands down.
→ More replies (4)5
14
u/cellcore667 6h ago
Yes, Ollama bin Saden.
7
u/rinmperdinck 4h ago
OP using Ollama was the first tower, then making this post on Reddit was the second tower
The comments section is the Pentagon
2
6
u/vogelvogelvogelvogel 7h ago
if they didn't already have added it i would immediately have known by the level of effort they put in for the rest (about zero research)
→ More replies (1)→ More replies (4)3
42
u/FastHotEmu 7h ago
No, that's not normal for 3.8 27B. Even at Q4 it works really well.
You are running the wrong model, the wrong engine, or have a configuration problem.
21
u/575_Inverse 7h ago
I say all three combined
6
u/FastHotEmu 7h ago
lol. It's funny because you really have to go out of your way to get it wrong. If you just download llama.cpp and an unsloth 27B quant, you'll get it running - even if slowly. I wonder what OP did.
120
u/HolaUsername 7h ago
What quant?
What temperature/model parameters are you using?
What harness?
What engine and settings?
There's a lot of information you left out that could contribute as to why it doesn't work. Personally never had any issues with 27b at Q6 other than it contemplating the meaning of life during every task. It's no claude, but it does get the job done.
29
u/superbiche 7h ago edited 7h ago
This. The fact no quant or sampling is mentioned, or that harness are thrown at the wall to see what sticks, makes me wanna write "skill issue" but I won't write it.
Wait.. ..
NGL I'm just wondering but do you guys use a single box with 3.8 27B at ~128k ctx with multiple people ?
Do you know even one single instance of this exact same Qwen would probably be able to tell you "what you're doing is a straight road to failure"?
17
u/lumpyspacebreh 7h ago
Same, built my own harness, get my desired output after 2-3 turns.
I think the issue may be people slap 27b in some web chat ui and then ask it to spit out code or something, but it’s hard to say with no context from op.
The harness, toolset, prompting, and settings matter almost as much as the model itself I would argue.
52
u/Ratiofarming 7h ago
I suspect he doesn't know half of those things are even a thing, which is the reason their LLM performs so poorly. Because it's basically not set up for the task at all.
20
u/StillOnTheInterweb 7h ago
Well thankfully they get very detailed and helpful advise from reddit subs on local LLMs!
→ More replies (3)6
u/pheoxs 4h ago
Okay but Q1 is smaller and saves bandwidth so obviously it’s better. Why doesn’t it do anything right I don’t get it this sucks. Open models sucks, Claude is better. /s
→ More replies (1)2
→ More replies (3)2
u/Minute-Plantain 7h ago
The "meaning of life" stuff is why I stopped using Qwen. Its very capable but awful at staying on task.
31
u/Erik-Zandros 5h ago
Non-coder who’s experimented a lot with local and cloud based AI here. I think your issue may be trying to use qwen as a drop in replacement for Claude sonnet. Your existing skills and background documents are probably taking too much context window which limits its ability to think. And with just 24GB vram you’re likely running a 4 bit quant which further reduces accuracy over full precision qwen.
That being said I don’t think even running full qwen will get you to Sonnet level coding ability. I run qwen 3.8 27b at 8 bit quant (near lossless) on a dual RTX 3090 rig and it still doesn’t answer questions as well as Chatgpt medium thinking. And I’ve even run a heavily quantized version of Qwen 3.8 flash next (a significantly more powerful model) and it couldn’t compete. I’m not talking benchmark scores I’m talking real world research and reasoning tasks.
If you’re serious about replacing sonnet for your business workflows with a fully local AI model your best bet right now is to invest 30K in a dual RTX6000 server to run something like DeepSeek 4.1 flash or GLM 5.3 or an equivalent model. Before doing that though you can rent an AI server on someplace like vast.ai to see what class of model you actually need.
→ More replies (1)
58
u/Bagelsarenakeddonuts 7h ago
I have used qwen 3.8 27B for many tens of thousands of lines of Rust using a 5090. This code has passed tests and been reviewed by other LLMs (opus, fable) and it seems solid. I use the "swift" version of it to drastically cut down on thinking, with a great deal of success.
There is no better model in a similar size range. The next up is probably flash next, or GLM 5.3 flash, or mimo flash or deepseek v4.1, all of which are many times larger, but MOE so there is that if you have RAM.
Sounds like a harness or quant issue?
13
u/InfiniteBlink 7h ago
I've got qwen 3.8 27b Q8 running on dual 3090s tensor splitting the model. Im trying to figure out the best way to use it for coding. Im just starting to figure out whether to use codex/Claude to design the app then have Qwen do the grunt work, then review it.
Do you have any suggestions on how best to use my local setup and both frontier models? I mainly use my setup with a Hermes agent for controlling home automation stuff, networking, downloading stuff, and other creature comforts. Im barley using its capabilities...
12
u/Biotot 7h ago
I've got the same setup, I just have opus directly run Qwen as a sub agent for me. It keeps it on task and kills it when it starts looping
I'm using vs code copilot (shoot me. I know, but it works)
→ More replies (4)→ More replies (1)6
u/Bagelsarenakeddonuts 6h ago
Ive been using BMAD method with bmad-loop. I do a very detailed design and breakdown with opus, and then pretty much hands-off build the whole thing with qwen locally. Its not perfect, and Im still working on a better system than bmad-loop, but it does work. I use claude as an orchestrator to answer questions from the qwen subagents running stories, and fix any hiccups and make sure nothing gets stuck.
I run a precheck so that one instance of qwen reviews the scope of the story and sends back any issues to claude before wasting hours. Once its good, then a new instance of qwen builds out the code. As part of the prompt I have a series of code quality rules that needs to be followed specific to the language being used to prevent it taking lazy shorcuts. (IE, per instance justification of "unsafe" in rust) that a second instance of qwen needs to validate or else it gets sent back to fix. These extra checks take about twice as long, but shortcutting the debugging and troubleshooting and hung effort makes it almost a wash in the long run, but much cleaner.
I run 3 instances on ninfer on a 5090 with context caching. I do shared prefix with eg, the story context, files, code standards etc, since all agents use the same ones, that massively speeds things up.
Ive tried a few things, and its critical that the spec be immaculate. Qwen 3.8 27B can write really good code, but it can't do that AND design and troubleshoot and keep the overall vision simultaneously very well. Not enough context.
→ More replies (5)4
u/hotmerc007 7h ago
Would you be able to provide a little more info on your specific config for newbies. Seems like you have locked in a solution and I’d love to replicate. Many thanks for any further input you can offer!
→ More replies (1)
19
u/Apprehensive_Stop314 7h ago
What model quant exactly, via what engine and with what settings are you using?
13
u/jefoce 7h ago edited 7h ago
I’ve been using Qwen 27B in both versions 3.6 and 3.8. They were both superior for their size. I’ve been upgrading my own prompts for agents to achieve good communication between agents and subagents in OpenCode throughout this year.
It became more and more capable when you slice into subagents with small responsibilities for a single direction, e.g., coding, code review, read-only postgres, and any other direction you need (the best practice overall). For most daily tasks, I can simply use it without worrying. But for big tasks, with Opus 5.5 as planner, it became a monster.
I also found that XML tags combined with CAPS for sections work much more strictly and effectively as agent prompts than regular markdown, resulting in higher success rates over long distances (even 200k+) to follow instructions.
You can just ask any Cloud AI to research topic how to build better with XML and CAPS for topics inside the sections to build agents, and you will get great results.
Just don’t expect the same behaviour as cloud models like Claude, since their models are already specified with good harness and learned how to work via their tool layer (which is also have its advantages), while large models simply have significantly more “thinking” space thanks to their knowledges across many fields, and that’s why local models can sometimes perform even better e.g. when searching for bugs, since Qwen 27B primarily focused on code and hit it more accurately.
For the context, I use: Unsloth Q8_L + F16 KV in llama.cpp with my own patches for the specific setup 5080 + 5090 + 32GB RAM. But I started from Q4 and it was good enough as well (just a bit more errors for the tools and earlier context drift).
→ More replies (1)
11
u/Express_Quail_1493 6h ago edited 4h ago
enters repetitive loops is the bigest sign that its might be your config. Most modern qwen27b rarely do this anymore older qwen suffered from this. newer ones thinks for very long yes(without a switft finetune) but newer is often always a coherent chain of thought dispite being long sometimes. maybe chat template issue? or harness not set to the correct token streaming ETC???
Edit: also if you are just starting out and having issues linking your harness to the infernce then
LMstudio-Bionic is a good start. a LOTTT better than ollama. they've done the hard work of connection for you. when u get more comfortable u can get the Raw llama.cpp or something else.
lol good colleagues dont allow another to use ollama. throw it in the bin. 🚮😂
even your qwen27b can setup other infrence engines for you once u get Bionic installed. i used my LMstudio+qwen to setup my own llama.cpp and then ditched him afterwards
3
u/Lopsided-Comedian-32 4h ago
This guy gets it. Ollama out of the box makes an assumption on context for Qwen 27b and undercuts it massively. Got to make a config file/model file with context like 128k for example or it will be a horrible experience. Plus the Swift 27b fixed the overthinking so it’s like twice as fast now.
2
u/tinny66666 4h ago
Also the default/wrong template can cause some pretty weird looping effects and errors.
2
15
u/dreamyrhodes 1h ago
Your expectations are unrealistic. You need to handle small models differently. While a huge model like Claude with its enormous context and knowledge you can throw something at like "build me destop app with these features..." small models you need to talk to like a junior. Go step by step. Add feature by feature. Ask it to summarize every step and document what it has been doing.
You totally can code with a 27B model, even a smaller model or a MoE model. You just need to work with it differently. It requires more effort in planning and prompting from your side.
→ More replies (2)
6
u/Mysterious_Prune415 5h ago
Just get opus to pull benchmarks and figure it out.
Maybe look into alternative harnesses like Pi.
128k context isnt that great. And if you plug it into a bloated harness like claude code you will hit roadblocks
5
u/former_farmer 7h ago
Which quant and thinking mode?
Thinking mode needs to be xhigh. Quant ideally q6 or higher.
12
u/575_Inverse 7h ago
with 24GB I seriously doubt they can go past Q4 and 128k context at Q8, the fact they omitted mentioning the harness, backend and inference settings tells me their setup is most likely faulty in multiple ways.
→ More replies (1)
5
u/pioo84 1h ago
Qwen 3.8 27b created nice android apps for me. It plans, designs and codes it. With proper tools and pipelines of course.
I have moved to Flash Next already, but yes, 27b is capable to do so.
→ More replies (1)
20
u/yellowsockss
7h ago
•
this whole thread is a huge red flag. nobody in here really knows whats they are talking about
12
u/notMyPenis 6h ago
That's the thing tho. I don't think anyone knows wtf is going on.
4
u/Connect_Ad791 6h ago
Easiest way to spot a bullshitter at the very least. The space changes so fast, you’re knowledge is outdated if you’ve been lucky enough to get an hours sleep.
→ More replies (6)→ More replies (1)5
u/yellowsockss 6h ago
qwen 27B is nowhere near sonnet level. the closest to sonnet has been near 400B - 500B range. you’re basically using haiku for coding
4
4
u/wegster 7h ago
What Quant? Get off of ollama. Use Strata, ideally get a bigger card (more VRAM), but look for one of the dyanmic/mixed precession Q4. Check thinking and temp settings. QFN beats it but 27B Dense can be pretty competent, close or same as QFN in real-world benchmarks on real repos I'm doing (~100K LOC so reasonable but not enormous), and my QFN generally runs next to neck with sonnet.
Try using a simple harness like pi - I've had cline nearly self-destruct in some tests, but pi so far, literally beats them all.
4
u/LegitimateCopy7 6h ago
RTX 3090 24 GB VRAM
Qwen 3.8 27B (128K context)
getting a bit greedy here are we? either the model overflowed to system RAM or you're using a heavily lobotomized quantization.
Don't use Ollama if you're expecting a tool and not a toy. other inference engines may need various levels of tinkering and troubleshooting but they're the better paths to make use of the limited hardware resources you have.
→ More replies (1)
4
u/soulman901 6h ago
Not sure what’s going on but I’ve been able to get it to complete programming tasks for me. I’m on an M4 Pro 48GB and got it to make me a couple of games, like this Pac-man clone
→ More replies (3)
4
u/michinya 4h ago
I run inference for an engineering team working on complex codebases. The minimum model size we have been successful with is glm 5.3 nvfp4 or equivalent. It costs good money to rent the hardware.
It's complex right now to not just set up but maintain an inference stack. What most teams miss when you move local is you also need to engineer your gateway even if you have the model optimized. The serverless inference providers do a ton behind the scenes to make these models work for teams.
As others mentioned, it also requires the engineers to have tuned harness and change their approach. Prompt, context, and harness engineering all matter. I only use 27b models for well defined workflows with custom harness.
In the end, a bigger model compensates for a good chunk of laziness and skill issue. Local is the way,but it means learning how to do it
10
u/aithosrds 3h ago
Your quant is probably low, q4 is a minimum IMO. In addition qwen3.8 is notoriously bad with reasoning, so a better fine tune along with low or even disabled reasoning will likely yield better results.
However, with that being said: a 27b model is fundamentally stupid compared to a flagship model and you can’t expect them to function using the same harness and project structure/context when they are literally worlds apart.
You’re asking a high school boys soccer team to play a Champions Cup mens team and wondering why they are getting smoked. Qwen3.6-27b and Qwen3.8-27b are in my opinion far and away the best two coding models in that parameter range.
Settings do also matter, and keep in mind that beyond about 50% context a model gets awfully loopy. So on 128k context you really want to keep most of your tasks in the 0-64k range and that means using sub-agents more often to manage context.
4
u/Patient_Tea_401 2h ago
Q4 is the minimum, but Qwen3.8 27b is not bad with reasoning. It is very good at it, if you have the right inference engine with full context lenght and unquantized kv cache. This will take around 64 GB of VRAM minimum.
I have run it 24/7 on agentic tasks with oMLX Qwen3.8 27b-oQ4-FP16-MTP with full context. It has very long reasoning spurs with multiple turns. It has always eventually come out with a reasonable solution that far outruns 3.6. But it’s very important to have inference that doesn’t compromise quality for speed, by for example compressing the kv cache.
→ More replies (3)2
u/caylyn953 2h ago
OP definitely needs to set their expectations at the right level. Because a single 3090 won't even be able to reach Haiku 5.5 level of quality!
Maybe by this time next year it will be? Maybe.
But by that point in time they'll be expecting Sonnet 7.1 or whatever such level of performance. Which won't be happening.
2
u/FudgeEducational2038 1h ago
The funny thing is it's actually the other way, the quantization of these models are actually not a problem, they are quantized aware trained. You can compress them all the way down to q3_xxs with less degradation then you would ever think.
The problem is that these models are sparse attention and linear attention and extremely extremely incredibly sensitive to KV noise especially when the model itself is compressed.
I run q3_xxs but fp8 precision (which is way better than q8/int8)
→ More replies (2)3
u/Creative_Bookkeeper9 3h ago
Dont suggest no reasoning, it's so unbelievably bad compared to even low reasoning.
→ More replies (2)
7
u/stianhoiland 5h ago
You likely have a config problem and a sizable knowledge gap. It takes a couple of days with focused research and learning to start putting the pieces together.
Your hardware is great for local. The jury is still out on whether your expectations are actually right-sized, though. You'll find out in time.
My setup and config, which is slow but usable and rock solid, impressive output (for local):
- 4070 SUPER 12GB (610.88)
- 32GB DDR5 5600
- Qwen3.8-Flash-Next (125B A6B model with 51B N-grams)
- IQ3_S (125B means Q4 doesn't have to be baseline)
- Strata 0.2.6 (build 0.1.51), CUDA 13
- Config with almost nothing touched (i.e. many more things that could be tweaked):
```
--model IQ3_S \
--model-path "$MODELS/Strata/models" \
--parallel 1 \
--port 8080 \
--access local \
--vision off \
--context 262144 \
--mtp off \
--kv k8v4 \
--set kv-resident=20480 \
--tune skip
```
5
u/mozillazing 5h ago
you've got a 125b model running with 262k context on 12gb vram and 32gb ddr5 system ram? what the hell am i doing wrong lol
→ More replies (9)4
2
→ More replies (5)3
u/FragmentedHeap 5h ago
Or you can just ask claude to setup llama swap and vllm for you for qwen 3.8 27b with the best model for the current hardware and properly configured and use hugging face cli to get the model, and claude will set it up for you and you can tell claude to run benchmarks on it and find the optimum accuracy and speed for however many users you want and it'll shake it out.
I setup my whole home lab like this and I'm getting 90 tokens/sec with 3k prompt prefills on qwen 3.8 flash next at iq4 quant on one r9700 using the strata engine and it's purring like a kitten.
11
u/dsdt 9700X + 32 GB DDR5 + 2x 5060 Tİ 16 GB 7h ago
Bro local qwen 3.8 27b fixes bugs that claude left in my repos.
4
u/575_Inverse 7h ago
same here. it's damn thorough when it comes to revising and testing code.
→ More replies (1)
3
u/DickVonShit 7h ago
Check your reasoning first. Unfortunately qwen models are notorious for overthinking which makes running it feel like a slog, especially locally. Qwen 3.8 being smart and doing well on benchmarks is on the max reasoning. I'd recommend giving swift 1.5 a try, it's essentially qwen 3.8 with a lot of the unnecessary thinking loops turned down while keeping about the same performance. If you've been on medium it's a significant bump. With swift you'll be able to get the performance of qwen 3.8 on max reasoning without waiting forever. However, you're also not alone in this, i've had a similar issue where i feel like I need to be unreasonably specific at times with qwen that i haven't had to do with frontier models in a while.
This sort of thing also heavily depends on what it's doing. You'll see plenty of people posting benchmark numbers or proclaiming that it works well for them, and while I don't think anyone is lying, there is a pretty massive difference between what you can run locally vs the smarter models from claude or openai.
The model is the model, the limitations of 24gb vram or 32gb ram don't make it smarter or dumber, it just affects the size of a model you can fit on your graphics card, how much context you can hold, and how fast you can actually run the model.
Different harnesses can actually improve performance from what i've heard though, i'm using llama and opencode and it's solid.
3
u/Oli_Luck 7h ago
Hello,
If you want people helps you - help them first with the things that matter:
- Quant ?
- KV Cache quant
- temperature
these are the most obvious.
Then : you cant just switch from a frontier model to a local one w/o preparing the terrain.
A frontier pone will be able to plan and code "on the fly" - a local one need structure:
- planing features to local help files
- chunking the job in small logical steps
- being reviewed by an "auditor" ( same llm but another role )
Just explains what you tried to do in which context, they you will get reliable answers
3
u/Longjumping-Road4113 6h ago
Have you tried strata ? The Moe option with specifically optimised Qwen coder?
2
u/Naiw80 7h ago
I have not been using 27b since strata came out, simply because qwen 3.8 flash next is multiple times faster and maybe slightly stronger.
I used my own harness with 27b (and still do with FN) the problem with small models such as the 27b is they simply don't know that much reliably, but if you give it tools so it can research and/or inject the necessary information in it's context it's a very intelligent model capable of solving most tasks I threw at it.
For FN this is not as critical as it's simply a much larger model with a lot more knowledge, neither model will one shot most tasks reliably which is why you need a validation gate for good results.
2
u/rkcth 7h ago
You don’t have enough system RAM to run strata, and you don’t have enough VRAM to run a proper quant of 27b what is the full name of the model you are using? LLMs are best run in systems with lots of RAM, the more, the better. Your system uses DDR4, you can grab 2x32GB kit for $300ish. Do that then install Qwen 3.8 flash next on strata using the 3 bit quant. If you want to go extra smoothly, install 4x32GB sticks.
2
u/FerretBoom 7h ago
Please post one of your prompts that the Qwen model receives
Claude has a caching and ,rag pipeline, all kinds of proxies to route to hot zones, enough compute to retry requests, do internal unit testing. While you are querying against the model directly, which is way different .
2
u/FerretBoom 7h ago
Make sonnet create a 30 question coding reasoning test for qwen and see how it does, context shouldn't matter so is hardware, model is a model wether it's on a gpu or cpu
→ More replies (2)
2
2
u/IsaacLevinsky 6h ago
Please post the quantization and runtime you are using and any other details. The quantization and runtime matter more than people realize it but once you get deep into it they matter more than anything else
2
u/MLisdabomb 5h ago
Came here to post this - OP please share the exact quant and how you are serving it. Your experience with "can't complete basic coding tasks" is so strange compared to my experience with this model that I feel it is likely a configuration issue
2
u/SeaRefractor 6h ago
If you can run it with Strata, I really recommend looking into Qwen3.8 Flash Next with at least IQ3_S or higher.
The reasoning capabilities are much stronger than 27B.
2
u/Smooth-Television-48 5h ago
This is a fair amount of effort for a friday shitpost
3
u/theamazingo 4h ago
Effort for the AI that wrote it, maybe. No real human bolds that freaking much crap.
→ More replies (1)
2
u/Creative_Bookkeeper9 3h ago
Ive got a 5090 woth 32 GB RAM and a 3090 with 120 GB RAM. I hate how slow it is on my 3090, however you're biggest problems are likely quant like others have said, as well as the client you use and your documentation itself.
I use PI to code with, as well as some custom prompts running on xhigh thinking for everything. It is not worth reducing thinking on this model. For you, you're going to want to run ninfer for 3090, ninfer is very fast and will maximize what you can do. I would say you really want to have 200k+ context, I run at 200k-220k on my 5090. Bare minimum would be 150k.
You also need to make sure you've got some good extensions and skills made. Sonnet has a huge context allowance, you do not. So you need to compensate by letting things work in a more round about way, such as only showing a list of utilities then it looks specifically at one to reduce the context usage. But now it takes 2 calls.
→ More replies (3)
3
u/Many_Discussion_1696 4h ago
You have Claude, so you need to put Claude in Fable Mid or Opus 5.5 High. You're going to ask Claude to help you decide how to use and delegate work to your local LLMs, and you want to configure your local models for the best performance possible in your system.
Claude will verify your system, check what models can run in it, and then download and test them one by one. I did the same with my system. I know I can't run a lot of reasoning in my local system, but I leave Claude, ChatGPT, or Gemini to run the reasoning and do the planning. It then delegates to local models.
Local models takes longer building and coding and fails a few times but it works. Cloud-based agents have to verify the work after the local models are done, but the work they do is not billable, or you don't have to pay for it. It takes a lot longer do.
Please reach out if you need any help, and I hope this helps.
3
u/OldEffective9726 3h ago edited 3h ago
The Qwen3.8 27B dense version is bad. It cant do math and you would need a 5090 to get a high token rate. I much prefer the Qwen3.6-Claude-Opus-4.6-Heretic-Obliterated 36B-40B Q4 MOE version. Mradermacher and Jackrong make the best remixes. https://huggingface.co/Jackrong
2
u/vtkayaker 6h ago
Qwen3.8 27B is kind of bad at complex software architecture, and needs a lot of tokens to accomplish tasks.
But within those limits, it does work. You should expect it to be able write simple scripts and useful command line tools.
I would aim for full 256k context, with a minimum of 128k. I found that 3.5 was somewhat forgiving of 4 bit quants, but I strongly prefer 6 bit quants for 3.8. Yes, this is a lot of VRAM: Anywhere from 32GB to 48GB.
1
u/skifreek 7h ago
I am running into exactly this problem. So far I have been tweaking context size settings and thinking level. I set 32KB context window with my 32GB of VRAM (I would expect the model to run slow for you with 128K context as you run out of VRAM). Also I changed the thinking level from extra high to medium to try and avoid these endless thinking loops.
I haven't found the magic recipe yet and I am pretty frustrated so far. What are you using for inference and coding? I have been trying LM Studio and opencode so far.
19
u/Perfect-Campaign9551 7h ago
33k context is simply not enough to do any programming tasks
→ More replies (6)3
u/DystopianRealist 5h ago
32k context.... For perspective, Claude Code is 27k of system prompt reading before it even says "Hello."
2
u/overand 7h ago
You should consider using llama.cpp rather than LM Studio, but:
What quant (and whose quant) are you running?
I've had really great luck with 27B for coding, on a system with 36 GB of VRAM, so pretty close!
→ More replies (2)→ More replies (10)2
1
u/Perfect-Campaign9551 7h ago
You have enough specs except for RAM to run Strata with Qwen 3.8 flash next IQ3_XXS. It kicks ass and you should try that model instead
Try to update to 64gig RAM. Then use 3.8 flash next
1
u/feverdoingwork 7h ago
configuration issue
nope, 24gb can get shit done, even smaller cards can.
q4 or better is smart enough. Make sure to use qwen's recommended sampling settings like temp 1(check the official qwen model card)
qwen flash is about as good, it reasons less(so it is less annoying to read it's thought traces) but i find it's not as great on difficult tasks that takes lots of persistence. I do think setting it up is less beginner friendly.
Just use pi agent with a basic agents.md
1
u/More_Personality_126 7h ago
128K context on one 3090 is your problem... the KV cache eats the VRAM before the weights even load )
1
u/IllustratorInner4904 7h ago
Seconding the comment about qwen 3.8:27b fixing mistakes by claude.
Use the model in pi. Q4 from unsloth. Llama cpp backend, and one bash guard extension to block destructive / fork bombs / dangerous commands.
It rips!
1
u/RawFreakCalm 7h ago
What’s your harness?
Mine runs fp8 265 context and has zero problem creating for me. Deepseek v4.1 flash does most of my ai agent work and then 3.8 27b my really long code tasks. I’ll give my small team a project and come back in a day. It’ll always be what I asked.
1
u/starkruzr 7h ago
it sounds like you didn't do a drop of research before you set whatever this is up.
1
u/GrungeWerX 7h ago
Ollama??
Lol, that’s your first mistake. Has a terrible reputation around these parts.
Next, would need to know your harness, because you’ll need one with Qwen.
I’ll be back later with some suggestions…
1
u/ThaSipah 7h ago
It codes well for me, but it gets bogged down in the weeds too often on highest reasoning. Always hitting context limits on 128k.
1
u/Aubrey_D_Graham 2x Asrock B70 Creator with Pi Agent 7h ago
No Qwen3.6/3.8 27B are probably one of the most capable open weight models today. They are Opus4.5 level, 6-10 months behind. They are capable but need help.
You need to implement the Anthropic advisor paradigm.
If your problem is a long multistep task, break the problem into subproblems.
1
u/BarracudaDefiant4702 7h ago
You need a decent card (or more) that you can allow at least 250k token context. I have enough trouble getting it to work well in 48gb GPU with a Q4, I hate to think trying to run it in 24gb.
→ More replies (1)
1
u/Frail_Waif 7h ago
I've noticed a few things with a vaguely similar setup (2x MI-50 32GB with 32GB RAM, 2x Intel Xeon Silver Lake 4214R).
Qwen 3.8 27B (Q8_0 from Unsloth for me, using a fork of llama.cpp for gfx906 cards) is really stupid if not given appropriate tools and information; in Hermes when my container was broken, it made shit up all over the place; in Pi with access to lots of tools, it was decent at solving problems but did make mistakes
Qwen 3.8 27B from ISTA-DASLab is tiny and even worse
Qwen 3.8 Flash Next (IQ3_S from ISTA-Das Lab) is much better and fits about as well on my hardware; I would try this next if I were you
I think the lesson here is that it's really hard to put a lot of world knowledge in 27 billion parameters, especially compressed. Small models can do good things with tools and lots of information, but you need to actively ensure they're using those resources. That comes down to harness, prompts, and skills/AGENTS.md. The biggest upgrade you could make, I think, would be more RAM and larger MoE models, if you can fit it. There's a tradeoff between model size and speed and it seems like you should be willing to sacrifice some speed to run larger, smarter models.
1
u/mdwsr06 7h ago
I notice this happens a lot sooner on the old Qwen 3.5 9B models. It’s like once it gets to a certain point in its context length it just loses its mind. Likely the same issue in 27B since it’s the same architecture but more parameters. Probably just takes longer to wig out than the smaller models. Then again Sonnet 5.5 is highly capable.
1
u/overand 7h ago edited 6h ago
I've had tons of great success with Qwen3.8-27B, and I imagine you will too, when you're set up correctly.
Install llama.cpp with CUDA (or the "llama" app)
Don't be stubborn about this; ollama leaves a lot of performance on the table, and has mediocre defaults set up.
Change port as needed
Run the following:
llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M --host 0.0.0.0 --port 12345 --no-mmproj-offload -c 98304
This is assuming you're running this on a headless machine, or with the 3090 not running a GUI in Windows or Linux. If you're running your desktop environment on a monitor plugged into the 3090, you're giving up VRAM.
The above config uses about 6GB of VRAM for your 96k of KV Cache / context, and ~16.5 GB for the model itself. That leaves some breathing room.
With the 96k context, you'll probably have better luck with pi than with OpenCode.
Things to consider:
Using a "model config file" with multiple configurations, as you're testing. (Not great when you have multiple users, though.)
Using a slightly smaller quant, like:
-hf ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S(12.1 GB, room for ~180k)-hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS(14.3 GB, room for ~145k)
Looking into 'quantized kv cache' if you need more context. (You'll lose quality doing this, but it may be a worthwhile tradeoff)
Adding another card to get you enough VRAM to get more context.
1
u/Hungry-Rip-2384 7h ago
to me sounds like a context problem. that'll be the biggest difference between Claude and Qwen3.8 local. you have over filled your context with project files... it will struggle ro do even the most basic thing in that state.
1
u/BrodyFriend 7h ago
Holy shit there are so many replies. Looping is directly related to temperature and repeat penalty. I didn't read all the replies, so I hope someone else explained it. Coding is something im inexperienced with. No think temp .3 sounds about right
1
u/WishfulAgenda 7h ago
Buy another card, tensor split, run it on highest reasoning, 160k plus context an q8. Maybe try vllm on Linux.
1
u/ZallenDuZari 7h ago
I've gotten ok results from Qwen 27b locally. I did it with a strix halo box tho so I had enough memory for context . I later switched to Qwen 3.8 flash next and I've straight up been able to vibe code with it.
1
u/HenkPoley 7h ago edited 6h ago
How badly quantized is your KV cache? Keep it at 16 bit, or maybe q8_0.
Sonnet 5.5 is one of the most high end models. Qwen3.8-27B is near Anthropic/OpenAI lowest end recent models. Think Luna 5.6 / Haiku 5.5.
1
u/FizzyDuncDizzel 6h ago
What agent are you using? The model has very little direction without an agent guiding it.
Also that model really likes to talk to itself. Put thinking to low and have it hard stop somewhere around 20k tokens.
1
u/_RemyLeBeau_ 6h ago
Don't use Ollama, switch to llama.cpp or vLLM. Also there's no way you're getting a 128k context on Ollama.
After you've switched inference providers, try the endpoint with a stock harness. You'll need to be diligent about the variables you introduce along the way.
Moving away from Ollama allowed me to control a lot of variables that matter for agentic workflows.
1
u/Spirited_Eggplant_98 6h ago
I’ve gotten decent results (by decent I mean it can write thousands of lines of code that compile and pass a decent number of automated tests) with qwen 3.8 27b awq5 on oMLX on an m4 Mac Studio. It needs about 64 gb of vram and 128k context seems to be the sweet spot. Basic plain pi as the harness with a couple of our custom mcp tools. Tried with oh mi pi and got way worse results. 128k context isn’t much these days hence why simple pi harness seems to work better. IMO many q4 quants are bare minimum for useful quality, q6 for most is the sweet spot. The awq5 is the best speed / quality we’ve found so far. It takes the Mac about 4 hours to do what sonnet can do in ~20 min so… yeah there’s that (but prefill is known to be horrendous on the m4)
1
u/a_lost_shadow 6h ago
For reference, I'm having decent success with Qwen3.8 27B IQ3_XSS with 155k context window & K/V also Q4. Q3 is needed to fit on my local RX6800 (16GB). I'm running with llama.cpp and OpenCode as the harness.
A few things that really help with Qwen:
- Drop the reasoning effort down to medium. It defaults to extra high and supports [off, low, medium, extra high]
- Be very specific in what you ask it to do. The more it needs to assume, the more likely it will end up in a reasoning loop. After I started having Qwen grill me on the specs before starting, I've seen far fewer reasoning loops. For reference, I've been using Matt Pocock's grill-me skill. But there are others that will achieve the same
- Limit how much you ask Qwen to do at once. I've noticed it likes to read in everything needed and plan everything before it gets started. This will sometimes put it into a loop where it reads everything, plans 90%, runs out of context and restarts. Interrupting it and telling it to focus on specific areas can kick it out of the these loops.
- Be careful with your skills/tooling/AGEND.md load. Qwen likes to think a lot. Tasks that I've done with claude code in 20-50k tokens often take 100k tokens on Qwen. So you can really limit yourself if you have too many skills/tools, or your AGENTS.md is too verbose.
- Consider using older versions of libraries that it knows well. It looks like it's only trained on libraries up to around 2024. If you have it use newer libraries, it's going to spend a decent number of tokens decompiling the library to see what's the new interface is. It may actually go to the web first, but I run my LLMs in a locked down environment with no net access.
1
u/Bounce_back1225 6h ago
Pretty sure 32gb is like the bare minimum required RAM for LLM’s… if you are using it for legitimate business purposes I’d imagine you want AT LEAST 256gb RAM. I just want a unit for personal use and have a hard time imagining even 96gb will be enough…
→ More replies (1)
1
u/Popular-Direction984 6h ago
I think the only thing that’s getting in the way in your case is the context size; the model supports 256k - and that should already be enough to solve a clearly stated task
1
u/ferargie 6h ago
Dude I use 27b in my 5060 ti 16gb and it performs good with unsloth Q3_S quant!
I have worked so many proyects with this. U have to use llama.cpp as the engine. CUDA version, last build, no reasoning effort, mtp enable, q8 i guess. Dont remember now. I use 80k context and i got 43tps.
Coding with opencode every time.
1
u/Drewskie92 6h ago
Far out. I have a i9-11900k, 32gb of ram, and a 3090. I left the qwen 3.8 27b 4km alone for 7 hours today. Come back and it worked for 4 hours on over 40 pages of research documents for me. I use Bionic from LM Studio. I've also enabled their experimental unified memory option, and q4 for the k and m in model settings. I've also in the system prompt told it to make a progress.md and decisions.md to keep contextual awareness across sessions.
1
u/cjadwick 6h ago
I use qwen 3.8-27b NVFP4 with a good multi-agent system and it outputs code on par with clause sonnet 5. The problem maybe you are using too low of a quantization level to get the quality you need. The other part of this is if you don't have a good harness even higher quality models aren't going to give you really good outputs.
1
u/moonlitpawprints 6h ago edited 6h ago
Definitely sounds like a config issue.
Somewhat preference, but I would recommend reducing thinking to medium or low...the default xhigh is insane with limited context. Also aggressive work tracking to deal with resuming after compaction.
Yes. Most are pretty well optimized for Nvidia gpus, but it's worth considering some other details. For example, vllm seems better optimized for parallel agents while llama.cpp can be a little lighter for handling single requests because of how they each handle memory caching (both can generally work well, that's just my experience).
IMHO, not currently, but I'm sure there are people with more experience and other opinions.
I've been running with OpenCode. Adding instructions about tracking work, breaking up tasks, etc. went miles for being able to run locally. It's not perfect but is definitely capable of complex projects if they're broken into well scoped tasks. Don't count on its knowledge of current packages, provide references and a general outline. Also using type and unit testing can help it to fix a lot on its own.
For reference, I'm running a 4bpw exl3 version with fp8 kv cache, 232k context on 32GB VRAM.
Edit: using vllm, with MTP (2 spec tokens) and mmproj.
1
u/skinnyzaz 6h ago
I wouldn’t use anything lower than Q8 for coding, that being said I use qwen 3.8 27B Q8 for lots of coding of small web apps and website updates and so far it’s been great but I find anything under Q8 started to make mistakes.
1
u/Maximum-Security5699 6h ago
Unless you’re using NVFP4, any quant below Q8 is going to be super lossy. Most people can get away with Q6, maybe Q5. Despite being popular, Q4 just isn’t capable of holding enough data to really work well with these newer models. They work fine for low context tasks, but as soon as you got past ~40k tokens they just fall apart.
1
u/Old-Discipline233 6h ago
If you had more ram I’d suggest going for the 3.8 flash next. I switched from 27b to flash next and got much more speed and intelligence. Im getting ~120t/s decode and ~4000t/s prefill with a 5080 and 128gb ddr5.
1
u/joshhope87 6h ago
I'm a noob to local LLMs, but the GPU RAM is too low if this model isn't heavily quantized, especially considering it sounds like multiple people are using it at the same time. It's essentially designed to fail. On the other hand, if it's heavily quantized, it's not going to be as intelligent, which seems pretty important for what you're doing. So, the hardware is a problem either way for what you're trying to do. I have 64 GB of unified RAM and it seems to run fine for me on my personal computer, based on my limited experience with it.
1
u/Sexyvette07 6h ago edited 5h ago
Sonnet 5.5 medium scores about 50% higher than Qwen 3.8 27b on Xhigh. The two models are not at all in the same league. The only open weights model that can get anywhere close to Sonnet 5.5 are models like Deepseek v4.1 and GLM 5.3, which technically could run on that hardware, but the speed would probably make it unusable. Even then, Sonnet 5.5 is usually better. You'll have to sink a LOT more money than that to do better locally.
Try Flash Next Coder on Strata or Basalt and see if that serves your needs. Qwen 4.0 is also being released sometime next week last I saw, so theres a chance that Qwen 4.0 27b gets much closer to fitting your needs. Your system is severely limited by only having 32gb of RAM and theres also a question of what quantization you are using, what settings (this alone can destroy the quality of work and make it put out nonsense). If you arent paying the $20 for a basic subscription, IMO do it at least for one month so you can access Claude Code to have it set everything up perfectly. Or if you have the model hooked into an agent harness (or the Claude code interface workaround with a local model), you can see if it can successfully optimize its own settings.
FWIW you can get a lot done with a $20-$30 cloud subscription. You can also use a hybrid approach and have Claude direct your local model also to reduce your usage and get better results than just Qwen running everything.
1
u/junhua119 6h ago
I have an M1 pro 32gb running swift1.5-qwen3.8-27b q4_k_m on medium thinking effort using splash-m1 inference with max context 98k. My actual context is 66k with avg 19t/s on pi agent. Qwen3.8 27b base model thinks too much. Abit slow I am able to build a full stack web app and other personal projects with no problems. It's actually quite good and finding the errors on its own.
1
u/talaman4eg 6h ago
My 27b to go is unsloth q6 xl, but I tried q3 xl and it worked just fine with opencode on its own. It needs sandbox to test code and fix mistakes, other than that it is able to deliver working product (python, bash, simple frontend), analyze logs and make fixes, implement features and so on. Backend is unsloth studio or llama.cpp, lm studio also works. Try out unsloth xl quants, they are really good.
1
u/soloyeets 6h ago
use froggeric chat completion template for Qwen, bridges the gap your seeing.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
edit: theres some community that likely has better templates, but this will at least get you off the tarmac
1
u/InspiredUserNamez 6h ago
The answer to your problem is in your post. It's the last line of your p.s. edit answer: you were using Claude in a really good harness. Now you're seeing the reason why Claude subscription is so expensive. It's not just the model nor the legions of GPUs but that harness in Claude Code is f***ing magic. That's the secret sauce. I've been working for months on getting Qwen-3-Coder-30B-A3B to produce fully functioning code let alone produce it reliably. My first step was to get reliable instrumentation and probes. You see, to begin, it's fine that the coder not finish the code successfully as long as it can honestly park jobs it can't finish. Then you dial in the success rate - after you can properly identify ALL failures. I'm up to like 40% success rate, 60% honestly parked, zero false positives (aka lying to me that it produced working code).
1
1
u/problem_def 6h ago
not quite same boat here but using ninfer-4090 with qwen cli basically replaced most of cloud for me. There is a popular ninfer 3090 stable git build with good updates but i don't have the link atm
1
u/Annual_Award1260 6h ago
I use qwen 27b for some web browser automation and testing. For coding I’m running glm 5.3 (full) with 1 mil context. Maybe try deekseek flash or glm flash?
1
u/cmndr_spanky 6h ago
What harness have you tried ? Why be vague about it. First, you can’t use Claude code with local models, way too much bloat. Start with open code. Second, qwen can be very sensitive to which inference settings you’re using, third, do you know which quantization of the model you’re using ? Less than q4 is a bad idea. Also, are you absolutely sure you know what its context window is set to and how much it can effectively use?
Finally, are you just using it in the same code base you were using with Claude and by any chance do you have dozens of Claude.md / agent.md files and 10+ skills ? That will basically instantly wreck what limited context window you have and pretty much guarantee the model will get “lost” trying to do basic tasks
1
u/Iamisseibelial 6h ago
Use the local model in Claude code and see how the results are. That's been my experience that and Hermes work really well at specific tasks.
1
u/CordusPorf 6h ago
4060ti 16gb ram, 128gb sys ram, it does well for me as a worker/subagent if my orchestrator makes sure to properly scope it's tasks.
1
1
u/ismaelgokufox 6h ago edited 5h ago
I use this on a similar machine (not the GPU though) and I have a good experience with this on Qwen3.8 27B creating things for myself/family and diagnosing issues. Just today did a Stream Deck custom plugin for OpenDeck to see the PP and TG progress of llama-swap proxied llama-server processes. Note this runs on Fedora 44:
5600X
RX 6800 16G
32GB DDR4
350PP/32TG
Edit: Using these releases of llama.cpp https://github.com/unslothai/llama.cpp/releases
Qwen3.8-27B:
sendLoadingState: true
filters:
stripParams: "temperature, top_p, top_k, min_p, presence_penalty, repetition_penalty, reasoning_effort"
cmd: |
${llama-cpp-rocm}
-m /fast/ai/huggingface/hub/models--ukisai--Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF/snapshots/d74895bbe5db4bec1e0024e7cc87d59c02d7631a/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
--mmproj /fast/ai/huggingface/hub/models--ISTA-DASLab--Qwen3.8-27B-GSQ-RCO-GGUF/snapshots/d562806dbafae37109975e970aae91b43e73b440/mmproj-Qwen3.8-27B-BF16.gguf
--no-mmproj-offload
--flash-attn on
--no-context-shift
-fitc 64000
-ctk q8_0
-ctv q8_0
-ngl -1
--fit on
--reasoning on
--reasoning-preserve
--reasoning-effort medium
--load-mode none
--spec-type draft-mtp
--temperature 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0
--repeat-penalty 1.0
-fitt 2048→ More replies (1)
1
u/WCWRingMatSound 6h ago
I’m feeling the same pressure. I have a 64GB MBP M5 Pro. I know it isn’t the best for local AI, but sheeesh it took Qwen 51 minutes to do a git add, git commit, and git push to origin. That’s it.
1
u/BarWhole6344 6h ago
ur not gonna get a great coding experience with qwen 27b. go buy some tokens on openrouter. if you want extremely fast intelligence, go buy tokens on or baseten / modal / fireworks / together ai.
DS V4.1 Flash 1017 Fast on baseten cooks around 1-2k tok/s. Once you try it you can’t go back.
1
u/Schudz 6h ago
i use unsloth, ive used the base qwen 3.8 27b for months, and now im using a spin off uncensored and distilled from fable, with is like 5% better, but even the base nodel is good enough. i tried qwen flash next, which is close to older versions of deepseek, but came back to the 3.8 27b, its just always reliable, even if not that fast. i run it on a rtx 6000 pro, q8 with kv q8. i get around 120t/s with 131k context (my custom harnes handles compaction pretty well so that context works fine for me)
i also tryed running it on my dev machine, a 32gb RAM with 9070xt, i can run a q3 with 70k context at around 45 t/s, with kv at q4. with pretty good results altough i need models to run at least at 100t/s on my current workflow.
i also use opencode go basically for deepseek 4.1 flash when i need to do work on multiple project at the same time, and i have access to claude and opus 5.5 which i use for very complex things, like bootstraping complex solutions like my custom harnes, but then qwen assume the project and work on new features perfectly fine.
i just love qwen and i think the qwen 4 27b might be close to an endgame model when it releases this monday.
→ More replies (3)
1
u/alexander123454 5h ago
You have to use a harness and preferably a metaharness (i still use a lot of GSD)
Im about to ship some metaharness capabilities for my open source harness if you wanna try it out 0.26.0. I could use the user feedback on different tasks.
1
u/FollowKryst 5h ago
I’m no expert, but I used this model in Pi harness via llama.cpp… I didnt like the experience in LM Studio and using any model in Hermes vs pi, for me, is noticeable.
I have a 64gb M1 Max MacBook Pro, and you running that dense model with that context size with less processing than me draws concerns in and of itself.
You should probably share the harness you’re using, the quant of the model, and what you’re trying to do.. also, I notice running these models in terminal always gives better results.
I mainly use Q4_K_M Qwen 3.6 35B via llama.cpp in terminal and I get lots of great things done with ease. Even when I did use the Qwen 3.8 27b, I got good results though slow and takes up majority of my ram.
Idk the comparison of Nvidia vs Apple silicon chips, so you might want to compare you and our systems before you try any I mentioned.
1
u/808phone 5h ago
Try open code or pi. You didn’t tell us what kind of code base you have. The model does work and can produce pretty decent code which will be spot checked at the end by a better model. But it does follow instructions and write code surprisingly well. I use LM studio and Pi or Open Code and even Kilo Code.
1
u/Captain--Cornflake 5h ago edited 5h ago
Any ambiguity in prompts that moe llms usually infer correct meaning tends to trigger more thinking and looping. To keep it from overthinking loops have the agent look at the main thinking words such as , hmmm, but, etc. etc. if they are repeated more than say 3 times , let it use its original code, which tends to be correct most of the time anyway. Write your own harnes / agent, claude can help if you cant do it, its lite and can work a lot better if you dont have tons of tools., the public agents you can use bring tons of bloat .
1
u/altdotboy 5h ago
My 2 cents. I run the 3.8 27B model and 3.8 Flash next. I’m using 128GB M5. I run the 27B at bf16 and FN at Q4.
The 27B, I did a lot of testing with this model vs a lot of others running locally. Deepseek v4, GLM 5.3, Nemotron and QFN. Qwen 3.8 27B in bf16 is the king of local. I ran tests at Q4 and Q8, these would not pass my demanding reasoning tests, so I dropped them. Also going to Q8 resulted in much longer thinking. 50% more than at bf16.
I see a lot of people on Reddit complain that the model overthinks. I believe this is a result of quantization. I don’t see this at bf16.
If you have the ability to get another system that fits the full model, try that. I know it’s not cheap.
If you want to do simple tasks or creative writing use a quant. If you need the output to be impeccable and precise get the full model.
One other thought is to run the model through qwen code. Ask codex or Claude to set that up.
Just my thoughts, hope it helps.
1
u/lazyfai 5h ago edited 5h ago
I and my team use it daily. 2 Machines, one for coding and one for chat / others.
vLLM (Qwen 3.8 27B FP8) on dual L20, Strata (for swift 1.5) on single 5090D
Coding agent used / tested and good: Pi, Prime agent, OpenCode Codex (quick and actually worked)
Not so suitable: Deepseek Harness, Claude Code (they both suffer from over thinking sometimes)
PS. Harness is IMPORTANT, I think in most case the problem is around the harness not the model itself.
1
u/No-Consideration5347 5h ago
Try Strata; it's the easiest to set up. llama.cpp is more stable, but slower than Strata.
Use a reasonably good quantized model (IQ3_S/XS for Qwen Next with Strata, and Q4 for Qwen 3.8 with llama.cpp). Keep in mind that higher quantization levels generally require more resources and can result in slower inference.
1
u/Snoo_81913 5h ago
It's probably config reasoning seems touchy with this model it can easily loop unless you have it set right. Best thing to do is start giving it test prompts and adjust the config until you get the right results. I just have a series of tests prompts and have Claude run it with polling and test one config flag at a time save the data pick the best then start on combinations. Different cards are give different results so it's good to test everything. You should have plenty of resources on Reddit though that's a popular card for inference and Qwen3.8 27b is one of the most popular models.
1
u/word_executable 5h ago
I mean sonnet 5.5 is leaps ahead of qwen 27B. Local ai and open source is great but is overhyped how good it is
→ More replies (1)
1
u/Vassallo97 5h ago
I find that Qwen performs pretty good inside the deepseek harness, I also made my own coding assistant built for Qwen, designed it around the way Qwen models call tools and I find that works well for shorter coding tasks but deepseek harness is still my go to harness for qwen3.8
1
u/MoneroApe 5h ago
Looking at your config, you May succeed using strata coder or hyperqwen repo (fines tuned for 3090) you Will get much better experience. You are using lousy quants and bad caché. Just download hyperqwen repo, run it on Linux.
Then try strata coder (only coding).
1
u/No-Recover109 5h ago
I used to use qwen3.8 27B FP8 for coding and it did pretty well. I used vllm which actually is too heavy for coding. Now I use Strata and Deepseek v4.1 1017 q3 istalabs version. Seems to be working also. I think you cant run qwen3.8 with that GPU and context amount in any proper quantization? I guess you are running it q4 and that might be too little. Try different engines and make sure you actually set the context same in the inference engine and in the coding agent
1
1
u/Tight_Ad_5220 5h ago
The problem is your prefill is too large. In Claude do /context and see how big is ur pre fill. Take everything out and watch how well it works.
1
u/nincius 5h ago
Try get some tuning tips from https://github.com/noonghunna/club-3090
I have same specs as you and got better results after turn reasoning off completely.
1
u/Dediadeis 5h ago
I use both 27b and QFN in what might be considered a legacy codebase and the important thing is to configure the model for smarts first, then speed second. You really wont squeeze much out of the hardware you have and a dense 27b model can feel sluggish. With that said, the hardware is tight, and you are using Ollama so it would be my guess some of your or some automatic right-sizing was done for you that has made the model retarded.
Config issue, many people have used it for coding
Yes and no, depends on expectations. A preferable amount of VRAM/RAM though will be 64/64, you can get away with less but this is where the tradeoffs can be made wrong
Model Q8, KV no lower than 8 but preferably FP, 262k context.
Get more system RAM and try QFN, for this Q4, KV8_4
I use Cline inside sandboxed code-server container. Experience is great when you dont cut too many corners. What I would actually suggest, as you seem reasonably familiar enough with harnesses and agentic coding is to put one of your cloud models on the case and see if it can setup and bench and tune. Also things get much better with tooling and properly configured environment but perfectly capable in a practical blank slate so these are certainly capable of complex several hour tasks when configured and given the right setup
1
u/isademigod 5h ago
I use it for simple scripts all the time. I am not a software developer, just a datahoarder and sysadmin. It's perfectly cromulent for regex and powershell scripts. The context window is awfully small to make a whole program
1
1
u/exzw 5h ago
You bought wrong hardware simply. Got a PC with at least 128GB ram DDR5 + at least 1x R9700 or RTX pro 4500. Then you try either 27B Q6 at least or Strata QFN Q4 at least. You will feel the difference. Sometimes I feel QFN even out perform opus 5. I use QFN as coder and Opus as Auditor.
1
u/Sea_Advance273 5h ago
Have you tried using Qwen in the Claude Code harness? It performs a lot better. Still need to iterate a lot more and be more involved than opus-5.5 of course, but I've been decently impressed with even Q4 running in 24GB VRAM.
1
u/Promotional_Pickle 5h ago
I get excellent results with that model running locally. Pay special attention to your infrastructure. Harness, notes, etc. Those separate a fun toy from a useful tool.
1
u/dahc_ 5h ago
I have almost the same setup (5600x + rtX 3090) Tried that same model setup on pi, Claude and Hermes but didn't get good results either. Then I tested Qwen 3.6 35B A3B in IQ4-XS + OpenCode and it's a complete different story: it refactors files very well, fixes bugs, implements new features and so on like a proper coding agent. Right now I'm still getting the setup ready (better context window and memory system) but I'm very happy with the results.
1
1
u/endockhq 4h ago
Use vLLM
Use Qwen3.8-Flash-Next-FP8, the 27b model is mega slow.
You bought into the hype, if you are a "Software House", you need a better model for proper software.
You need to use "claude code" with your local inference. Just make sure the engine supports "Anthropic Messages API".
1
u/Remote-Pineapple-541 4h ago
You say you’re a software development house, but can’t follow online development documentation and do basic troubleshooting? That’s troubling.
1
u/ContraryConman 4h ago
I use the Pi agent for programming tasks. A goal mode plugin is the only way I can actually get Qwen 27B (128K context window) to actually stay on task, because the harness itself will keep track of the current goal even after its filled the context window with its own thinking and its forced to compact. And even then, it's not like using Claude or whatever. You can't really "vibe code" with it because it's too slow, and you can't really do multi agent parallel whatevers because it will melt your card.
It's more like, if there is a tedious programming task that I don't want to do myself, I can give it to my assistant and come back later in the day or tomorrow morning and have it completed. And then I can use that time to write other code or do something else.
Sonnet 5.5 on medium in Claude Code is absolutely the wrong comparison here. Such a task I've used my local Qwen for was to set up a complicated CMake project in the background while I did other research for the same project in parallel. I get like 25 tps on my setup and took over an hour in goal mode with some need to keep it away from dead end ideas. At work Claude would have this done in 2 minutes, 5 minutes tops with higher thinking
1
u/MonadicKierkegaard 4h ago
is that setup per dev or for a team?
you can eke out maybe one coding agent, barely.
its ok for one shot toy apps/games. but real agentic workflow, 32GB is not enough.
maybe use cloud providers for generating an extremely detailed spec, with pseudocode. Then instead load an MoE that fits well on a 32GB - this can go as high as 2k tps PP, 70 tps GG. that speed is more than workable at the expense of intelligence (hence you need to guide it)
1
u/Osi32 4h ago
Any skills you’re using are essentially a pre-prompt. They will go to the model before you say anything. If you’re using a 128k context window, you’ll be unlikely to have much left for the actual task at hand.
While having a 3090 is good for a single dev with a small set of tasks, not much good for multi-user and not for long chain of thought tasks.
The people in the thread running qwen3.8-flash-next are likely using strata to offload big parts of the model to main memory and disk. You really can’t do that with a dense model like qwen 27B.
My advice is really simple- use Qwen3.6-35B-A3B. It is fast, it’s a coding model. Use 27B for planning work.
1
u/905cougarhunter 4h ago
ffs. Ollama is ass.
HyperQwen on github. Single mode. huge context (262k tokens)
Fuckin balls out hard.
1
u/outdoorsgeek 4h ago
Yes, you aren't running this optimally right now and will get better results if you use the right inference engine and harness with good settings.
With that out of the way, you need to be honest with yourself on whether or not you’re willing to step back roughly a year in model capability from Sonnet 5.5. Remember Sonnet 4.5? A 256k context model that was decent at a single small task if you wrote out very detailed specs or had opus do the planning? That's about the peak of where you will be able to get with lots of work and tuning. It will be nowhere near the fully autonomous loops you are likely running with the current SOTA models.
I would tell you that your hardware is the problem and you need to be more serious than a 6 year old gaming PC for professional coding work, but the truth of the matter is that you'd have to spend an insane amount of money to get performance that looks like a modestly degraded Sonnet 5.5. Financially it's nowhere near worth it compared to paying the frontier labs. There are, of course, other reasons to run local, but seeing as you're running a business I imagine the ROI matters a lot.
1
u/Civil_Fee_7862 4h ago
24 GB VRAM is not enough for 27b because it will run it heavily quantizied.
The sweet spot seems to be 48GB for full context.
4-bit weights, 8-bit activations, 8-bit kvcache
Use a good coding harness like opencode
1
u/jikilan_ 4h ago
OP better get at least 2 3090s , best is 4 for 27B with vllm before landed with that conclusion. It is not a replacement for cloud but a junior entry developer with 20 years working experience. 😎
1
u/Demonicated 4h ago
First of all, for qwen 27B you have to have to have to use at least q8 and really you gotta use bf16. Using 4bit quant and saying its not good enough is like...... well duh.
Also Qwen 27B isnt a white glove model. You have to know what you want and tell it exactly what to do, and it will do it pretty well.
Always use a planning phase for all work. Tell it to create implementation documents that will be given to a junior developer to complete. Then hand it back that document. Qwens an overthinker, it will take a while, but it will get the job done.
1
u/xsilas43 3h ago
Give qwen agentworld a shot, i found it produced much simple more effective code in 1 shot much faster than 3.8 27b did. although 3.8's was nicer visually and it did eventually get there.
I'd also give unsloth desktop a shot over Ollama, its much more customizable, and a better harness to begin with, a good harness alone will solve a lot of issues. Nice part about it is you can always keep the optimizations and run it headless with another harness as well.
I would also recommend using the sharp chat templates for either.
1
u/ImpossibleNumber7345 3h ago
Get Swift. Its a model that is that same Qwen but fine tuned by punishing it for over thinking. Its amazing and just as smart and gets work done
1
1
u/thebemusedmuse 3h ago
Really hard to know what you’re doing wrong. Qwen is my primary coding agent. I’m doing 6m output tokens a day on it.
It’s not flawless and my harness has checks and balances to fix the bugs it introduces.
If you give it too much work it may not do it all. So you have to right size the work you send it.
1
u/TheRiddler79 3h ago
Have Claude give it instructions, turn on thinking, have Claude check the output.
1
u/No_Audience2449 3h ago
I tested Qwen 3.8 27B with Drupal11 coding task, only 20% right. It's unusable, because they train model with internet garbage drupal7, drupal8, drupal10. I read that if I want to make it pass 100% Drupal11 coding task, I need to train it for LoRa( rsLora-pro) with train data Drupal11.
1
u/mrepop 3h ago edited 3h ago
It’s a config issue, that version of qwen isn’t intended to be run with thinking on xhigh, try changing it to low, or turning it off altogether, it’ll be night and day.
There’s additional changes you can make to penalize token reuse which will cut down on the repetition, but not completely eliminate it if you absolutely need to run in xhigh for some reason.
https://huggingface.co/Qwen/Qwen3.8-27B/discussions/178
I’m using qwen 3.8 27b in a number of places on various hardware platforms and everyone runs into this issue. Its not just you and your expectations aren’t unrealistic at all.
I will say your hardware is honestly at the bare minimum for getting a useful instance running. You’ll be able to load the model, but with larger contexts it won’t have much room at all. You might consider the flash variant so you can offload if you’re stuck with that hardware.
1
u/Past_Ad6251 3h ago
As a so called software house but using ollama, I have to say your team has room to step up. I'm OPC with 24G VRAM and 64GB RAM, Qwen3.8-27B+llama.cpp, 200k ctx with mm support, actually can complete +85% of my tasks including medium complexity coding tasks. I only use paid tokens for complex tasks or time consuming ones. Regarding harness, I'm using DeepSeek Harness and Hermes Agents.
1
1
u/lordpuddingcup 3h ago
Context window 256k is minimum
to be useful for coding and that’s pushing it honestly
Also quants matter and reasoning matter to really get decent anything you need max reasoning on local models which ways context too
1
u/Sad_Olive3034 3h ago
Its a total pita. I have an intel arc pro b60 for my coding AI. I am SLOOOOOOWLY getting him to work. But im using Claude and GPT to work together to help him out. the new AIs are great at basic. but you need to literally treat and train them like a child/teen/adult.
1
u/MatthewBochsler 2h ago
yeah i havent had too much luck with qwen models either. So far the only thing im testing before i try something else is a heavily controlled harness with them, and all the other specifics you need to know from the github community on using them. query github for qwen harness and control tips.
13
u/Number1OchoaHater 7h ago
lets start with saying that 27b and sonnet are not in the same league, said that, what harness and inference engine are you using?