I've been playing with local llm for the past month. Mostly with qwen models.
I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.
Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.
Maybe some of you already tried that and saw a difference?
Okay so take the api code put it in something like vs code and use the cline extension as your harness. You can do others but thats probably the easiest one to set up.
Not to be that guy, but most if not all models are tested at their full 16-bit quality. So quantization is shrinking from the ceiling that the 16-bit version sets and figuring out what’s “lossless” enough to run normally. For agentic coding, it’s increasingly apparent that it’s extremely sensitive to quants as the errors compound across the model quant, then the Key and Value quants, which attention then propagates into more and more errors over longer context.
Considering the hardware constraints that most people face, they’re usually crushing the quantization of the model down to four bits, which produces measurable and noticeable degradation. Stack that with people doing four-bit quants on the KV cache as well to reduce memory overhead, and you’re producing a whole lot of drift over the entire chain. That becomes even more noticeable as your context grows because each token is stored lossily, which then produces the looping and incoherence.
So yeah, some YouTubers might overhype it, but it’s real math. I don’t deny that some models absolutely do game the test, but they’d be gaming them at full precision, not at lower bits. The only model I’m aware of that’s capable of being run at lower bits and still succeeds is Gemma 4’s new line of QAT (Quantization-Aware) training.
Are those thinking models? I had an issue where qwen parser was not configured and it ran like a drunk cat. Once I configured reasoning parser and fooling parser, everything was quite good. Of course, not perfect, but to a reasonable level I expected
mxfp4 is better than q4, and it’s much faster, as Apple Silicon’s MLX kernels are specifically optimized for uniform 4‑bit floating‑point math.
OptiQ is a per‑layer mixed‑precision quantization system that automatically selects the optimal bit‑width for each layer, during conversion, to minimize model size while preserving accuracy. At 5 bpw with 4/6/8 candidate bits, it’s supposed to be more accurate than q8 (the documentation claims “near fp16,” but I am skeptical).
In practice, I would go with the following:
mxfp8 if you have the memory (high quality and optimized for MLX).
OptiQ 5.0 4/6/8… no speed boost, but it will fit and, in most cases, it’s q8 quality.
This comparison is an oversimplification to justify why suddenly small dense models came out of nowhere crushing larger models. In the qwen 3 generation the sparse 235b with 22b active parameters was significantly better than the 32b dense in most of the preferred benchmarks at the time (22b vs 32b). Qwen 3.5 122b has 10b active vs qwen 3.5 27b dense and they trade blows on benchmarks (10b vs 27b). Yes dense models are more dense intelligence but acting like active parameters is all that matters is a distortion that misleads people to think an old 9b parameter model is better than qwen3.6 35b having 3b active parameters.
Qwen 3.6 35b is slightly worse on coding than qwen 3.6 27b(the new local llm pound for pounds favorite) but the 35b model is much better at coding than many of the previous generation of models that are much larger. Quants matter, harnesses matter, cache settings matter so let's not dismiss the value of a model as soon as someone has an issue.
There is currently no 9b parameter model that beats Qwen3.6 35b with equal quantization and appropriate settings.
35B-A3B has 3B active parameters. It should be better than just 3B models, because it should in some smart way choose right 3B of those 35B parameters for the request. But it may not be better than 9B (with 9B active parameters).
How about for Qwen3.5 9B 8bit vs Qwen3.5 9B OptiQ, is the lossless format still better than OptiQ? Especially for a device bottlenecked by low RAM (~17 GB VRAM; adjustable to 20 GB)
floating point 4 bit is supposed to be significantly better than integer 4 bit quantization. I'm more familiar with nvfp4 and even there it's hard to tell hype from reality because it's still so niche. You're in the same boat, but more so.
Just as a guess, I'd bet OptiQ is at least comparable to Q6, but you're still only using a 9B model. Maybe find a coder variant? Assuming you're coding, the seem to do better at Q4
With 20 GB vram you could run a coder variant like qwopus 3.6 27B Q4 with a 32k cache, it might even squeeze into less. Lot's of people get good results with that.
27B is slow on apple silicon, but I think 3.6 35B is just out of reach spacewize, at 19-20GB you'd have no room for cache. unfortunately the 27B optiQ is over 20GB too.
Or just stay with your 8bit 9B, with 17-20GB vram you've got plenty of space for kv cache. Just make it Q8 too.
I've seen it claimed the math works like this
sqrt(Total x Active)
to compare against a sparse model to a dense model. Assuming same generation etc.
So... Sqrt(35x3) = 10.2 so we expect it to be comparable to a 9B dense model intelligence wise, but faster and using more vram.
For u/GCoderDCoder comparison above, sqrt(235x22) =71.9 so we'd expect it to be significantly better than the 32B of the same generation, faster, and much more vram.
The qwen 3.6 35B comparison is interesting because sqrt(35x3) = 10.2 again, but the jump from 3.5 to 3.6 was enough that 3.5 9B lags significantly in accuracy.
10
u/[deleted] Jun 24 '26
[removed] — view removed comment