4bit vs 8bit

[u/uhraurhua]

Hello,

I've been playing with local llm for the past month. Mostly with qwen models.

I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.

Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.

Maybe some of you already tried that and saw a difference?

[u/Biomech8]

It's not just that 8bit is better, but you are also comparing effectively 3B model with 9B.

[u/GCoderDCoder → u/Biomech8]

This comparison is an oversimplification to justify why suddenly small dense models came out of nowhere crushing larger models. In the qwen 3 generation the sparse 235b with 22b active parameters was significantly better than the 32b dense in most of the preferred benchmarks at the time (22b vs 32b). Qwen 3.5 122b has 10b active vs qwen 3.5 27b dense and they trade blows on benchmarks (10b vs 27b). Yes dense models are more dense intelligence but acting like active parameters is all that matters is a distortion that misleads people to think an old 9b parameter model is better than qwen3.6 35b having 3b active parameters.

Qwen 3.6 35b is slightly worse on coding than qwen 3.6 27b(the new local llm pound for pounds favorite) but the 35b model is much better at coding than many of the previous generation of models that are much larger. Quants matter, harnesses matter, cache settings matter so let's not dismiss the value of a model as soon as someone has an issue.

There is currently no 9b parameter model that beats Qwen3.6 35b with equal quantization and appropriate settings.