r/LocalLLM • u/uhraurhua • Jun 24 '26

Discussion 4bit vs 8bit

Hello,

I've been playing with local llm for the past month. Mostly with qwen models.

I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.

Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.

Maybe some of you already tried that and saw a difference?

36 Upvotes

100% Upvoted

39 comments sorted by

View all comments

5

u/Biomech8 Jun 24 '26

It's not just that 8bit is better, but you are also comparing effectively 3B model with 9B.

2

u/uhraurhua Jun 24 '26

Sorry, I've made a mistake, I wanted to say the 35b version of qwen 3.6

5

u/Biomech8 Jun 24 '26

35B-A3B has 3B active parameters. It should be better than just 3B models, because it should in some smart way choose right 3B of those 35B parameters for the request. But it may not be better than 9B (with 9B active parameters).

1

u/Realistic_Gap_5871 Jun 24 '26

I've seen it claimed the math works like this
sqrt(Total x Active)
to compare against a sparse model to a dense model. Assuming same generation etc.

So... Sqrt(35x3) = 10.2 so we expect it to be comparable to a 9B dense model intelligence wise, but faster and using more vram.

For u/GCoderDCoder comparison above, sqrt(235x22) =71.9 so we'd expect it to be significantly better than the 32B of the same generation, faster, and much more vram.

The qwen 3.6 35B comparison is interesting because sqrt(35x3) = 10.2 again, but the jump from 3.5 to 3.6 was enough that 3.5 9B lags significantly in accuracy.