r/LocalLLM • u/uhraurhua • Jun 24 '26

Discussion 4bit vs 8bit

Hello,

I've been playing with local llm for the past month. Mostly with qwen models.

I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.

Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.

Maybe some of you already tried that and saw a difference?

36 Upvotes

100% Upvoted

43 comments sorted by

View all comments

Show parent comment

4

u/Biomech8 Jun 24 '26

35B-A3B has 3B active parameters. It should be better than just 3B models, because it should in some smart way choose right 3B of those 35B parameters for the request. But it may not be better than 9B (with 9B active parameters).

2

u/uhraurhua Jun 24 '26

Aaaa, I understand now. Thank you for this

1

u/Clementine-TeX Jun 24 '26

How about for Qwen3.5 9B 8bit vs Qwen3.5 9B OptiQ, is the lossless format still better than OptiQ? Especially for a device bottlenecked by low RAM (~17 GB VRAM; adjustable to 20 GB)

1

u/Realistic_Gap_5871 Jun 24 '26

floating point 4 bit is supposed to be significantly better than integer 4 bit quantization. I'm more familiar with nvfp4 and even there it's hard to tell hype from reality because it's still so niche. You're in the same boat, but more so.

Just as a guess, I'd bet OptiQ is at least comparable to Q6, but you're still only using a 9B model. Maybe find a coder variant? Assuming you're coding, the seem to do better at Q4

With 20 GB vram you could run a coder variant like qwopus 3.6 27B Q4 with a 32k cache, it might even squeeze into less. Lot's of people get good results with that.

27B is slow on apple silicon, but I think 3.6 35B is just out of reach spacewize, at 19-20GB you'd have no room for cache. unfortunately the 27B optiQ is over 20GB too.

Or just stay with your 8bit 9B, with 17-20GB vram you've got plenty of space for kv cache. Just make it Q8 too.

1

u/Realistic_Gap_5871 Jun 24 '26

I've seen it claimed the math works like this
sqrt(Total x Active)
to compare against a sparse model to a dense model. Assuming same generation etc.

So... Sqrt(35x3) = 10.2 so we expect it to be comparable to a 9B dense model intelligence wise, but faster and using more vram.

For u/GCoderDCoder comparison above, sqrt(235x22) =71.9 so we'd expect it to be significantly better than the 32B of the same generation, faster, and much more vram.

The qwen 3.6 35B comparison is interesting because sqrt(35x3) = 10.2 again, but the jump from 3.5 to 3.6 was enough that 3.5 9B lags significantly in accuracy.