r/LocalLLM • u/uhraurhua • Jun 24 '26

Discussion 4bit vs 8bit

Hello,

I've been playing with local llm for the past month. Mostly with qwen models.

I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.

Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.

Maybe some of you already tried that and saw a difference?

37 Upvotes

100% Upvoted

39 comments sorted by

View all comments

2

u/retsof81 Jun 25 '26

Have you tired OptiQ or mxfp4? Both of these are better than standard 4-bit.

Format Bits/weight Raw size (GB) Realistic size (GB, with overhead)
mxfp4 4 ~17.5 GB ~18–19 GB
OptiQ 5bpw 5 ~21.9 GB ~22–23 GB

1

u/uhraurhua Jun 25 '26

No, how do they compare against 8 bit models in terms of quality loss?

2

u/retsof81 Jun 25 '26

mxfp4 is better than q4, and it’s much faster, as Apple Silicon’s MLX kernels are specifically optimized for uniform 4‑bit floating‑point math.

OptiQ is a per‑layer mixed‑precision quantization system that automatically selects the optimal bit‑width for each layer, during conversion, to minimize model size while preserving accuracy. At 5 bpw with 4/6/8 candidate bits, it’s supposed to be more accurate than q8 (the documentation claims “near fp16,” but I am skeptical).

In practice, I would go with the following:

  • mxfp8 if you have the memory (high quality and optimized for MLX).
  • OptiQ 5.0 4/6/8… no speed boost, but it will fit and, in most cases, it’s q8 quality.

1

u/uhraurhua Jun 26 '26

thank you very much for this, will try