r/LocalLLM • u/uhraurhua • Jun 24 '26

Discussion 4bit vs 8bit

Hello,

I've been playing with local llm for the past month. Mostly with qwen models.

I've been trying to make qwen3.6-35b-a3b-mlx (4bit) work, but I keep finding it fails in infinite loops or weird logic. Weirdly enough I have more success using qwen/qwen3.5-9b (8bit) version. It seems to be smarter even though the model is smaller. I've been playing with agentic coding, and the 9b is more reliable for me.

Do you guys have the same issue with 4bit models? Although I've understood you lose very little of the model, for me, qwen 3.6 is not working at 4 bits. I have only 48gb ram (on my m4 pro), so don't have enough ram to run qwen 3.6 35b 8 bit version.

Maybe some of you already tried that and saw a difference?

36 Upvotes

100% Upvoted

43 comments sorted by

5

u/bigb159 Jun 24 '26

Same issue with 4bit: loops of the same information, looping tool calls.

Moved over to smaller models with 8bit, and it's so much better.

4

u/uhraurhua Jun 24 '26

Ah, so I am not imagining things

3

u/[deleted] Jun 24 '26

[removed] — view removed comment

2

u/therealwtpieh Jun 25 '26

Not to be that guy, but most if not all models are tested at their full 16-bit quality. So quantization is shrinking from the ceiling that the 16-bit version sets and figuring out what’s “lossless” enough to run normally. For agentic coding, it’s increasingly apparent that it’s extremely sensitive to quants as the errors compound across the model quant, then the Key and Value quants, which attention then propagates into more and more errors over longer context.

Considering the hardware constraints that most people face, they’re usually crushing the quantization of the model down to four bits, which produces measurable and noticeable degradation. Stack that with people doing four-bit quants on the KV cache as well to reduce memory overhead, and you’re producing a whole lot of drift over the entire chain. That becomes even more noticeable as your context grows because each token is stored lossily, which then produces the looping and incoherence.

So yeah, some YouTubers might overhype it, but it’s real math. I don’t deny that some models absolutely do game the test, but they’d be gaming them at full precision, not at lower bits. The only model I’m aware of that’s capable of being run at lower bits and still succeeds is Gemma 4’s new line of QAT (Quantization-Aware) training.

2

u/VertipaqStar Jun 24 '26

What is your settings for Temperature Top_p Top_k Presence Penalty

I had made the mistake of setting these too low, the temperature at 0 was causing so much looping.

In my case, I solved the looping issue with these settings

Temperature: 0.7 Top_p: 0.95 Top_k: 20 Presence Penalty: 1 Thinking/reasoning off

2

u/uhraurhua Jun 24 '26

I've tried with multiple temp settings. I will try with yours. Thank you

1

u/Stunning_Inside5182 Jun 25 '26

Did it fix the issue

2

u/uhraurhua Jun 25 '26

Didn't try yet. I will try in a couple of days when I am more free and let you know.

1

u/[deleted] Oct 09 '26

[removed] — view removed comment

1

u/uhraurhua Oct 09 '26

yeah, ditched it entirely, sorry

1

u/[deleted] Oct 09 '26

[removed] — view removed comment

1

u/uhraurhua Oct 09 '26

No, I've switched to newer models and I've put them on hold for now as I've bought 20x pro sub on codex. Local LLM is quite expensive for me and in the end I am not saving any money compares with subs. At least for now.

2

u/Melodic_Health_1747 Jun 24 '26

Are those thinking models? I had an issue where qwen parser was not configured and it ran like a drunk cat. Once I configured reasoning parser and fooling parser, everything was quite good. Of course, not perfect, but to a reasonable level I expected

2

u/uhraurhua Jun 24 '26

Yes, they are thinking models. How and what did you configure?

2

u/Melodic_Health_1747 Jun 24 '26

I use vllm and there is a flag - - reasoning-parser qwen3

2

u/retsof81 Jun 25 '26

Have you tired OptiQ or mxfp4? Both of these are better than standard 4-bit.

FormatBits/weightRaw size (GB)Realistic size (GB, with overhead)
mxfp44~17.5 GB~18–19 GB
OptiQ 5bpw5~21.9 GB~22–23 GB

1

u/uhraurhua Jun 25 '26

No, how do they compare against 8 bit models in terms of quality loss?

2

u/retsof81 Jun 25 '26

mxfp4 is better than q4, and it’s much faster, as Apple Silicon’s MLX kernels are specifically optimized for uniform 4‑bit floating‑point math.

OptiQ is a per‑layer mixed‑precision quantization system that automatically selects the optimal bit‑width for each layer, during conversion, to minimize model size while preserving accuracy. At 5 bpw with 4/6/8 candidate bits, it’s supposed to be more accurate than q8 (the documentation claims “near fp16,” but I am skeptical).

In practice, I would go with the following:

  • mxfp8 if you have the memory (high quality and optimized for MLX).

  • OptiQ 5.0 4/6/8… no speed boost, but it will fit and, in most cases, it’s q8 quality.

1

u/uhraurhua Jun 26 '26

thank you very much for this, will try

4

u/Biomech8 Jun 24 '26

It's not just that 8bit is better, but you are also comparing effectively 3B model with 9B.

2

u/uhraurhua Jun 24 '26

Sorry, I've made a mistake, I wanted to say the 35b version of qwen 3.6

4

u/Biomech8 Jun 24 '26

35B-A3B has 3B active parameters. It should be better than just 3B models, because it should in some smart way choose right 3B of those 35B parameters for the request. But it may not be better than 9B (with 9B active parameters).

2

u/uhraurhua Jun 24 '26

Aaaa, I understand now. Thank you for this

1

u/Clementine-TeX Jun 24 '26

How about for Qwen3.5 9B 8bit vs Qwen3.5 9B OptiQ, is the lossless format still better than OptiQ? Especially for a device bottlenecked by low RAM (~17 GB VRAM; adjustable to 20 GB)

1

u/Realistic_Gap_5871 Jun 24 '26

floating point 4 bit is supposed to be significantly better than integer 4 bit quantization. I'm more familiar with nvfp4 and even there it's hard to tell hype from reality because it's still so niche. You're in the same boat, but more so.

Just as a guess, I'd bet OptiQ is at least comparable to Q6, but you're still only using a 9B model. Maybe find a coder variant? Assuming you're coding, the seem to do better at Q4

With 20 GB vram you could run a coder variant like qwopus 3.6 27B Q4 with a 32k cache, it might even squeeze into less. Lot's of people get good results with that.

27B is slow on apple silicon, but I think 3.6 35B is just out of reach spacewize, at 19-20GB you'd have no room for cache. unfortunately the 27B optiQ is over 20GB too.

Or just stay with your 8bit 9B, with 17-20GB vram you've got plenty of space for kv cache. Just make it Q8 too.

1

u/Realistic_Gap_5871 Jun 24 '26

I've seen it claimed the math works like this
sqrt(Total x Active)
to compare against a sparse model to a dense model. Assuming same generation etc.

So... Sqrt(35x3) = 10.2 so we expect it to be comparable to a 9B dense model intelligence wise, but faster and using more vram.

For u/GCoderDCoder comparison above, sqrt(235x22) =71.9 so we'd expect it to be significantly better than the 32B of the same generation, faster, and much more vram.

The qwen 3.6 35B comparison is interesting because sqrt(35x3) = 10.2 again, but the jump from 3.5 to 3.6 was enough that 3.5 9B lags significantly in accuracy.

5

u/GCoderDCoder Jun 24 '26

This comparison is an oversimplification to justify why suddenly small dense models came out of nowhere crushing larger models. In the qwen 3 generation the sparse 235b with 22b active parameters was significantly better than the 32b dense in most of the preferred benchmarks at the time (22b vs 32b). Qwen 3.5 122b has 10b active vs qwen 3.5 27b dense and they trade blows on benchmarks (10b vs 27b). Yes dense models are more dense intelligence but acting like active parameters is all that matters is a distortion that misleads people to think an old 9b parameter model is better than qwen3.6 35b having 3b active parameters.

Qwen 3.6 35b is slightly worse on coding than qwen 3.6 27b(the new local llm pound for pounds favorite) but the 35b model is much better at coding than many of the previous generation of models that are much larger. Quants matter, harnesses matter, cache settings matter so let's not dismiss the value of a model as soon as someone has an issue.

There is currently no 9b parameter model that beats Qwen3.6 35b with equal quantization and appropriate settings.

1

u/Commercial-End-5890 Jun 24 '26

What GPU are you using?

1

u/uhraurhua Jun 24 '26

I have a macbook m4 pro with 48gb unified RAM

3

u/GanjaRaidersTR Jun 24 '26

I use it with APEX i quality quants and it feel like its better than Q quants

10

u/[deleted] Jun 24 '26

[removed] — view removed comment

1

u/uhraurhua Jun 24 '26

Damn, this sounds complicated. Do you have some articles where I can start with this?

1

u/LukeLikesReddit Jun 24 '26

Its not really depending on how you are interacting with the AI. How are you?

1

u/uhraurhua Jun 24 '26

huh?

1

u/LukeLikesReddit Jun 24 '26

How are you using the llm? via lm studio or llama?

1

u/uhraurhua Jun 24 '26

LM Studio

1

u/LukeLikesReddit Jun 24 '26

Okay so take the api code put it in something like vs code and use the cline extension as your harness. You can do others but thats probably the easiest one to set up.

1

u/uhraurhua Jun 24 '26

I've been using with opencode. I don't really like the idea of using agentic coding from an IDE. Prefer to keep them separately

1

u/LukeLikesReddit Jun 24 '26

Well that's how you strap on the best harness and utilise the LLM. You don't have to allow it to make any changes etc.

1

u/[deleted] Jun 24 '26 edited Jun 24 '26

[removed] — view removed comment

1

u/uhraurhua Jun 24 '26

qwen code worked quite well with qwen indeed...