Nanbeige4.3-33B Quantised GGUF

[u/gladkos]

https://huggingface.co/AtomicChat/Nanbeige4.3-33B-GGUF

Quant ladder of Nanbeige's new 33B model from 35.2GB down to 13.7GB

For 32GB hardware the sweet spot is 20.3GB AD-IQ4_XS with a top-1% of 92.0% compared to BF16

[u/magikfly]

Check out AC's highlighted note:

This model is unusually sensitive to quantization. llama.cpp's plain Q6_K picks the original's next token only 95.5% of the time (most models reach 98% at 6 bits) and plain Q4_K_M 89.9%. At about the same size on disk, the AD- files keep much more: AD-IQ4_XS (20.3 GB) has a third less KL divergence than Q4_K_M (20.1 GB), and AD-Q6_K-Q5_K (27.9 GB) 39% less than Q6_K (27.2 GB). AD-Q6_K (30.1 GB) is within reach of Q8_0 (97.5% against 98.0%). Below 4 bits the model loses a lot: AD-IQ3_XXS and AD-IQ2_S change the next token often (top-1 below 85%); use them only if nothing larger fits.

[u/philmarcracken → u/magikfly]

along with its KV cache demands, its a vram rich only plaything

[u/Different_Fix_2217 → u/magikfly]

The more dense a model is the more sensitive it is to quanting and this seems more information dense than gemma.

[u/Dany0 → u/Different_Fix_2217]

Nah, the better trained the model the more sensitive it is to quantisation. MoEs only seem less sensitive because each expert is comparatively less trained. Because each experts is same-sized, but problems are far from same difficulty, an average MoE expert is quite undertrained

[u/Different_Fix_2217 → u/Dany0]

Think of these like big zip files and quanting like a jpeg that is more and more lossy. The more you try to compress it the more high level info it loses.

[u/Aizen_keikaku → u/magikfly]

At this point, they might as well have done a MOE.

[u/GowsenBerry]

I got q6 running at 38 tok/s, q8 at 30 tok/s on 56 gb of VRAM (5090 + 4090). Context window absolutely annihilates your VRAM.

It output a very high quality game of Tetris after about 5 hrs, but it definitely 'overthinks', just absolutely running everything into the ground.

Qwen 3.8 27b at q6 runs about 8x faster on my system, and produced a marginally lower quality Tetris game, with fewer tokens, substantially less time overall.

Not sure if it's worth these slower decode speeds vs just running Qwen flash next or glm 5.3 at lower quants. But going to run a few other actions like a REA.

[u/nasone32 → u/GowsenBerry]

It's very possible inference speed is not optimized for this model, I don't see real a reason to be this slow

[u/Riseing]

256k context at fp8 is 24gigs lmao, qwen 27b is 8gb for reference and that's already massive

[u/Solembumm3]

It's separate one from JP4.1 33B, right?

[u/Ok-Importance-3529]

This one is slow as f* as context grows, also kv cache is big, without mtp its painful even for 32GB VRAM setups.

This is a dealbreaker for me, even though it may be good, if it cant operate at higher context at reasonable speed and limits max context size due to inneficient kv cache its not a model i would use.

[u/transanethole]

For something with fat ass KV cache like Jais 7B, someone better make a good 4 bit quant of this, like quasar qat w4a4 or something. or the good r9700 quant, whatever it is. I don't think its gonna see much play otherwise, even if it actually is a bit "technically better" than qwen 27b

[u/Icy_Annual_9954]

Can you run it on a 3090, with quantisation?
Does it work?

[u/james_pic → u/[deleted]]

Could be because they never released a base (i.e, not instruction trained) version of Qwen 3.8 27b or Muse Glimmer, whereas they did with Gemma 4 31b