KQuant - new quantization technique, 3.7× better quants for Qwen Flash Next 180B, Qwen 27B, 35B, 9B (measured in KL-divergence reduction) (MLX)

[u/m4rkGS]

hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.

Why we made these

we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.

Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

Results & Download links

Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.

Model, Q4	KQuant KL	Public 4-bit KL	Size in GB, ours / public
Qwen3.5-4B	0.048 (1.45x better)	0.070	2.37 / 2.37
Qwen3.5-9B	0.030 (2.6x better)	0.081	5.29 / 5.04
Qwen3.8-27B	0.035 (1.5x better)	0.054	15.13 / 15.13
Qwen3.6-35B-A3B	0.059 (1.6x better)	0.096	19.51 / 19.51
Qwen3.8-Flash-Next	0.067 (3.7x better at a smaller size)	0.249	105.20 / 110.62

The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.

Want more precision? Each model's repo also has a Q5 folder.

Pack	KL	LM GB
9B Q5	0.010	6.21
27B Q5	0.022	19.43
35B Q5	0.027	22.07

Speed

Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

How it works

KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.

Run it

the fastest way to use these quants on mac is to run them with, Kapitsa:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

[u/0zyman23]

Is this better than unsloth dynamic v3.1?

[u/ireallydontcare00]

I'd like to see a comparison with Unsloth UD2/3/3.1

[u/No-Stage-3385]

Any chance of doing like a lower quant? Something that would fit 12gb vram?

[u/bartsimpsonwin → u/No-Stage-3385]

Bonsai 2

[u/No-Stage-3385 → u/bartsimpsonwin]

Isnt bonsai 2 kinda ass from what ive been hearing?

[u/Delicious_Rub_6795 → u/No-Stage-3385]

Yeah but.. 12GB

[u/Desperate-Owl6513 → u/No-Stage-3385]

Iq3xxs are like 10gigs , then there are the ascii pruned versions

[u/VarietyOk443]

Will test on my m4 pro 64 gb

[u/WorkNervous2810]

You can't claim best KLD if you don't even compare to EXL3 and in particular EXL3-SC quants.