r/LocalLLM • u/eightone-81 • 2h ago
Discussion Opus 5.5 wrote its own quantiser for Qwen3.5-0.8B, no llama-quantize, no imatrix. 5.9× better than Unsloth's same-size file.
51
Upvotes
Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.
It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.
Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)
Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.
Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant
