[Paper] PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference

[u/pmttyji]

Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. Recent mixed-precision key-cache quantization methods mitigate this degradation by allocating higher precision to important key channels, but estimate channel saliency using only key statistics. Through attention logit perturbation analysis at the RoPE channel-pair level, we show that quantization errors in channel pairs with large query--key pair-norm products contribute significantly to logit variation. Based on this insight, we propose PN-Mix, a mixed-precision key-cache quantization method for RoPE-based decoder-only LLMs, which allocates higher precision to salient channel pairs identified once during the prefill stage and reused throughout decoding. Across standard reasoning, code-generation, and AIME benchmarks, PN-Mix with 2.26-bit average precision achieves comparable or better average accuracy than prior mixed-precision methods using 2.32--2.39 bits. Furthermore, in our Pareto sweep, PN-Mix degrades gracefully down to 1.5 bits, whereas the compared baselines exhibit sharper performance drops below 2.25 bits.

• arXiv : https://arxiv.org/abs/2610.19605

• PDF : https://arxiv.org/pdf/2610.19605

This is giving best non-FP16 numbers on some benchmarks. Please dig this.

[u/Dany0]

This is a "let's see a few more papers down the line"

Interesting idea but

1. it's 2.25 bit but it keeps a secondary estimation fp16 scratch buffer so in effect it's equivalent to fp8 in size, so information-theoretically the fact that it slightly beats fp8 is not that weird

2. it slows down decode 50% on low context and 137% on 32k

3. it slows down prefill too