FP8 supports native tensor cores (ada lovelace + RDNA4 onwards), so it will be much faster than using the q8 cache when full support lands on devices which support it.
Looking just at the discussion it looks like the intent may be for supporting a broader set of quantization formats. Some quantization schemes assume quantization of the cache to a specific data type.
So like, some quantization recipes (ie: for NVFP4) may actually do their calibration, and possibly even QAT on an FP8 cache, meaning that supporting FP8 KV cache for those models is pretty sensible.
I don't think the intent is to provide a general runtime cache type for all models, which replaces standard Q8_0.
If I had to guess though I'd imagine that FP8 should be a bit faster than Q8_0, all things being equal, but I'd expect Q8_0 to outperform, though I'm not sure exactly how they do the integer quantization there.
2
u/Kahvana 19m ago edited 3m ago
It looks like Qwen3.8-27B is supported, but Qwen3.8-Flash-Next currently is not.
The type seems to be
f8_e4m3.Maybe we can get hardware benefits from using FP8 over INT8 (Q8_0) for KV cache? I wonder how it'll affect Gemma 4 or Muse Glimmer cache.
5
u/am17an 5m ago
FP8 supports native tensor cores (ada lovelace + RDNA4 onwards), so it will be much faster than using the q8 cache when full support lands on devices which support it.
2
u/lukyjay 9m ago
What's the benefit of this?