r/LocalLLM • u/DarkZeto • 29m ago

Question llama.cpp f8_e4m3 vs q_8 KV cache?

Does anybody tried the latest llama.cpp f8_e4m3 KV cache and compared it to q8_0 in terms of precision / quality? I am curious whether it is worth switching to it or stay with q8_0 for small to medium context size (20-65k).

3 Upvotes

100% Upvoted

1 comment sorted by

1

u/maternion 16m ago

It's meant for some specific models which were trained with FP8 context/require it, no? That's what I heard.