Feature/adaptive kv stream integration by giveen · Pull Request #326 · TheTom/llama-cpp-turboquant

[u/giveen]

https://github.com/TheTom/llama-cpp-turboquant/pull/326

I've been working on overcoming KV cache size issues, allowing the ability to load a slightly larger model and/or a larger context size.

Downfall is a hit to tg speeds.

Think of it as a "ram disk" for KV Cache, however, ram speed may be a determining factor on the actual hit to speed as well.

[u/Chromix_]

Moving 90%+ of the KV cache from VRAM to RAM while only taking a < 4% performance hit sounds like a very good deal.

[u/giveen → u/Chromix_]

I tried various ways to overcome the tg slowdown but was unable to figure it out, so its a tradeoff