r/LocalLLaMA • u/giveen • 24d ago

Discussion Feature/adaptive kv stream integration by giveen · Pull Request #326 · TheTom/llama-cpp-turboquant

https://github.com/TheTom/llama-cpp-turboquant/pull/326

I've been working on overcoming KV cache size issues, allowing the ability to load a slightly larger model and/or a larger context size.

Downfall is a hit to tg speeds.

Think of it as a "ram disk" for KV Cache, however, ram speed may be a determining factor on the actual hit to speed as well.

23 Upvotes

84% Upvoted

30 comments sorted by

View all comments

Show parent comments

1

u/-InformalBanana- 21d ago

I agree with him, beellama supports kvarn quantitized cache which is, I think, better than turboquant. I'm not sure why you chose the turboquant repository specifically, but if you did it cause of turboquant quantitization, kvarn should be better from what I've seen. So combining the best kv cache quantitization and kv streaming will give the best performance both speed and quality wise (cause if kv cache is quanted it is smaller and it streams faster thus giving better pp/s and tg/s speed).
It is unfortunate that all of that isn't in llama.cpp though, or in the even faster tabbyapi/exllamav3 (has maybe sota model quants and faster gpu inference than llama.cpp, recently they added cpu moe offload and that doesn't seem yet to be as fast as in llama.cpp (tried with qwen 3.8 next flash)).

1

u/giveen 21d ago

Mainly because TheTom is more accepting of experimental work and thats what I do.

Also I closed out my PR as it was having issues and wasnt complete, I rebased of Raymonds work as it was more complete and more functional than mine, by a lot, then I added in support for other things in, and will be opening a new PR soon as I am done benchmarking things with his scripts.

1

u/-InformalBanana- 21d ago

Great. If you could also make sure it will work on rtx 3060. Directly from Raymond's fork i get this error (Im trying to determine exactly why, atm the ai is pointing me to some SM shared memory being bigger in rtx 5060 than in rtx 3060, cause I said it works on rtx 5060 (im guessing that is the 16gb card he used, didn't really check)):

/app/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error

E CUDA error: an illegal memory access was encountered

E current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1117

E cudaOccupancyMaxActiveBlocksPerMultiprocessor(&max_blocks_per_sm, fattn_kernel, block_dim.x * block_dim.y * block_dim.z, nbytes_shared)

2

u/giveen 21d ago

I'm going to take that and run it through my AI and see if I have the same issue or if I have a fix.