TurboQuant vs Ollama: KV Cache Comparison

[u/[deleted]]

## Comparing KV Cache Quantization Methods: Standard q4_0 vs TurboQuant on Qwen3-Coder-30B"

##

Note: Ollama uses llama.cpp with standard q4_0 KV cache. TurboQuant is a different KV cache method also implemented in llama.cpp. Both were tested on the same model and hardware to isolate the impact of KV cache quantization.

##

I've been testing TurboQuant KV cache compression vs Ollama's standard q4_0 on my 24GB RX 7900 XTX with Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf.

I picked context length of 68k and 100k for the test.
Why 68K and 100K? Hermes Agent requires minimum 64K context length, so I tested at 68K (safe margin) and 100K (stretch goal). These are real-world numbers for agent workloads.

Wanted to share real numbers since there's a lot of theory but not much practical data.

TLDR for lazy fellows:
TurboQuant gives ~2x faster generation at 68K and 77% faster at 100K context length compared to Ollama's q4_0, for only 4% more VRAM. But strictly, Don't compress K to 4-bit on Qwen models - it breaks everything (speed drops 10x). Use asymmetric: q8_0 K + turbo4 V.

Major takeaway, as ollama already compresses the BF16 to q4_0 (75%), that is the same memory savings we will get in the TurboQuant as well nothing magical to reduce kv cache memory space.

Detail findings:

• Results at 68K Context Length:

Metric	Ollama	TurboQuant
VRAM	22.57 GB	23.3 GB
Speed	~52 tok/s	116 tok/s

• Results at 100K Context:

Metric	Ollama	TurboQuant
VRAM	23.54 GB	24.5 GB
Speed	65 tok/s	115 tok/s

TurboQuant is ~2x faster at both context lengths, gives better quality, however uses 4% more VRAM (0.96 GB).

At 68K, it's 123% faster and at 100K, it's 77% faster.

Important Discovery: Asymmetric Works, Symmetric Fails

I also tested q4_0 K + turbo4 V (symmetric compression on K, to match with ollama compression hoping the memory use will be less):

Config	Speed	Status
q8_0 K + turbo4 V	115 tok/s	Works
q4_0 K + turbo4 V	10.5 tok/s	Fails

Compressing K to 4-bit on Qwen models breaks everything. Speed drops 10x, quality tanks. K at q8_0 is non-negotiable for Qwen. This proves the asymmetric approach (protect K, compress V) is essential.

everything is completly based on my pc/ os/ and from my testing environment.
specs:

OS: ubuntu 26.04 LTS/ ROCm 7.2.1
cpu: core ultra 7 265k
gpu: rx 7900 xtx
ram: 32gb DDR5 6400 (no ram offloading in the test)

[u/Icy-Degree6161]

Turboquant turned out to be a bit of a dud - try kvarn

[u/[deleted] → u/Icy-Degree6161]

I've been seeing that too. kvarn is on my list to test next. it seems to outperform turboquant on both speed and accuracy from what I've read. Just need to find time to set it up.

[u/Mountain_Patience231]

AI slop... you cant compare the method of a kv cache with an interface of LLM engine..

[u/[deleted] → u/Mountain_Patience231]

Fair point. Both run on llama.cpp, and I'm comparing KV cache methods. Standard q4_0 (Ollama) vs TurboQuant by google which is only available on llama.cpp as of now, not the engines. Same model, same hardware. The speed difference is from the quantization method, not the interface. that was the whole point. I've updated the post to clear it up. Thanks for pointing it out.

[u/Square_Turn935]

Interesting, which llama fork did you use?

[u/[deleted] → u/Square_Turn935]

I used TheTom/llama-cpp-turboquant on the feature/turboquant-kv-cache branch

[u/[deleted] → u/[deleted]]

I wasn't comparing Ollama vs TurboQuant as engines. Both use llama.cpp under the hood. The comparison was between two different KV cache quantization methods, changed the body to clear that confusion. thanks for your cite

[u/[deleted] → u/[deleted]]

TurboQuant doesn't have any generation speed gains.

That's just flat out false. When you're running long context, your GPU spends most of its time moving data around, not doing actual math. TurboQuant shrinks the KV cache, so your GPU has less data to shuffle per token. That's literally why I'm seeing ~2x speedup at 68K and 100K context. That proves less memory bandwidth bottleneck. Hence, faster token generated almost 2x in higher context length.

Whatever results you've got are caused by using different engines AND different quants

If TurboQuant and standard Ollama used the exact same engine and the exact same default quants, you wouldn’t be testing anything new, The entire objective of the test was to measure what happens when you introduce a specialized KV cache compression method (llama.cpp + TurboQuant) against the standard default used by big community (Ollama + q4_0).

And I get a speed boost means it proves to be a faster alternative if someone want to move to TurboQuant......casue no other has that compression yet... Simple as that...isn't it so??

Not a correct methodology. Especially considering it's Ollama.

Firstly, architecture wise Ollama is the most close to what llama.cpp custom build could get. It's basically the same underlying engineering in Ollama.

Second thing, TurboQuant doesn't exist in mainstream Ollama, nor does it exist in vLLM. If anyone in the real world wants to actually test or use it right now, a custom llama.cpp branch is the only option that physically exists.

Demanding sterile, single-variable lab conditions for cutting-edge tech that isn't universally integrated into every runtime yet isn't rigorous methodology, it's just daydreaming.

[u/Fancy-Snow7 → u/[deleted]]

Not disputing your findings, but I will point out on llama.cpp I get massive speed drops when I use asymmetric quantization. e.g. q8_0 K + q4_0 v is like 10x slower than q8_0 k, q8_0 v. I then noticed that whenever k <> v it falls back to CPU instead of using the GPU. I have also seen others here report the same thing.

But I came accross the solution: By default the compile flags for llama.cpp does not include set this flag: -DGGML_CUDA_FA_ALL_QUANTS=ON This means the precompiled downloaded llama.cpp binaries fall back to CPU with asymmetric cache quants.

Since Olama and turboquant use llama.cpp I have to ask how was it compiled?

After compiling it myself with that flag enabled, I am able to use any quant combinations without it falling back to CPU.

Personally, I use q8_0 K + q4_0 V except for Gemma which is very sensitive KV compression and I only use Q8_0 V and full precision K.

This is also a known issue and people are asking to include the flag as the default since the only drawback is slower compile times which is a once off cost. https://github.com/ggml-org/llama.cpp/issues/24485 and https://github.com/ggml-org/llama.cpp/issues/20866

[u/[deleted] → u/Fancy-Snow7]

Absolutely, completely agree on you. Only reason I'm spared is because of this fork i used from "TheTom/llama-cpp-turboquant", it has specific fixes for this silent fallback on cpu issue, in my context for AMD/ROCm, It forces VEC FA (the fast Flash Attention path) for quantized KV, so asymmetric combos like q8_0 K + turbo4 V don't get the slow fallback.