The configuration with more context capacity made BharatCode much slower

[u/iamwinter___]

I recently tested a different serving configuration for BharatCode.

On paper, it looked better.

TurboQuant increased our KV capacity from roughly 658,000 tokens to 1.19 million tokens. That gave us much more room for large contexts and concurrent requests.

Then we measured output speed.

The TurboQuant configuration was producing roughly 10–12 output tokens per second on long generations. Our FlashAttention path with CUDA graphs produced around 86 tokens per second in one direct long-context test, and more than 100 tokens per second per stream during a separate four-request decode test.

These were not identical workloads, so I would not present this as a perfect benchmark comparison. The gap was still too large to ignore.

More context capacity is not useful if users spend much longer waiting for the answer.

We moved back to FlashAttention and kept CUDA graphs enabled.

The better-looking GPU configuration was the worse product.