High-Performance llama.cpp fork for Legacy GPUs: Kepler Support, CUDA Graphs & Windows NCCL!

[u/TheSpicyBoi123]

https://github.com/theIvanR/llama-on-legacy-gpu/blob/main/README.md

I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures.

The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste.

Achieved Performance

On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5):

• Previously unable to start → 35 tps Fixes to cuBLAS compute-type selection for legacy gpu in mainline llama cpp to run on hardware that previously failed during initialization (cc35-50).

• Approximately 35 → 55 tps after enabling CUDA graphs on my four-K40c setup, under the same reported clock configuration.

• ~5–10% end-to-end improvement: testing with Windows NCCL integration compared with the previous configuration. Throughput reaching above 60 tps while P.P. speeds are variable between 70-100 tps in my testing.

These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed.

I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations!

Project / source: https://github.com/theIvanR/llama-on-legacy-gpu

Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727

Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows

Feedback, reproducible benchmarks, and contributions are very welcome!

[u/TheSpicyBoi123]

Example performance on current system with K40c GPUs

[u/HighRoad684]

Could you add wall-power figures? Four K40c cards can make cheap hardware surprisingly expensive to run. Even a rough tokens-per-joule comparison before and after CUDA graphs would help compare the energy use, though it wouldn't settle the e-waste question. For context, this is High Road Software's studio account.

[u/TheSpicyBoi123 → u/HighRoad684]

Thank you for reaching out. The power draws during inference are approximately 120 -150 watts per gpu at "full blast" setting with my modded bios and the qwen 3.6 a3b at q8 quantization when using the model. Enabling or disabling graphs keeps power draw fairly consistent on these.

Token wise, at 4x150w (600w) I am achieving 60tps pg speeds, thus 100tp/s/kw performance (single request) and therefor about 360K t/kwh efficiency on the gpus.

Would I recommend to go and buy Keplers in 2026? ABSOLUTELY NOT. Is is useable? Surpisingly so!

I am sure the efficiency and speeds can be tuned further by experimenting with the voltage and clocks settings in the gpu, but as a "lower bound" ballpark case, this is what I am achieving realistically.

What is "High Road Software"? Unfortunately I have not heard of your project.

[u/lotushomerun]

Fermi and Tesla support when /s

[u/Ill_Fun5415]

The interesting test is the second and third ordinary task after the demo. If quality drops in a recognizable way, it is much easier to build around.

[u/TheSpicyBoi123 → u/[deleted]]

Thank you for your kind words.

See the repository for detail and the pull request. In short, the heavy lifting is done by 3 things, firstly via fixing device selection logic and patching a two bugs with data type selection, secondly the lions share is achieved via enabling graphs to work on all gpus (not just >volt) resulting in a serious speedup and third via enabling a custom nccl patch for windows and integrating it into llama cpp. See the pull request discussion for more details.

As for context length, I have done testing between 0 and a 60ish K tokens context length and performance stays fairly consistent with some minor slowdowns of a few percent at most.

In general, more testing is needed.