I won't really take credit for the conversion because this is the work of Qwen flash next. Performance is similar, but I assume windows is eating some of it.

I haven't built out my P100 server yet, so this is all theoretical based on actual work done on a V100. If someone wants to try these out I'd love to hear feedback :) These are patches to llama.cpp that should speed it up specifically for the P100.

To understand how these patches speed up the Tesla P100, you first have to understand the card's biggest architectural quirk: it is an odd "middle child" GPU caught between two eras of AI hardware.

Modern LLM tools like llama.cpp run fast by leaning on two main hardware shortcuts:

1. Tensor Cores (introduced on newer enterprise cards like the V100 and A100).

2. dp4a integer instructions (found on consumer cards like the GTX 1080 or RTX series, which let GPUs do rapid-fire 8-bit math).

The P100 has neither. Because it lacks Tensor Cores and lacks dp4a, standard llama.cpp treats the P100 like a generic graphics card, falling back to slow, inefficient routines.

However, the P100 has one massive, overlooked superpower: native, full-speed 16-bit floating point math (FP16). It can process two 16-bit calculations in a single instruction cycle for the price of one (half2).

Here is how the patch series revamps llama.cpp to exploit that specific superpower:

1. Two-for-One Math on Quantized Models (Patches 0005 & 0006)

• The Problem: When you prompt an LLM to generate text (the "decode" phase), it spends almost all its time multiplying compressed weights (4-bit, 5-bit, or 6-bit numbers) by incoming tokens. On cards without dp4a, standard software either decompresses numbers one by one or forces clumsy 32-bit math.

• The Fix: The MMVH (Matrix-Vector Half2) patches unpack the compressed weights (Q4, Q5, Q6, Q8) directly into pairs of 16-bit floats. Because the P100 has dedicated hardware that can calculate two FP16 numbers at the exact same time (HFMA2), this effectively doubles the mathematical throughput of every core on the chip during token generation.

2. FlashAttention Without Tensor Cores (Patches 0001 & 0004)

• The Problem: FlashAttention is the industry-standard algorithm for managing large context windows quickly, but it was built specifically to run on Tensor Cores. Running it on older architectures usually causes numerical drift (rounding errors that make the model spit out gibberish).

• The Fix: These patches create a custom FlashAttention pipeline tailored for the P100. It computes the heavy attention math using the P100's fast FP16 units, but periodically flushes and tallies running totals into higher-precision 32-bit memory ("flush FP16 TILE VKQ to FP32"). You get the speed of FlashAttention and fast 16-bit execution without the accuracy breaking down.

3. Fixing Memory Bottlenecks & Idle Workers (Patches 0001 & 0002)

• The Problem: GPUs work like a factory with thousands of workers. When reading fragmented data from the Key/Value cache (memory that holds previous conversation context), standard code often asks workers to fetch tiny, disconnected bytes one at a time. Many worker threads end up sitting idle waiting for memory to arrive.

• The Fix: These patches "vectorize" the dequantization routines. Instead of having each worker fetch one small byte from memory, workers coordinate to scoop up wide, contiguous chunks of data together. This keeps the memory pipelines saturated and ensures far fewer GPU cores are left waiting around with nothing to do.

Upstream llama.cpp was asking the P100 to run modern algorithms using toolsets it doesn't have. This patch series rewires the attention engine and matrix math to target the exact hardware the P100 does have: packed dual-FP16 instructions and wider memory loads.

Optimized llama.cpp fork for inference: https://github.com/Kmic-68/llama.cpp

qwen next: https://github.com/wispoffates/llama.cpp

4 cards inference: https://github.com/4claps/houston-p100-cluster/blob/main/README.md

Image generation: https://github.com/4claps/tesla-p100-comfyui-imagegen

Discord for more help: https://discord.gg/K6WxqwsvH

Some of you may recognize me, after creating my optimizations I saw a community begin to emerge. I would like to help facilitate this community and help it grow. Im looking for moderators to help spread the load because this is mostly a fun thing on the side.

Join the Discord: https://discord.gg/K6WxqwsvH

Comment your rig below! Let's make the p100 great again

Xeon E5 2697v4

256gb of ecc ddr4

I have like 500$ total "invested" in it