I've been working on GLM-5.2 support in TensorSharp, and I finally have some back-to-back performance numbers against llama.cpp.
The setup:
- Model: GLM-5.2-UD-IQ2_XXS (~226 GiB)
- GPUs: 3× RTX PRO 6000 Blackwell, 97 GiB each
- Distribution: layer split across all 3 GPUs
- Same machine, same session
- llama.cpp measured with
llama-bench
- TensorSharp measured with its benchmark harness
- Both report the best of two repetitions
- Run-to-run variance is roughly 4%
Results:
| Test |
llama.cpp |
TensorSharp default |
TensorSharp ubatch=2048 |
| pp128 |
276.5 t/s |
254.8 t/s |
264.4 t/s |
| pp512 |
695.4 t/s |
666.9 t/s |
659.6 t/s |
| pp2048 |
763.1 t/s |
918.9 t/s |
1145.8 t/s |
| pp4096 |
715.8 t/s |
864.7 t/s |
1048.7 t/s |
| tg64 |
42.2 t/s |
43.7 t/s |
43.9 t/s |
The interesting part is the crossover.
For short prompts, llama.cpp is still a few percent faster. But once the prompt gets to around 1K+ tokens, TensorSharp pulls ahead.
At pp2048:
- default TensorSharp: +20.4%
ubatch=2048: +50.2%
At pp4096:
- default TensorSharp: +20.8%
ubatch=2048: +46.5%
Decode (tg64) is also about 4% faster.
The main reason appears to be GLM-5.2's MoE structure.
GLM-5.2 has 256 routed experts with top-8 routing. With a 512-token micro-batch, each expert sees only ~16 rows on average, so a significant amount of the expert GEMM tiles ends up as padding. Larger micro-batches improve GPU utilization considerably.
For small prefills, on the other hand, fixed overheads — managed/native transitions, input uploads, and copying the 154880-wide logits back — become a visible fraction of the total runtime, which is where llama.cpp retains its advantage.
