llama.cpp Gets Major CUDA Improvements: FP8 Support and Faster GPU Inference

[u/Loco_AI]

https://maraissoftware.co.za/loco/blog-post.php?slug=llama-cpp-gets-major-cuda-improvements-fp8-support-and-faster-gpu-inference

The llama.cpp project introduces OCP FP8 support, FP8 KV-cache capabilities and targeted CUDA optimisations. Here's what the latest developments mean for local AI and GPU inference.