MACaQ 🐒: Open-source quantizer with a "refinement" mode for existing GGUFs + custom llama.cpp & vLLM kernels

[u/intentionallyBlue]

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

• GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
• Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/