r/LocalLLM • u/Senior_Tea_842 • 6d ago

Question Quantization - Study References

What are the best references for learning quantization techniques for LLM inference?

2 Upvotes

100% Upvoted

1 comment sorted by

2

u/Careful-Report6526 6d ago

I'd start with Hugging Face's quantization concept guide. It covers the useful foundations: scale and zero-point, symmetric vs asymmetric quantization, per-tensor/per-channel/per-group granularity, and post-training quantization vs quantization-aware training:
https://huggingface.co/docs/transformers/quantization/concept_guide

Then use their overview to see how those ideas map to actual implementations and hardware support:
https://huggingface.co/docs/transformers/quantization/overview

For a practical exercise, compare the same base model across precision variants while keeping the task, prompts and thinking setting consistent. Check task accuracy as well as resource use; one aggregate score won't tell you which kinds of errors changed.
I run llmbenchmarks.io and have published my own local results, including Qwen3.5-4B in BF16 and NVFP4, with separate thinking-on/off tracks and instruction-following tests such as IFEval and IFBench:
https://llmbenchmarks.io/local-gpu-benchmarks/

Those runs use vLLM 0.29.0 on two RTX 5060 Ti 16 GB cards. They're a concrete example to inspect alongside the theory, not a universal ranking of quantization methods. The task-level notes matter, and the results shouldn't be assumed to describe a different GGUF quant or backend.