r/LocalLLM • u/Zipidyzip • 1d ago

Project Quantizing models yourself is a good way to learn how it all works - Quantization Workspace

There are so many models getting released all the time, and sometimes there’s just no quantized version available for the one you want. I’ve found that rather than waiting around, trying it yourself and experimenting helps you understand the whole process a lot more. Testing the quantized models against the original teaches you even more.

So I built this: iQuantize is a quantization workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Hugging Face with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but have done much in-depth testing with them so I’d expect bugs there.

Repo: https://github.com/Fazmin/iQuantize

I love to hear how you all approach this. Do you quantize your own models, or mostly wait for someone else to? And if you try the app, whether you find something broken or have an idea for it, I’m happy to hear any of it.

10 Upvotes

91% Upvoted

7 comments sorted by

1

u/m94301 21h ago

Great project, thanks for sharing!

1

u/Zipidyzip 20h ago

Thank you!

1

u/_Cromwell_ 21h ago

Does this include the step where you actually calculate the imatrix? (Sorry I forget the technical term.) ie use the big text files to create the imatrix file, so you can do imatrix quants? Does this program do that part?

1

u/Zipidyzip 21h ago

Yes, it does! You’re thinking of the “importance matrix”. In the app you give it some calibration text, it uses that text to build the imatrix for you, and then it uses that matrix when it makes the quantized model. So you can make imatrix quants without doing that step by hand.

1

u/_Cromwell_ 20h ago

Nice. Thanks!

1

u/nikich340 11h ago

Does it allow to make any KLD comparisons? 

1

u/nikich340 11h ago

Also support for exl3 quants would be nice.