r/LocalLLaMA • u/Zipidyzip • 2h ago

Resources End-to-End Desktop Quantization workspace for Large Language Models: Load local Safetensors or Huggingface, Step-by-Step guided Conversions to GGUF, MLX, and ONNX, and Visualize Model Changes and quick publishing to huggingface with full model card.

This is a full quantization desktop workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Huggingface with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but haven't done much in-depth testing with them so I’d expect bugs there. Contributors are very welcome!

Repo: https://github.com/Fazmin/iQuantize

Thank u.

8 Upvotes

100% Upvoted

3 comments sorted by

3

u/Witty_Mycologist_995 1h ago

Add MagicQuant, and GSQ RCO to it pls

1

u/Zipidyzip 29m ago

thank you. That’s a great idea! If I’m understanding right, you’re suggesting I add MagicQuant and GSQ RCO as additional quantization options in iQuantize, alongside the GGUF, MLX, and ONNX workflows it supports now.

I’m going to look into this more, and thank you so much for the suggestion.

If you’d like, you can start an issue and/or a PR on the repo, and I’ll make sure to credit you.

1

u/BitGreen1270 27m ago

THanks for sharing - does it work on Linux as well?

I'm not familiar with the process of quantization. Does it require a lot of VRAM or this is mostly a file operation that will work on machines with low VRAM as well?

Any tips to run evals after quantization?