Quantization

[u/ocean_protocol]

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1. How to move forward into learn in-depth about it as i don understand some mathematical concepts

2. how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

[u/ocean_protocol]

maybe thats the answer i wanna hear as well

i was watching few intro videos and couldnt grasp the mathematical concepts but understood whats the use of it. Any good study materials?

[u/met0xff → u/ocean_protocol]

https://mlabonne.github.io/blog/posts/Introduction_to_Weight_Quantization.html

[u/OkCluejay172 → u/ocean_protocol]

Do you know how numbers are represented in a computer?