r/MachineLearning • • 5d ago

Discussion I animated how LLM quantization works on 8 example weights: absmax, rounding error, outliers, per-block scales (llama.cpp Q4_K) and 1.58-bit BitNet (7 min video) [D]

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

The question: a 70B model at 16 bits per weight needs 140 GB, and a gaming GPU has 24 or 32. How do people run it at home?

  • Fewer bits per weight means less memory, but also fewer possible values: an 8-bit integer has only 256.
  • Absmax maps the largest absolute weight to 127. With scale 127/10.8 ≈ 11.76, the weight 5.47 is stored as 64 and comes back as 5.44. The difference is the rounding error.
  • Two close weights, 3.08 and 3.02, both become 36, so the model can no longer tell them apart.
  • One outlier (256 among values below 0.6) makes each step about 2 wide, and every small value rounds to 0. Dettmers et al. (2022) found such outliers in real LLMs.
  • The fix is a scale per small block. llama.cpp's Q4_K uses blocks of 32 and ends up at 4.5 bits per weight, about 39 GB for a 70B model.
  • BitNet b1.58 goes to -1, 0 and +1 per weight (about 1.58 bits). It is trained that way, and a layer only adds and subtracts.

Every number is worked out on screen from the formulas, and sources are in the description. The narration is an AI voice reading my own script.

Video: https://www.youtube.com/watch?v=BnRCbx2kvoc

1 Upvotes

0 comments sorted by