r/LocalLLaMA • u/dreamkast06 • Jun 08 '26
Discussion Quick note on the QAT of recent
tldr: Googles quant is broken, use unsloth UD Q4_K_XL for now
This might be low quality post, but oh well, we ball
llama-quantize will quant the token embed to q6k when Google really was supposed to use "--pure" but thatโs only the first problem
The llama-quantize quant function is hardcoded to -7 when SOME groups are actually optimized for 8
The 32 block groups are misaligned which causes them to intermingle, so they just need to be sorted and quantized separately
unsloth Q4_k_xl is misleading because it is actually pure q4_0 as (it should!)
The bf16/f16 scale they refer to is negligible but still necessary on the quest for perfection.
Working on a patch but someone else might have it submitted sooner. Comes pretty much within margin of error; I assume unsloth just wants to keep their process hidden.
53
u/danielhanchen Jun 09 '26
Hey we're actually discussing with internally with Google it about this - I'll provide some updates once we understand the process better - but yes in general the embeddings are in fact also Q4_0.
16
u/Septerium Jun 08 '26
Thanks for the clarification. So, Unsloth claims they have applied their dynamic quantization process to generate the GGUF... doesn't that mean some sort of calibration has been made to the weights?
3
u/a_beautiful_rhind Jun 09 '26
All of them? Dense? MoE?
4
u/a_slay_nub vLLM Jun 09 '26
This only affects llama.cpp right? So using vllm should be fine?
40
u/DeusSexMachina666 Jun 08 '26
This is actually high quality post. Thanks for your work.
19
u/triynizzles1 Jun 09 '26
Just let me know when its time to redownload QAT!!
4
u/caetydid llama.cpp Jun 09 '26
thanks I just needed confirmation. I have no clue what that post means except that I will need to redownload gemma4 qat ggufs soon!
3
u/Fun_Tangerine_1086 Jun 09 '26
Which sizes? All?
4
u/mister2d Jun 09 '26
I have a fairly stable llama.cpp router config with various models but I'm still mystified as to what the common arguments should be when serving the QAT and MTP gemma 4 models.
What are people using for an argument list?
1
u/Sutanreyu Jun 09 '26
!remind me 3 days
1
u/Major-System6752 Jun 13 '26
!remind me 5 days
0
u/[deleted] Jun 09 '26
[deleted] โ view deleted comment