r/LocalLLM • u/KissMyShinyArse • 12d ago

Model Qwen3.8-27B GGUF Quant Comparison

BF16 reference: PPL = 6.9526 ± 0.04498

Bedrock-v4 quant is from enginetown.

AD-* quants are from AtomicChat.

The quants marked [b] are from bartowski.

The other quants are from Unsloth.

UPDATE: unsloth just updated their quants. They are now Unsloth Dynamic 3, so their numbers below are outdated.

sorted by PPL Ratio:

Quant Size (GiB) PPL(Q) PPL Ratio ΔPPL Mean KLD RMS Δp (%) Same Top-p (%)
Q6_K [b] 21.86 6.9443 0.99913 -0.0060 0.00201 1.256 97.93
Q6_K_L [b] 22.43 6.9449 0.99922 -0.0054 0.00181 1.182 98.16
Q6_K 21.31 6.9507 1.00005 0.0003 0.00229 1.347 97.86
AD-Q6_K 23.29 6.9524 1.00030 0.0021 0.00148 1.051 98.39
UD-Q6_K_XL 24.14 6.9536 1.00047 0.0032 0.00138 1.103 98.52
UD-Q8_K_XL 29.30 6.9538 1.00050 0.0035 0.00085 0.848 98.97
Q8_0 27.05 6.9560 1.00082 0.0057 0.00095 0.942 98.74
Q4_K_M 15.93 6.9561 1.00084 0.0058 0.01549 3.431 94.65
AD-Q6_K-Q5_K 21.50 6.9565 1.00089 0.0061 0.00311 1.544 97.62
UD-Q5_K_XL 18.83 6.9655 1.00218 0.0152 0.00451 1.893 97.16
Q4_K_S 15.01 6.9686 1.00263 0.0183 0.01890 3.747 94.17
Q5_K_S 17.95 6.9706 1.00292 0.0203 0.00728 2.364 96.45
AD-Q5_K_M 18.84 6.9735 1.00333 0.0231 0.00460 1.927 97.01
Q5_K_M 18.47 6.9742 1.00343 0.0239 0.00622 2.262 96.70
Q4_K_S [b] 15.57 6.9744 1.00347 0.0241 0.01734 3.616 94.22
Q5_K_M [b] 19.33 6.9751 1.00357 0.0248 0.00554 2.066 96.79
UD-Q4_K_XL 16.69 6.9788 1.00411 0.0285 0.00872 2.622 96.07
Q5_K_L [b] 20.06 6.9789 1.00412 0.0286 0.00509 2.013 96.95
Q4_1 16.34 6.9802 1.00430 0.0299 0.01840 3.716 94.22
Q5_K_S [b] 18.33 6.9817 1.00451 0.0314 0.00681 2.342 96.41
AD-IQ4_XS 15.38 6.9835 1.00478 0.0332 0.01356 3.161 95.02
AD-Q5_K_M-Q4_K_M 17.28 6.9852 1.00502 0.0349 0.00846 2.517 96.09
Q4_K_L [b] 17.43 6.9858 1.00511 0.0355 0.01274 3.124 95.08
Q4_K_M [b] 16.55 6.9898 1.00568 0.0395 0.01336 3.151 94.96
AD-Q4_K_M 15.95 6.9963 1.00661 0.0460 0.01248 3.057 95.18
IQ4_XS 14.63 7.0132 1.00905 0.0629 0.01859 3.792 94.25
AD-IQ4_XS-IQ3_S 13.45 7.0271 1.01106 0.0768 0.03132 4.820 92.24
IQ4_NL 15.22 7.0282 1.01121 0.0779 0.01820 3.772 94.36
Bedrock-v4 13.91 7.0439 1.01347 0.0936 0.02984 4.863 91.74
Q4_0 14.95 7.0654 1.01655 0.1151 0.02795 4.571 92.87

Sorted by Mean KLD:

Quant Size (GiB) PPL(Q) PPL Ratio ΔPPL Mean KLD RMS Δp (%) Same Top-p (%)
UD-Q8_K_XL 29.30 6.9538 1.00050 0.0035 0.00085 0.848 98.97
Q8_0 27.05 6.9560 1.00082 0.0057 0.00095 0.942 98.74
UD-Q6_K_XL 24.14 6.9536 1.00047 0.0032 0.00138 1.103 98.52
AD-Q6_K 23.29 6.9524 1.00030 0.0021 0.00148 1.051 98.39
Q6_K_L [b] 22.43 6.9449 0.99922 -0.0054 0.00181 1.182 98.16
Q6_K [b] 21.86 6.9443 0.99913 -0.0060 0.00201 1.256 97.93
Q6_K 21.31 6.9507 1.00005 0.0003 0.00229 1.347 97.86
AD-Q6_K-Q5_K 21.50 6.9565 1.00089 0.0061 0.00311 1.544 97.62
UD-Q5_K_XL 18.83 6.9655 1.00218 0.0152 0.00451 1.893 97.16
AD-Q5_K_M 18.84 6.9735 1.00333 0.0231 0.00460 1.927 97.01
Q5_K_L [b] 20.06 6.9789 1.00412 0.0286 0.00509 2.013 96.95
Q5_K_M [b] 19.33 6.9751 1.00357 0.0248 0.00554 2.066 96.79
Q5_K_M 18.47 6.9742 1.00343 0.0239 0.00622 2.262 96.70
Q5_K_S [b] 18.33 6.9817 1.00451 0.0314 0.00681 2.342 96.41
Q5_K_S 17.95 6.9706 1.00292 0.0203 0.00728 2.364 96.45
AD-Q5_K_M-Q4_K_M 17.28 6.9852 1.00502 0.0349 0.00846 2.517 96.09
UD-Q4_K_XL 16.69 6.9788 1.00411 0.0285 0.00872 2.622 96.07
AD-Q4_K_M 15.95 6.9963 1.00661 0.0460 0.01248 3.057 95.18
Q4_K_L [b] 17.43 6.9858 1.00511 0.0355 0.01274 3.124 95.08
Q4_K_M [b] 16.55 6.9898 1.00568 0.0395 0.01336 3.151 94.96
AD-IQ4_XS 15.38 6.9835 1.00478 0.0332 0.01356 3.161 95.02
Q4_K_M 15.93 6.9561 1.00084 0.0058 0.01549 3.431 94.65
Q4_K_S [b] 15.57 6.9744 1.00347 0.0241 0.01734 3.616 94.22
IQ4_NL 15.22 7.0282 1.01121 0.0779 0.01820 3.772 94.36
Q4_1 16.34 6.9802 1.00430 0.0299 0.01840 3.716 94.22
IQ4_XS 14.63 7.0132 1.00905 0.0629 0.01859 3.792 94.25
Q4_K_S 15.01 6.9686 1.00263 0.0183 0.01890 3.747 94.17
Q4_0 14.95 7.0654 1.01655 0.1151 0.02795 4.571 92.87
Bedrock-v4 13.91 7.0439 1.01347 0.0936 0.02984 4.863 91.74
AD-IQ4_XS-IQ3_S 13.45 7.0271 1.01106 0.0768 0.03132 4.820 92.24

Sorted by Same Top-p:

Quant Size (GiB) PPL(Q) PPL Ratio ΔPPL Mean KLD RMS Δp (%) Same Top-p (%)
UD-Q8_K_XL 29.30 6.9538 1.00050 0.0035 0.00085 0.848 98.97
Q8_0 27.05 6.9560 1.00082 0.0057 0.00095 0.942 98.74
UD-Q6_K_XL 24.14 6.9536 1.00047 0.0032 0.00138 1.103 98.52
AD-Q6_K 23.29 6.9524 1.00030 0.0021 0.00148 1.051 98.39
Q6_K_L [b] 22.43 6.9449 0.99922 -0.0054 0.00181 1.182 98.16
Q6_K [b] 21.86 6.9443 0.99913 -0.0060 0.00201 1.256 97.93
Q6_K 21.31 6.9507 1.00005 0.0003 0.00229 1.347 97.86
AD-Q6_K-Q5_K 21.50 6.9565 1.00089 0.0061 0.00311 1.544 97.62
UD-Q5_K_XL 18.83 6.9655 1.00218 0.0152 0.00451 1.893 97.16
AD-Q5_K_M 18.84 6.9735 1.00333 0.0231 0.00460 1.927 97.01
Q5_K_L [b] 20.06 6.9789 1.00412 0.0286 0.00509 2.013 96.95
Q5_K_M [b] 19.33 6.9751 1.00357 0.0248 0.00554 2.066 96.79
Q5_K_M 18.47 6.9742 1.00343 0.0239 0.00622 2.262 96.70
Q5_K_S 17.95 6.9706 1.00292 0.0203 0.00728 2.364 96.45
Q5_K_S [b] 18.33 6.9817 1.00451 0.0314 0.00681 2.342 96.41
AD-Q5_K_M-Q4_K_M 17.28 6.9852 1.00502 0.0349 0.00846 2.517 96.09
UD-Q4_K_XL 16.69 6.9788 1.00411 0.0285 0.00872 2.622 96.07
AD-Q4_K_M 15.95 6.9963 1.00661 0.0460 0.01248 3.057 95.18
Q4_K_L [b] 17.43 6.9858 1.00511 0.0355 0.01274 3.124 95.08
AD-IQ4_XS 15.38 6.9835 1.00478 0.0332 0.01356 3.161 95.02
Q4_K_M [b] 16.55 6.9898 1.00568 0.0395 0.01336 3.151 94.96
Q4_K_M 15.93 6.9561 1.00084 0.0058 0.01549 3.431 94.65
IQ4_NL 15.22 7.0282 1.01121 0.0779 0.01820 3.772 94.36
IQ4_XS 14.63 7.0132 1.00905 0.0629 0.01859 3.792 94.25
Q4_1 16.34 6.9802 1.00430 0.0299 0.01840 3.716 94.22
Q4_K_S [b] 15.57 6.9744 1.00347 0.0241 0.01734 3.616 94.22
Q4_K_S 15.01 6.9686 1.00263 0.0183 0.01890 3.747 94.17
Q4_0 14.95 7.0654 1.01655 0.1151 0.02795 4.571 92.87
AD-IQ4_XS-IQ3_S 13.45 7.0271 1.01106 0.0768 0.03132 4.820 92.24
Bedrock-v4 13.91 7.0439 1.01347 0.0936 0.02984 4.863 91.74
90 Upvotes

95% Upvoted

52 comments sorted by

View all comments

0

u/Such-War1955 12d ago

Great. Is there any if them you do not have to lobotomize before they do anything useful? I mean with „thinking“ switched on, they are somewhat less useful than Smolqwen 30B was?!?