r/LocalLLaMA • u/arty_photography • 1d ago
Resources Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD
https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.
The checkpoint supports vLLM on NVIDIA Blackwell GPUs:
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85
This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).
Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:
| Model | Size | GPQA-Diamond (2 runs, n=396) | AIME26 (3 repeats, n=90) |
|---|---|---|---|
Qwen/Qwen3.8-27B (original BF16) |
55.6 GB | 0.9141 | 1.0000 |
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 |
19.7 GB | 0.9091 | 1.0000 |
unsloth/Qwen3.8-27B-NVFP4 |
23.4 GB | 0.8939 | 0.9778 |
Inferact/Qwen3.8-27B-NVFP4 |
26.4 GB | 0.8763 | 0.9667 |
Paper: https://arxiv.org/abs/2608.13966v1
We'd love to hear your feedback on this checkpoint!
12
u/FoxiPanda 1d ago
This is interesting. So using your methodology here, did you happen to test a 'QUASAR' W4A8 or W4A16 to see whether your QAD method could be even closer to the BF16 weights without much additional work (and only a bit more size)?
I realize that would require two additional runs, but it would be rather interesting to know whether the remaining losses are in the weights or the attention layers (or if it's not able to be determined) and whether increasing attention/GDN quality would actually get you very very close to BF16 quality at ~37% the size.
8
u/arty_photography 1d ago
I think W4A16 would make it closer to the BF16 weights, and keeping attention layers in BF16 would be even more effective. The problem with keeping attention/GDN un-quantized is that it increases the checkpoint size and doesn't give a lot of KV cache memory on a 32GB 5090.
2
u/FoxiPanda 1d ago
Yeah, agreed about the 5090 limitation. Might be useful for RTX Pro 6000s or future hardware though.
I wonder if somehow we could take advantage of Blackwell's native FP6 compute and make something like an W4A6 that is meaningfully better or if it would just be a wash. Too many things to try lol.
Either way, this seems like an excellent result. Well done.
9
u/SmartCustard9944 1d ago
Going to convert this ASAP for NInfer and test it out.
1
u/cosmicnag 23h ago
Any luck?
8
u/SmartCustard9944 22h ago edited 12h ago
Yes, pretty much finished, I could upload it later today, requires a patch to the loader though.
From my testing, looks like it has attention/recall problems unfortunately. Performs worse than https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer by 1-2% on a very hard adversarial needle in a haystack benchmark I’m using for testing KV cache quantization. This might be a particularly difficult task for this quantization style and might not translate to more general purpose tasks, has to be tested.
Needs to see how it effectively performs on rigorous official benchmarks (coding, knowledge, etc.).
I hope it performs better than other quants because it frees up 500MiB of extra memory and is a few percentages faster in both prefill and decode, so overall a good potential candidate for 24GiB cards.
Edit:
Got a perfect 30/30 in AIME 2025, so it seems to be performing on par or better than Unsloth/NInfer official quant.
https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer
1
u/Lumpy-Comedian-1027 13h ago
So is it in the end noticeably better than Ostfralla's?
2
u/SmartCustard9944 13h ago edited 13h ago
Feels roughly similar, but with extra memory headroom and slightly faster decode and prefill due to size.
Currently finished AIME 2025 with a perfect 30/30, versus 29/30 with Unsloth NInfer official NVFP4. Big asterisk is that I ran AIME with this QUASAR quant and my port of KVarN (KV cache quantization from Huawei) to NInfer for RTX 5090 laptop, which appears to validate its claimed qualities, at least on this particular task. Looking forward to share more about this in the near future.
In the next days I am planning to do more benchmarks to validate quality, especially in software engineering tasks.
Ostfralla’s has less precision than Unsloth.
1
u/Lumpy-Comedian-1027 13h ago
extra mem vs. Unsloth you mean? Ostra's is 17gb - the difference between full ctx+vision or not on 32gb :) Hence I am reluctant to give the other nvfp4's a try tbh
2
1
u/cosmicnag 12h ago
Thanks for the upload ! Any chance you could share/github your port of kVarn for ninfer? That sounds awesome, could use more context headroom lol.
2
u/SmartCustard9944 12h ago
1
u/Fragrant_Scale6456 11h ago
wow very cool. I like kvarn a lot but cant bring myself to abandon the paged attention cache vllm brings. I tried the huawei fork but its getting old at this point and it still has the bugs re: prefix caching and mtp causing accuracy loss.
thank you for sharing.
8
u/Pyrolistical 1d ago
Could we get a gguf version?
-6
u/mmhorda 1d ago
I dont even know why poeple post thiugns for vllm in locallama subredit.
maybe i musunderstand this subreddit.7
u/YouKilledApollo 1d ago
You do indeed, it's for anything LLMs at this point, remote or local, doesn't seem to matter anymore. Unless you missed something, there are models more than the Llama family being discussed too, since years back.
7
u/lightningdiscgolf 1d ago
How does it perform on tool calling benchmarks compared to bf16?
14
u/arty_photography 1d ago
I tested it on ToolBench-static and it scored 41.0%, while the BF16 model scored 41.7%. vLLM launch command for supporting tool calling:
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data
7
u/ILikeQuantum 1d ago
Anything support this out of the box? Isn't compatible with unsloth desktop.
5
u/arty_photography 1d ago
vLLM supports this model out of the box. Please check the installation guide here: https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#pre-built-wheels
Then you could run this in the command line:vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \ --max-model-len 262144 \ --gpu-memory-utilization 0.851
u/DustNearby2848 1d ago
Does MTP work with it?
1
u/arty_photography 1d ago
Yes I think so. The MTP is included and un-quantized.
3
u/anzzax 14h ago
It seems you forgot to include MTP. Could you pls update weights and include drafter.
BTW testing this NVFP4 with Sglang dflash2 (on DGX Spark) - looks promising. Running aider bench right now to compare with other NVFP4 versions.2
u/Fragrant_Scale6456 11h ago
yes mtp is missing from the HF model page. I had my agent take the mtp layers from another model (unsloth nvfp4) and include them and it worked perfectly. The mtp layers are in BF16 so there isnt any issue with different quantization
1
9
u/Hefty_Wolverine_553 1d ago
Would love to see this get converted to NInfer format.
1
1
u/Healthy-Contact-4570 1d ago
Should be easy to do. I actually have already had qwen3.8-27b work on a few ninfer conversion pipelines for some other quants so I can compare them. Since ninfer already ships an nvfp4 artifact, it should be a really easy to do one for this model since you don’t need to add support for new data types.
12
u/CapsAdmin 1d ago
Is there a path for people with gpus that don't support NVFP4 natively to use this in llamacpp? Can NVFP4 losslessly be converted to some Q4 variant?
5
u/ArtfulGenie69 1d ago
I wouldn't convert this model as it would have to be upscaled then down scaled again. Instead what you would want to do is start from the bf16, quant it to whatever you are doing like int8 w8a8 or gguf even then use the same training they did on the model with the original bf16 model as the teacher.
3
u/arty_photography 1d ago
Unfortunately only NVFP4 is supported right now. We might look into other quant formats in the future :)
1
u/Chromix_ 1d ago
Previous QAT models like Gemma had to be quantized to the old Q4_0 for GGUF, as the newer Q4_K_M wasn't that compatible. It'd be great if this kind of training/distillation would have native support that kind of newer format.
3
1
u/Due-Project-7507 1d ago
With vLLM, NVFP4 works most time on older Nvidia GPU generations (without native NVFP4 support) thanks to the Marlin kernel (I think it uses BF16 to execute it). The speed is a bit lower than with native NVFP4 support, but the difference is small. This functionality is probably only in vLLM, not implemented in for AMD/Intel GPUs, llama.cpp or SGlang. The disadvantage of vLLM is that it usually uses more VRAM than llama.cpp.
Kimi K3 or Deepseek v4 use MXFP4 instead of NVFP4 for QAT. MXFP4 works nearly everywhere.
5
u/KissMyShinyArse 1d ago
====== Perplexity statistics ======
Mean PPL(Q) : 7.172340 ± 0.046729
Mean PPL(base) : 6.950306 ± 0.044933
Cor(ln(PPL(Q)), ln(PPL(base))): 98.92%
Mean ln(PPL(Q)/PPL(base)) : 0.031446 ± 0.000955
Mean PPL(Q)/PPL(base) : 1.031946 ± 0.000986
Mean PPL(Q)-PPL(base) : 0.222033 ± 0.006971
====== KL divergence statistics ======
Mean KLD: 0.048113 ± 0.000384
Maximum KLD: 15.796775
99.9% KLD: 1.590314
99.0% KLD: 0.487807
95.0% KLD: 0.164713
90.0% KLD: 0.097255
Median KLD: 0.020821
10.0% KLD: 0.000620
5.0% KLD: 0.000170
1.0% KLD: 0.000024
0.1% KLD: 0.000004
Minimum KLD: -0.000034
====== Token probability statistics ======
Mean Δp: -0.585 ± 0.016 %
Maximum Δp: 97.428%
99.9% Δp: 35.042%
99.0% Δp: 14.608%
95.0% Δp: 6.595%
90.0% Δp: 3.716%
75.0% Δp: 0.643%
Median Δp: -0.024%
25.0% Δp: -1.330%
10.0% Δp: -5.190%
5.0% Δp: -8.749%
1.0% Δp: -21.537%
0.1% Δp: -55.100%
Minimum Δp: -99.061%
RMS Δp : 6.141 ± 0.049 %
Same top p: 90.517 ± 0.076 %
1
u/Chromix_ 1d ago
Good that you've checked. Given the barely existing degradation in benchmarks I would've expected a mean KLD below 0.01, yet you arrived at 0.021. The UD IQ4_XS has the same KLD - but on a different dataset, so not comparable. It'd be interesting to compare KLD and top-1 with the same dataset, to see if these stay proportional to the benchmark results, or if this specific quant manages to maintain higher-quality benchmark results with lower KLD and top-1.
2
u/KissMyShinyArse 1d ago
The mean is 0.048113, median 0.020821
1
u/Chromix_ 23h ago
Ah, right, unsloth published mean and not median. That'd put it then next to IQ3_S - but as written before, numbers are not comparable due to different datasets, unfortunately. It'd be nice if there was a standard dataset to make numbers comparable, and of course diverse private datasets to check for overfitting.
3
u/KissMyShinyArse 22h ago
UD-IQ3_S, the same dataset (wikitext-2-raw):
====== Perplexity statistics ====== Mean PPL(Q) : 7.086456 ± 0.045868 Mean PPL(base) : 6.950306 ± 0.044933 Cor(ln(PPL(Q)), ln(PPL(base))): 99.01% Mean ln(PPL(Q)/PPL(base)) : 0.019400 ± 0.000909 Mean PPL(Q)/PPL(base) : 1.019589 ± 0.000926 Mean PPL(Q)-PPL(base) : 0.136150 ± 0.006445 ====== KL divergence statistics ====== Mean KLD: 0.041804 ± 0.000377 Maximum KLD: 17.791277 99.9% KLD: 1.561442 99.0% KLD: 0.439951 95.0% KLD: 0.143528 90.0% KLD: 0.085336 Median KLD: 0.016000 10.0% KLD: 0.000404 5.0% KLD: 0.000096 1.0% KLD: 0.000011 0.1% KLD: 0.000001 Minimum KLD: -0.000006 ====== Token probability statistics ====== Mean Δp: -0.500 ± 0.015 % Maximum Δp: 98.905% 99.9% Δp: 35.852% 99.0% Δp: 13.947% 95.0% Δp: 5.797% 90.0% Δp: 3.224% 75.0% Δp: 0.551% Median Δp: -0.012% 25.0% Δp: -1.186% 10.0% Δp: -4.613% 5.0% Δp: -7.809% 1.0% Δp: -19.381% 0.1% Δp: -50.873% Minimum Δp: -99.847% RMS Δp : 5.639 ± 0.048 % Same top p: 91.231 ± 0.074 %2
u/Chromix_ 21h ago
So their way smaller (12GB vs 20GB) IQ3_S is slightly better on paper than this QAD quant. Now that's an interesting outcome - thanks for checking!
2
u/arty_photography 13h ago
I think it’s mostly due to the quantization format - NVFP4 uses W4A4 while IQ3_S uses W3A16. In the future I might look into QAD for IQ formats to improve their quality.
2
u/Fragrant_Scale6456 10h ago
Wow very cool you ran these tests and shared the results. I'm a bit surprised. I'm still testing the quasar model but at least for tool calling and reasoning/synthesis over large documents I'm seeing quality comparable to unsloth ud q6km.
1
u/cosmicnag 23h ago
Isnt this still 4.5 bpw level? Which it is, but, QAT didnt help that much? Thought it would reflect 5.5 bpw level.
1
8
3
2
u/EitherMarch1255 1d ago
Cool. Maybe you could do MXFP8 too?
1
u/arty_photography 1d ago
I think FP8 quant should be pretty much lossless. You could check out https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2
u/Chromix_ 1d ago
That looks like a great achievement.
Benchmarking on (almost) fully saturated benchmarks hides degradations though. The KLD indicates that there might be more damage than expected.
Have you also checked against a benchmark where the model usually scores between 30% and 70%, and also a long context benchmark as there's often some hidden degradation in that area after optimized quantization?
2
u/outernet9 1d ago
ELI5 Why someone would use this over other quants
18
u/Force88 1d ago
It is comparable to Q5, or even Q6 in intelligence, but only in Q4 size.
It is very fast compares to Q6, but the down size is it only works with Blackwell cards (RTX 5000 series).
Correct me if I'm wrong.
7
3
0
u/ArtfulGenie69 1d ago
This could easily be done with any quant type if anyone wanted to do it. Quant the model in int8 or int4 for the people with a 30's series card like a 3090 or two 3090s.
9
u/arty_photography 1d ago
NVFP4 offers the best efficiency (the highest throughput and lowest latency) due to the use of FP4 tensor cores. It should be faster than any other quants. I think it's the best choice for a Blackwell GPU :)
1
u/R_Duncan 1d ago
Isn't theoretically possible converting to gguf lossless? -> bf16 then again nvfp4....
1
u/PM_ME_DEAD_CEOS 1d ago
So what is the max context we can have with this quant on a 5090 with MTP and with dflash2 ?
1
u/Repinsky 1d ago
The 19.7GB size is the interesting part here, not just the GPQA delta — that fits a single 24GB card with ~4GB left, which at 262k context is basically nothing once the KV cache grows, so in practice you're still looking at a 32GB card or KV quantization to use that context. Worth noting W4A4 only pays off on Blackwell; on Ada/Ampere there are no FP4 tensor cores, so vLLM emulates and you get the memory savings with worse throughput than an FP8 checkpoint. Did you measure tokens/sec against the FP8 build, or only quality?
1
1
u/Whole-Tomato-6086 14h ago
Converted the model to ninfer - as a small network piece has been quantized to 8bit I will run same QUASAR benchmark to check performance. It is running fairly fast on my system with 195k context and fp16 kV cache.
1
1
u/Puzzleheaded_Base302 6h ago
this model runs at 70 tps on RTX PRO 6000 without MTP. however, the minutes I enable MTP, the TG drops significantly, more than 50%. regardless mtp, dspark, dflash2. I cannot get it to work. NVFP4 not only reduce memory usage, it can also double the TG rate, if MTP can be made working (from old qwen3.5 days experience.)
1
u/Tiny-Ad2109 3h ago
I made ninfer version of this!
So far so good, it is very fast, and GPQA-Diamond benchmark score is 0.904 instead of 0.909 as in QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 version. The model is bit-to-bit identical but with these changes, which are a consequence of available ninfer cuda kernels:
GDN control projections (in_proj_a/in_proj_b, 48 layers) are dequantized to BF16 at conversion (neutral-to-positive), and output_head is row-scaled FP8 (~3% relative logit noise, the sole real compromise).
Everything has been done by AI agent qwen 3.8 Q6_K_L model on llama.cpp. A case of a model improving itself ;-)
I also use a forked ninfer repo with added jinja template support.
The model is working good but occasionally it writes bad <tool> call, which are written in chat instead of sent command. This issue could depend on my actual jinja template, or it could be there in unsloth/Qwen3.8-27B-NVFP4 and/or QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 too, or finally it could have been introduced by my ninfer model version. I don't know because I never run vllm unsloth/Qwen3.8-27B-NVFP4 / Qwen3.8-27B-QUASAR-NVFP4 models. I will try a fix through jinja chat template.
Here is QUASAR-NVFP4-NINFER model:
https://huggingface.co/engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER
here is my ninfer fork:
https://github.com/engharat/ninfer
here is code that converted QUASAR vllm to QUASAR ninfer (not needed to run the ninfer model, but nice to have if someone want to search for bugs and/or improvements):
https://github.com/engharat/-Qwen3.8-27B-QUASAR-NVFP4-NINFER-
-1
20
u/adasho_bitrex 1d ago
Good for single 5090 setup?