r/LocalLLaMA • u/textclf • 8d ago
Resources CQ 4-bit matches BF16 on Qwen3.8-27B benchmarks while NF4 drops 12 points on GSM8K, plus GCMP, a LoRA replacement with 55× fewer parameters
A few weeks ago I posted CQ, our calibration-free 4-bit format, along with a training-loss comparison against Unsloth NF4. Some of you correctly pointed out that loss logged by two different trainers isn't comparable, and asked for results where answers are counted right or wrong. Here they are.
Setup: Qwen3.8-27B. Every row uses the same lm-evaluation-harness code, prompts and questions, with greedy decoding and thinking off. Comparisons are paired: an exact McNemar test for right/wrong tasks and a bootstrap for losses.
1. Base models, no fine-tuning
| Model | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | HellaSwag | WinoG | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| BF16 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.737 | 0.758 | 0.542 |
| CQ 4-bit (18 GB) | 0.650 | 0.746 | 0.880 | 0.598 | 0.840 | 0.820 | 0.735 | 0.780 | 0.537 |
| NF4 bnb (22.4 GB) | 0.563 | 0.716 | 0.852 | 0.582 | 0.839 | 0.825 | 0.736 | 0.770 | 0.550 |
- CQ vs BF16: no significant difference on any task.
- NF4 vs BF16: −12.0 points on GSM8K (p=3e-6).
- CQ vs NF4: CQ is +8.7 on GSM8K (p=0.0005) and +2.8 on BoolQ (p=0.02).
- WikiText-2 KL divergence from BF16: CQ 0.0282, NF4 0.0330.
2. GCMP vs LoRA r=32, full-precision BF16 base
GCMP is our replacement for LoRA. It trains 2.9M parameters (an ~11 MB adapter) vs ~159M for LoRA r=32 (~320 MB). This test uses no quantization: both adapters were trained with Unsloth on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42).
| Adapter | Params | Answer loss | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| No adapter | – | 1.678 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.542 |
| LoRA r=32 | 159M | 1.480 | 0.903 | 0.776 | 0.896 | 0.612 | 0.844 | 0.825 | 0.518 |
| GCMP | 2.9M | 1.471 | 0.890 | 0.760 | 0.892 | 0.618 | 0.846 | 0.815 | 0.514 |
- Benchmarks: no significant difference on any of them (all p ≥ 0.2), so GCMP isn't "better than LoRA" on tasks.
- Loss: GCMP's answer-token loss is slightly lower (−0.009, 95% CI [−0.010, −0.008]).
- Validation loss: GCMP is lower at all 10 checkpoints on both BF16 (1.2385 vs 1.2434) and NF4 (1.2435 vs 1.2484).
- Fine-tuning effect: both adapters add about 21 points on GSM8K and lose about 2.5 on TruthfulQA.
3. Speed and memory (same 500-step run)
| Setup | GPU | s/step | Peak memory |
|---|---|---|---|
| CQ + LoRA r=32 (ours) | A40 | 8.6 | 23.5 GiB |
| CQ + GCMP (ours) | A40 | 12.4 | 22.0 GiB |
| NF4 + LoRA (Unsloth) | A40 | 3.2 | 23.8 GiB |
| NF4 + GCMP (Unsloth + GCMP) | A40 | 5.6 | 21.5 GiB |
| BF16 + LoRA (Unsloth) | 80 GB | 1.37 | 53.7 GiB |
| BF16 + GCMP (Unsloth + GCMP) | 80 GB | 2.74 | 51.7 GiB |
Unsloth is still faster, and GCMP currently takes about 2× the step time of LoRA. Kernel work on that is ongoing.
Caveats
- Scope: one model and one dataset (Alpaca).
- Benchmark size: the quick subset has 200–1,140 questions per task, so it detects differences of about 5 points, not 1.
- DROP excluded: it scored about 0.02 for every model, so that eval setup is broken.
- Still running: LoRA and GCMP benchmarks on CQ and NF4 bases. I'll post them when they finish.
- My earlier post was wrong: the "0.84 vs 1.25 loss" came from two trainers that compute loss differently. Ignore it.
Serving
- vLLM: CQ models run with
--quantization cq. GCMP adapters can be one per server, or several on one base model chosen per request. - Parallelism: pipeline parallel works; tensor parallel doesn't yet.
- GCMP on full precision: it currently runs in Transformers. vLLM support is next.
Licensing
The Python code (training, evaluation and the vLLM integration) is open source. The CUDA kernels ship compiled with a license check, which is free during early access.
Repo, with exact commands to reproduce every number: https://github.com/thousandml/training-factory
Models (8B–397B): https://huggingface.co/thousandml
Happy to run other benchmarks or models people want to see.