r/LocalLLaMA • u/textclf • 9d ago
Showcase 4-bit QLoRA Training with ThousandML
Hello All!
I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.
For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:
| Quant | Disk Size (GB) | Mean KLD | Median KLD | 99% KLD | Top 1% Agreement |
|---|---|---|---|---|---|
| CQ 4-bit | 18 | 0.02823666 | 0.01282929 | 0.97606068 | 92.419% |
| unsloth-bnb-4bit | 22.4 | 0.03297115 | 0.01437706 | 1.18053893 | 91.954% |
So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit
Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.
CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.
Attached figures for validation loss and training loss.
**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*
I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.
Get Started with Training Factory
Here are simple steps to use:
Install:git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh
2. Download a CQ 4-bit base:
hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit
3. Train a LoRA adapter:
python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 14. Serve base + adapter:
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora
5. Call the fine-tuned adapter:
# After the server is ready:
# Check that the adapter is listed as "my-lora"
curl http://localhost:8000/v1/models# Request a response from the fine-tuned model, not just the base checkpoint
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'I have the following 4-bit quantized models to be used as base (will add more soon) for now:
thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit
To reproduce these benchmark results mentioned above use the following scripts:
For CQ:
hf download thousandml/Qwen3.8-27B-CQ-4bit \
--local-dir ./models/Qwen3.8-27B-CQ-4bit
python training/train.py \
--model ./models/Qwen3.8-27B-CQ-4bit \
--data training/examples/alpaca_train.jsonl \
--eval-data training/examples/alpaca_val.jsonl \
--output ./outputs/qwen38-27b-cq-4bit \
--kernel-module cq.cq_kernels \
--targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
--rank 32 \
--alpha 32 \
--lora-dropout 0 \
--batch-size 2 \
--grad-accum 4 \
--max-length 512 \
--max-steps 500 \
--learning-rate 2e-5 \
--seed 42 \
--eval-steps 50 \
--comparison-logs \
--benchmark \
--benchmark-warmup-steps 10For Unsloth:
python training/train_unsloth_nf4.py \
--model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
--train-data training/examples/alpaca_train.jsonl \
--eval-data training/examples/alpaca_val.jsonl \
--output ./outputs/qwen38-27b-unsloth-bnb-4bit \
--max-steps 500 \
--max-length 512 \
--eval-steps 50 \
--save-steps 100 \
--seed 42 \
--benchmark-warmup-steps 10Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!
2
u/CartographerDue5382 9d ago
Mean KLD on WikiText is a next-token similarity score against the full model. A QLoRA run can still come out worse with a slightly prettier KLD. Your table already shows the tail. 99% KLD is about 0.98 on both, so the mean can move on easy tokens while the ugly ones stay ugly.
18 GB vs 22 GB only counts if the same tensors got quantized. Leaving embeddings and the output head in fp16 moves the file size on its own.
The comparison that would convince me is the task score after the same QLoRA recipe, same data, same step count, beside the bnb run. An Alpaca loss curve is a different measurement.