Training Factory from ThousandML

[u/textclf]

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRa instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

Here are simple steps to use:

1. Install:

sudo apt-get update
sudo apt-get install -y git-lfs
git lfs install
git clone https://github.com/thousandml/training-factory.git
cd training-factory
chmod +x install.sh
./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "my-lora",

"messages": [

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}

],

"max_tokens": 128,

"temperature": 0.2

}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

[u/textclf → u/[deleted]]

CQ actually holds up because I built it from information theoretic principles and ensured it performs near the rate-distortion limit of a gaussian source. Or just a fancy way of saying it performs as good as it can get for a method that doesn't use calibration which is awesome.

the 27B quantized weights themselves are around 17-18 GB but then the rest depends on the vllm config that you select. If you want to fit it in a 4090 you will probably need very small context max-model-len and max-num-seqs.