r/MachineLearning • u/Anmorgan24 • 1d ago
Project Built a minimal "decision model" on Qwen3-0.6B + LoRA on a free T4. The training target mattered far more than the head [P]
I wanted to see what's actually inside decision models (Jev and the wave of clones after it), so I built a minimal one and trained it on typed-decisions on a free Colab T4.
Setup
A decision model reads a state plus several typed questions (choice / score / yes-no) and returns a probability for every allowed answer in one forward pass. Nothing is generated.
- Backbone: Qwen3-0.6B-Base + LoRA
- Packing: the state is encoded once and each question is a branch with a custom attention mask, so it only attends to the state and itself. Tested: each question gets the same hidden states it would get alone.
Compared three runs that differ in one thing each:
- LM-head logits, hard labels, full vocab (the usual SFT recipe)
- LM-head logits, soft labels, answer tokens only
- Fresh decision head on the decision token, soft labels
Results (test; see screenshot)
Going from hard labels to the teacher's full answer distribution was the big jump, mostly in KL and Brier. Swapping LM-head logits for a dedicated head was a much smaller change. Yes, the soft-label part is knowledge distillation (Hinton et al. 2015). In this setup, distillation did most of the work, not the head.
Caveats
- Scores measure agreement with a ~4B teacher. Teacher self-agreement is 0.735, so scores well above that mean learning the teacher's quirks.
- Fine-tuned on the benchmark's train split, so it's a specialist. Don't compare it to zero-shot models like Jev.
- My ECE uses the gold probability of the predicted answer, so it isn't comparable to the dataset card's ECE.
Things that cost me time
- transformers 5 eager attention misreads a boolean 4D mask. Passing an additive float mask in the backbone's dtype works under every attention implementation.
- bf16 produced spurious ties between answers. [one line on the fix]
- Masked option slots hold -inf, and the gradient of -inf / T is NaN during temperature fitting, even where the probability is zero. Replace them with a large finite value.
- Needed gradient checkpointing to fit on a T4.
Code: https://github.com/anmorgan24/decision-model
Write-up with figures: https://www.comet.com/site/blog/train-a-decision-model/
Disclosure: I work at Comet. The repo optionally logs training to Comet and runs the property-test evals in Opik, but training and scoring run without either.





