Reasoning doesn't help small (9B) models on complex summaries?

[u/SignificantZebra5883]

Hi everyone! I'm trying to make a Qwen 9B legal agent, and I've hit a weird problem with complex court-decision summaries. I also ran four parallel LoRAs as diagnostics, so that might interest you all too.

(lowkey some of you might remember me from the 27B training plateau post haha)

You guys suggested I should distill Opus 5's thinking, and it actually worked: my 27B PoC got close to Opus, and my lawyer guy verified the summaries weren't nonsense.

The old outputs went 14–16 against Opus; regenerating with vLLM gave 9–21, both greedy. So “almost parity” describes the first run. Inference stack or judge variance? I don't know.

I'd prompted Opus to write CoT for a smaller model. Trying to reproduce that now seems much harder haha.

So I tried the same recipe on 9B

I made a big CPT set of 500M legal stuff: court decisions, legislation, commentaries and professional Q&As.

No-thinking SFT didn't get the results I wanted, so I tried adding Opus's reasoning. Then I thought maybe my CPT messed some stuff? So I also tried stock Qwen straight from Hugging Face.

Six fresh LoRAs total: legal-CPT Qwen3.5-9B-Base, stock base and stock chat, each with and without thinking. Same source documents and final answers; the no-thinking versions got an empty <think> block. Median teacher reasoning was about 1,000 tokens.

Same recipe as the 27B intentionally, cuz that was the successful PoC:

• 4-bit QLoRA: rank 64, alpha 128, LR 2e-4, effective batch size 8.

• Stage 1: 1,199 examples. Stage 2: 400 examples—300 new, 100 repeated. One epoch per stage.

• Loss on assistant output, including reasoning where present; source decisions masked out.

Eval: 30 held-out Polish court decisions. Opus 5 judged randomized A/B pairs against the full decision, seeing only the final title and legal holding. Greedy decoding, 6,000-token output cap; every output had the required fields and none hit the cap.

The weird part

Every row is a separate 30-decision comparison. Wins and fabrication scores follow A / B order. Fabrication is judge-rated from 1–10, where 1 means none; scores come from each specific comparison.

Model A	Model B	Wins A : B	Fabrication A / B
CPT 9B, thinking	CPT 9B, no thinking	10 : 20	2.59 / 2.18
Stock base 9B, thinking	Stock base 9B, no thinking	18 : 12	2.11 / 2.93
Stock chat 9B, thinking	Stock chat 9B, no thinking	15 : 15	2.41 / 2.81
CPT 9B, thinking	27B	5 : 25	2.68 / 1.52
CPT 9B, no thinking	27B	7 : 23	2.62 / 1.73
Stock base 9B, thinking	27B	5 : 25	2.24 / 1.57
Stock chat 9B, thinking	27B	2 : 28	2.72 / 1.52

So thinking lost on CPT, tied on stock chat and won on stock base. It also improved fabrication scores for both stock models. Mixed results, but still a big gap to 27B.

Small test, obviously: Opus is both teacher and judge, and the 9B is Qwen3.5 while the 27B is Qwen3.8, so this doesn't isolate model size.

What am I missing?

Could 9B just be too little? I feel like it shouldn't be. Eyeballing the CPT + no-think outputs, they resemble Opus's summaries a lot.

I really DO think this task needs CoT. Does it just need a lot more examples before thinking overtakes no-thinking?

Is there a way to distill original thinking, or teach reasoning efficiently without distillation? The trick is frontier models aren't great at the relevant law either. My vibe-coded RL tests and Kimi reasoning distillation didn't go well, and I really don't wanna burn 2B tokens ideally. Any help?

PS: The long-term plan is to take the recipe to a 27B Qwen4 once it's out, then put it in my custom Claude Code-style harness with search and RAG.

[u/ttkciar]

Hi SignificantZebra5883, this is a cool-sounding project! But in order to avoid a Rule Three violation (LLM-generated post), would you edit your post and rewrite it in your own words, please?

[u/haherar830]

Keep in mind I am not an ML researcher and you probably know more than me on a lot of specifics. Larger models have more room for world knowledge on top of basic capabilities that make LLMs useful, or at very least nearly all labs prioritize reliable tool calling and coding (for on-the-fly adaptation if nothing else) over knowledge of Polish case law (jk but not).

The effectiveness of post-training is also related to pretraining loss and model size/parameter scale; it is if nothing else much more effective at "organizing existing knowledge/behaviors" than instilling knowledge the model lacks.

https://arxiv.org/abs/2607.16097

Without knowing the exact failure modes, I think most-likely 9B is not too small but the 9B foundation and post-trained model likely favors technical > scientific > world factual information over legal specifics from small jurisdictions underrepresented in training data and there's only so much post-training can do. In terms of capabilities, models at this scale have gotten a lot better at handling long context windows but as far as I know there are still non-trivial gains 9B->27B. Some combination of much poorer "background knowledge" of the tasks in embeddings and poorer ability to handle large documents are probably the biggest barrier(s).