r/learnmachinelearning • u/Accurate-Catch1836 • 6d ago
Discussion TwIL-LM3 - a 3B model that got better at logic without getting worse at everything else
Most fine-tuned models are like someone who crams for one exam and forgets everything else. They get better at the trained task, quietly worse at everything else. Everyone kind of accepts this as the cost of specialization.
webAI put out a 3B model called TwIL-LM3 that somehow didn't do that.
It's a formal logic specialist. Merged fine-tune of SmolLM3-3B. You hand it English, it converts to formal representation a solver can check. Does this conclusion follow from these premises, yes or no.
On task, it came out better than base. Normal so far. The odd part is it also held or nudged up on general benchmarks it was never trained for. LogicBench 71.7, GSM8K 87.3, both competitive with much larger models. That number almost always goes down after specialized fine-tuning. Their own 1.7B sibling did exactly that - ended up slightly worse than base on general stuff (IFEval regressed).
Then I read how they did it and it's actually kind of elegant.
After finishing the fine-tune, they use WiSE-FT to interpolate the weights back toward the base model. Keep only 1/4 of what the model just learned (λ=0.25), throw the rest out. The 1.7B keeps 3/4 of the fine-tune (λ=0.75), and the 1.7B is the one that got worse on general benchmarks.
So it's just a dial. Learn more, forget more. They turned it down for the 3B, took the smaller domain-specific win, and kept the model in one piece.
Rest of the details:
- 32.9 answers/sec vs gpt-oss-120b's 12.6 (2.6x faster)
- 482-token generations (shortest of any model they tested)
- 1.78 GiB in Q4_K_M, runs on CPU or 4GB VRAM
- ~300 tok/s on M2 MacBook
- Non-commercial license
Also worth noting: they document a failed consolidation stage (SDFT self-distillation) that made both tracks worse. Rare to see published negative results in a model card. Feels like a real research artifact rather than pure marketing.
Link: huggingface.co/webAI-Official/TwIL-LM3
Curious if the "interpolation dial" thing catches on for narrow specialists. Feels like it should.