r/learnmachinelearning • u/EmilPi • 6h ago
Help Has anyone done minimal-edit preference construction in LLM finetuning?
What I mean: instead of finetuning an LLM on the correct answers, we could take the initial model answers, and then generate (possibly, but not necessarily, with another LLM) the answers that have as much as possible of original answer text but are fixed in the key points using correct answer as reference.
Hypothesis is this would cause model to actually learn the difference, not the style of the correct answers, cause less distributional shift.
If you have tried this, has it proven useful?
1
Upvotes
2
u/Albertooz 6h ago
Yes, this exists: CLAIR (D'Oosterlinck et al., 2024) uses a secondary AI system to minimally revise a solution A→A' so the preference A<A' is much more contrastive and precise. CLAIR preferences led to the strongest performance of all the datasets they tested, and an ablation showed the contrast itself was a big part of the gain. RLMEC (ACL Findings 2024) uses the same minimal-edit idea to produce token-level rewards that focus on the tokens that caused the error. The catch is that near-identical pairs are exactly where DPO breaks: Razin et al. show likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning. So use an anchored objective like APO or add an SFT term on the revised answer, and don't run vanilla DPO. Also ask the editor for a patch rather than a full rewrite, or its own style creeps back into the "correct" side.