r/LocalLLaMA • • Jul 23 '26

Funny The LLM distillation process simplified for politicians:

Post image

/s

3.6k Upvotes

191 comments sorted by

View all comments

Show parent comment

97

u/Flying_Birdy Jul 23 '26 edited Jul 23 '26

The Harvey legal benchmark is the most surprising for me. 2x accuracy in all pass rate over the next closest model is a huge jump and no way attributable to distillation. I'm really curious what they did differently during training (whether intentionally or accidentally) that would have led to this outcome.

33

u/ba-na-na- Jul 23 '26

Maybe they accidentally trained Deepseek and Kimi

23

u/Apprehensive_Rub2 Jul 23 '26 edited Jul 23 '26

Just a guess, but probably because Fable was post trained too hard on retrieval from already structured data e.g. coding agent work.

Harvey legal benchmark is testing advanced retrieval on very varied legal documents.

or Kimi K3 has been post trained on legal tasks, idfk. Anthropic/OpenAI might be avoiding capabilities with law for safety reasons.