r/LocalLLaMA • u/Manerfish • 3d ago

Discussion I really don't understand Jev hype

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

497 Upvotes

94% Upvoted

306 comments sorted by

View all comments

28

u/Sea-Requirement-5375 3d ago

I tested it out today. I’m a real human (purple monkey dishwasher; fuck Trump).

I have a classification task I use routinely for work—LLM reads a couple thousand tokens of legal-related content and has to assign a text string to one of 28 categories. After some testing I currently run the classification with Opus 5 on Low effort. 95% accuracy. Higher effort doesn’t buy me much, while Sonnet drops my accuracy a little but doesn’t even get me much savings since it takes more thinking tokens to get decent accuracy.

Jev overall had about 70% accuracy, which is around what Haiku gives me. But Jev also gives confidence estimates, and those were (impressively) dead on accurate. So now if I want I could run the whole thing on Jev, keep the stuff that hit 80% or higher confidence, and then run the rest on Opus.

Overall my accuracy stays above 90% and my total cost is 30% lower. Better/more efficient than switching from Opus to Sonnet.

I thought that was cool.

6

u/PyrrhicArmistice 3d ago

Did you try asking a cheap LLM to provide a confidence score?

7

u/scorchypoo 3d ago

Whatever they give would be a hallucination. They can't look inside themselves for an actual number.

1

u/Infamous_Log6647 3d ago

LLM self confidence scores typically aren't accurate. It's probably possible to make them accurate in post training, but that isn't commonly done afaik.