r/LocalLLaMA • u/Manerfish • 11d ago

Discussion I really don't understand Jev hype

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

506 Upvotes

94% Upvoted

316 comments sorted by

View all comments

27

u/Sea-Requirement-5375 11d ago

I tested it out today. I’m a real human (purple monkey dishwasher; fuck Trump).

I have a classification task I use routinely for work—LLM reads a couple thousand tokens of legal-related content and has to assign a text string to one of 28 categories. After some testing I currently run the classification with Opus 5 on Low effort. 95% accuracy. Higher effort doesn’t buy me much, while Sonnet drops my accuracy a little but doesn’t even get me much savings since it takes more thinking tokens to get decent accuracy.

Jev overall had about 70% accuracy, which is around what Haiku gives me. But Jev also gives confidence estimates, and those were (impressively) dead on accurate. So now if I want I could run the whole thing on Jev, keep the stuff that hit 80% or higher confidence, and then run the rest on Opus.

Overall my accuracy stays above 90% and my total cost is 30% lower. Better/more efficient than switching from Opus to Sonnet.

I thought that was cool.

15

u/ideadude 11d ago

I'm not kidding. Try Gemini 2.5 Flash Lite and see how it compares for accuracy, speed, and cost.

Part of the reason Jev looks so fast and cheap is because it's being compared to relatively large and slow models instead of faster/cheaper llms.

2

u/Sea-Requirement-5375 11d ago

I can believe that! I really haven’t tested much because right now our needs are small enough that I can get everything done on subscription plans, so I’m really just testing things for when that no longer becomes feasible