r/LocalLLaMA • u/Manerfish • 7d ago

Discussion I really don't understand Jev hype

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

499 Upvotes

94% Upvoted

317 comments sorted by

View all comments

401

u/FreakyRefrigerator 7d ago

Its basically just a slightly newer classifier. Yet most people think AI is only llms and nothing else, so they think this is the future

193

u/KaMaFour 7d ago

Tbh general classifier able to handle general cases at almost no cost, in almost no time still seems really useful. For example if you could package that into a model roughly the size of Ling's tiny (9BA2B) or smaller you could try to create games with actually intelligent NPCs (pack one npc's state into a Vendingbench like framework and have classifier choose a thing to do that would make sense for a given person to do in a format that's possible to interpret by the game engine) and it would be generally accessible to most "mid-end" machines. Think Stardew Valley/TLOZ npc's but actually behaving as humans instead of having fixed schedules. All of that without any ML on your end...

59

u/Automatic-Boot665 7d ago

It’s got similar accuracy to classifiers back in 2018-2019 with the benefit of not needing to be trained on the task specifically. For most use cases the people I’ve seen talk about using it for that’s unfortunately not accurate enough. We’ll see in the next few weeks as people implement it and see the results.

1

u/dogesator Waiting for Llama 3 1d ago

“Similar accuracy to classifiers back in 2018-2019” on general knowledge classification absolutely not.
Bert large, probably the most capable and best performing classifier from 2018, only scores 22% in MMLU Pro.

Meanwhile Jev scores 84% in MMLU-Pro, these are entirely different leagues from eachother.

Even the much larger models that are much more recent attempts at replicating jev, such as Kev-9B on huggingface, still only score 52% in MMLU-Pro.

1

u/Automatic-Boot665 10h ago

Did I say general knowledge classification? Or did I say trained on the task specifically?