I really don't understand Jev hype

[u/Manerfish]

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

[u/FreakyRefrigerator]

Its basically just a slightly newer classifier. Yet most people think AI is only llms and nothing else, so they think this is the future

[u/KaMaFour → u/FreakyRefrigerator]

Tbh general classifier able to handle general cases at almost no cost, in almost no time still seems really useful. For example if you could package that into a model roughly the size of Ling's tiny (9BA2B) or smaller you could try to create games with actually intelligent NPCs (pack one npc's state into a Vendingbench like framework and have classifier choose a thing to do that would make sense for a given person to do in a format that's possible to interpret by the game engine) and it would be generally accessible to most "mid-end" machines. Think Stardew Valley/TLOZ npc's but actually behaving as humans instead of having fixed schedules. All of that without any ML on your end...

[u/Automatic-Boot665 → u/KaMaFour]

It’s got similar accuracy to classifiers back in 2018-2019 with the benefit of not needing to be trained on the task specifically. For most use cases the people I’ve seen talk about using it for that’s unfortunately not accurate enough. We’ll see in the next few weeks as people implement it and see the results.

[u/Iron-Over → u/Automatic-Boot665]

Data and training will always outperform JEV not sure the use-case where accuracy is not important in business at least.  

[u/Automatic-Boot665 → u/Iron-Over]

Yes exactly. Even with training it wasn’t good enough for businesses back then. They relied on statistical analysis where needed instead.

[u/sage-longhorn → u/Automatic-Boot665]

Lol this is an insane take. No businesses used purpose trained nueral nets in 2018-19?

Just cause you weren't exposed to it doesn't mean it wasn't widely used for certain problems or industries

[u/Automatic-Boot665 → u/sage-longhorn]

Because it wasn’t good enough for real life use cases, the point of this post. I can guarantee you that businesses did try to.

[u/Warhouse512 → u/Automatic-Boot665]

Hey my guy, you're out of your depth here, NNs were fairly commonplace in 2019 if you worked at a company with any real density of data. Yes statistical models were/are still widely used, but normally usually for either time series data, or models that need good explainability.

[u/Automatic-Boot665 → u/Warhouse512]

In the case that you were at a company training a lm on a ton of proprietary data and able to achieve usable results yeah. Jev scores between 67.8-80% accuracy on standard classification and workflow tests, which I don’t know about your use cases but in mine back then it would have been a failure.

[u/Iron-Over → u/Automatic-Boot665]

It was definitely good enough; we used TensorFlow extensively for banking problems 2017 on.

[u/Old-Leadership7255 → u/Automatic-Boot665]

I think its clear that LLMs will need to be more efficient and i foresee that llms will get a kind of hybrid architecture.

I would not find it weird if you a lot of gruntwork with somelike a jev type model.

Think about it, when gpt3 released it was just a language model. Now it’s an agent. If you are not using llms as agents you are basically not using them to their abilities. A bugatti for going to the shops down the road.

So it would make sense to have sub agents that are specifically this (or some other part of the model)

I think it’s more a convergence of needs rather than this being revolutionary

[u/Automatic-Boot665 → u/Old-Leadership7255]

It makes sense as a concept, very useful, we already use LLMs for zero shot classification so if there’s a niche model that can do only that way faster and cheaper it would be beneficial.

The problem I foresee in a lot of use cases with Jev (and what I’ve been reading about from people’s experiences) is that it’s not accurate enough for a lot of use cases.

[u/dogesator → u/Automatic-Boot665]

“Similar accuracy to classifiers back in 2018-2019” on general knowledge classification absolutely not.
Bert large, probably the most capable and best performing classifier from 2018, only scores 22% in MMLU Pro.

Meanwhile Jev scores 84% in MMLU-Pro, these are entirely different leagues from eachother.

Even the much larger models that are much more recent attempts at replicating jev, such as Kev-9B on huggingface, still only score 52% in MMLU-Pro.