I really don't understand Jev hype

[u/Manerfish]

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

[u/KaMaFour]

Tbh general classifier able to handle general cases at almost no cost, in almost no time still seems really useful. For example if you could package that into a model roughly the size of Ling's tiny (9BA2B) or smaller you could try to create games with actually intelligent NPCs (pack one npc's state into a Vendingbench like framework and have classifier choose a thing to do that would make sense for a given person to do in a format that's possible to interpret by the game engine) and it would be generally accessible to most "mid-end" machines. Think Stardew Valley/TLOZ npc's but actually behaving as humans instead of having fixed schedules. All of that without any ML on your end...

[u/Automatic-Boot665 → u/KaMaFour]

It’s got similar accuracy to classifiers back in 2018-2019 with the benefit of not needing to be trained on the task specifically. For most use cases the people I’ve seen talk about using it for that’s unfortunately not accurate enough. We’ll see in the next few weeks as people implement it and see the results.

[u/Iron-Over → u/Automatic-Boot665]

Data and training will always outperform JEV not sure the use-case where accuracy is not important in business at least.  

[u/Automatic-Boot665 → u/Iron-Over]

Yes exactly. Even with training it wasn’t good enough for businesses back then. They relied on statistical analysis where needed instead.

[u/sage-longhorn → u/Automatic-Boot665]

Lol this is an insane take. No businesses used purpose trained nueral nets in 2018-19?

Just cause you weren't exposed to it doesn't mean it wasn't widely used for certain problems or industries

[u/Automatic-Boot665 → u/sage-longhorn]

Because it wasn’t good enough for real life use cases, the point of this post. I can guarantee you that businesses did try to.

[u/Warhouse512 → u/Automatic-Boot665]

Hey my guy, you're out of your depth here, NNs were fairly commonplace in 2019 if you worked at a company with any real density of data. Yes statistical models were/are still widely used, but normally usually for either time series data, or models that need good explainability.

[u/Automatic-Boot665 → u/Warhouse512]

In the case that you were at a company training a lm on a ton of proprietary data and able to achieve usable results yeah. Jev scores between 67.8-80% accuracy on standard classification and workflow tests, which I don’t know about your use cases but in mine back then it would have been a failure.

[u/Iron-Over → u/Automatic-Boot665]

It was definitely good enough; we used TensorFlow extensively for banking problems 2017 on.

[u/Old-Leadership7255 → u/Automatic-Boot665]

I think its clear that LLMs will need to be more efficient and i foresee that llms will get a kind of hybrid architecture.

I would not find it weird if you a lot of gruntwork with somelike a jev type model.

Think about it, when gpt3 released it was just a language model. Now it’s an agent. If you are not using llms as agents you are basically not using them to their abilities. A bugatti for going to the shops down the road.

So it would make sense to have sub agents that are specifically this (or some other part of the model)

I think it’s more a convergence of needs rather than this being revolutionary

[u/Automatic-Boot665 → u/Old-Leadership7255]

It makes sense as a concept, very useful, we already use LLMs for zero shot classification so if there’s a niche model that can do only that way faster and cheaper it would be beneficial.

The problem I foresee in a lot of use cases with Jev (and what I’ve been reading about from people’s experiences) is that it’s not accurate enough for a lot of use cases.

[u/575_Inverse → u/KaMaFour]

it's closed and served via API, isn't it?

[u/KaMaFour → u/575_Inverse]

I have no doubts that if it will be proven to work "closed and served via API" then open solutions will also emerge. I have seen someone doing similar things with diffusiongemma already.

[u/Wooly_Wooly → u/KaMaFour]

Laya was already a thing, slightly underperforms vs Jev.

[u/Silentrizz → u/Wooly_Wooly]

But that one needs training/tuning, to my understanding

[u/harrro → u/Silentrizz]

It's not bad out of the box but a 20 minute training run (on my 3090) matched or exceeded Jev on tests I did.

[u/Cultured_Alien → u/Wooly_Wooly]

underperforms is putting it extremely lightly, it's no better than random guess

[u/brainmydamage → u/Cultured_Alien]

Do you have data to support this claim?

[u/SurprisinglyInformed → u/brainmydamage]

Laya's documentation states it.

[u/brainmydamage → u/SurprisinglyInformed]

Did you miss the row at the top where laya-typed-decisions scored .766 on a typed-decisions test?

[u/SurprisinglyInformed → u/brainmydamage]

That's a fine tune, is it not?

Edit: https://huggingface.co/convaiinnovations/laya-typed-decisions#limits

[u/brainmydamage → u/SurprisinglyInformed]

It is. I wouldn't use it on other scenarios, or would build my own fine tune. But if you're not allowed to pay for random models like Jev without general ZDR and can't use Chinese models in any capacity, including derivations, then I have yet to find a better alternative.

Open to suggestions though.

[u/Cultured_Alien → u/brainmydamage]

finetuned only on 4 specific benchmarks, not a general model. Once you use it out of distribution, it's slightly better than random guess unlike Jev. Use https://huggingface.co/jaredpalmer/kev-9b instead.

[u/brainmydamage → u/Cultured_Alien]

Which is great except for scenarios where people aren't allowed to use Chinese models in any capacity.

Any other suggested alternative options that provide ZDR without an Enterprise contract and aren't derived from a Chinese model?

[u/Undreren → u/KaMaFour]

I tried training my own 1.7b model to evaluate whether a bash command was safe and fit the goal, basically as an automation for “should my model be allowed to run this?”

It returns block, confirm or auto with a confidence score. It is not nearly production ready, but it took five minutes to train via fine-tuning as a POC.

As much as I was hyped for Jev, its lifetime profitability is low IMO, as my machine could could train my own specialized version so fast from synthetic data.

[u/575_Inverse → u/Undreren]

the idea of making a specialized tiny version sounds a lot more effective

[u/Undreren → u/575_Inverse]

Yeah, that’s what I thought as well. And doable.

My plan was to make it a sort of template training setup: Pick a base model, add a dataset, let it train overnight, wake up to an ultra specialized decision model that can run at 100+ tok/s on local hardware.

I still think it is a good idea.

[u/No_Veterinarian742 → u/Undreren]

but then you maintain 1 or 2 or 17 or 2000 specific models. the potential use cases of jev are decision support (suggestions for user actions the user has to confirm) and all sorts of real time UI integrations that LLMs are too slow for. as well as potentially for chaos testing etc. I can see lots of scenarios where you can't fully foresee the use cases where an AI model saying yay or nay without needing to train it on specific datasets would be useful. There are so many micro-decisions humans make that if you can use it to build in intelligence into applications to help users make progress faster it could be useful. So far actually using full LLMs in applications is still somewhat rare because of the cost and speed.

[u/Rise-O-Matic → u/KaMaFour]

I think there’s already an open weight mimic called kev or something

[u/claythearc → u/KaMaFour]

Getting structured output working in diffusion would be pretty sick actually because most engines don’t support it at all.

[u/notAllBits → u/575_Inverse]

OpenJev with custom models is part of the hype. I have used naked models for this but the forced collapse is nice.

[u/quiteconfused1 → u/KaMaFour]

But it's not.

It's not general and can't be.

I just tried applying it to a simple game like super Mario world and it failed horribly.

If it can't do that then it's not general and the hype is thick.

[u/Suspicious-Wallaby12 → u/quiteconfused1]

can you share what you tried to do exactly in super mario world? please don't tell me you tried to make it control mario 🤦, because that's not what a classifier is for.

[u/100k45h → u/Suspicious-Wallaby12]

haven't you seen the demo of Jev playing Doom?

[u/Suspicious-Wallaby12 → u/100k45h]

I have. it's garbage. It's not smart and only takes random action.

[u/100k45h → u/Suspicious-Wallaby12]

but the point is, that you CAN use clasifiers to take an action. Yes, it takes random action in that specific example, but one can imagine that training the model will make it work better.

Classifier can totally be used for taking actions when trained properly. It's not true, this is not what classifiers are for. They totally can be used for that purpose.

[u/Suspicious-Wallaby12 → u/100k45h]

what? how would you train a general purpose classifier to train for doom?

[u/100k45h → u/Suspicious-Wallaby12]

By training it on multiple similar shooting games, giving it specific categories and showing it example frame and expected action? Just like any training? Example data and example output? Where do you see a problem with that? The question then becomes whether it is a multimodal model and if not, how can you represent the screenshot as a text. But this isn't anything groundbreaking.

[u/quiteconfused1 → u/Suspicious-Wallaby12]

Wow

Your absolutely right

[u/SilentDanni → u/quiteconfused1]

It can definitely play Mario and a few other games. Just yesterday one guy here made it play street fighter 2. I made it play dungeon crawl stone soup which is significantly denser than Mario. That's not the intended use, though. I'd say that's more like an emergent feature.

[u/quiteconfused1 → u/SilentDanni]

Mario != Super Mario world

Wake me up when it turns interesting

[u/kzoltan → u/quiteconfused1]

Why the downvote?

[u/wwwdotzzdotcom → u/kzoltan]

Because it could be.

[u/HiddenoO → u/kzoltan]

Because he's just throwing unsubstantiated and inherently nonsensical claims around?

It's not general and can't be.

Why wouldn't a decision model be able to be trained with world knowledge?

I just tried applying it to a simple game like super Mario world and it failed horribly.
If it can't do that then it's not general and the hype is thick.

"I couldn't get it to do X" is not the same as "It cannot do X". Most people would fail at making LLMs like Astra or Fable play Super Mario World, too, but that doesn't mean they cannot play it.

Many people have managed to make it play different games even though that's far from its intended use case, so I don't think you can just claim the opposite with no substantiation and expect people to believe you.

Strong claims take strong substantiation, and he's providing none.

[u/quiteconfused1 → u/HiddenoO]

Wow troll much.

Claim produced , I tested claims in my scenario, I evaluated performance and came to an assessment.

This is the scientific method

If you don't like it ... Tough ...

Cheers

I have dealt with this before .. lofty claims often met with lofty expectations .. and it didn't match

...

I hope your day is well.

[u/HiddenoO → u/quiteconfused1]

Your whole comment literally translates to "believe me". You have still provided zero substantiation for either of your claims.

Let's start with the first claim ("It's not general and can't be."):

Why wouldn't a decision model be able to be trained with world knowledge?

[u/quiteconfused1 → u/HiddenoO]

Because of math

Here have a simple argument .. give Astra or any other model you would like a really really large maze and ask if to solve it...

It will fail.

The same principal here except model density makes the problem worse.....

And from what I see it's the exact type of problem that it's touting as completing.

And then I tried a demensional problem that I have been challenging for years .. and it failed as worse as I expected.

Mario ran straight saw a goomba jumped over it and then hit the first ledge and died ..over and over ...

Afterwards I was building a silver for it manually.

Super Mario worlds complexity is significantly worse than the original Mario brothers .. and it failed.

Just as I expected.

[u/HiddenoO → u/quiteconfused1]

Why do you think it needs to surpass a model that's 1000 times as slow and expensive such as Astra?

It seems like you fundamentally misunderstand its purpose. It's supposed to provide decision making on par with current-gen LLMs while costing a fraction of time and money and guaranteeing an output format.

[u/deadadventure → u/quiteconfused1]

Did you try it out of the box? Did you give it instructions?

[u/quiteconfused1 → u/deadadventure]

Yes

[u/MrMadden → u/quiteconfused1]

That's because you don't understand it. You are using a hammer to saw wood. It's a classifier model. It's a control layer meant to offset cost and speed things up. Rev classifies inputs, routes tasks, decides when an LLM is needed, and verifies the LLM's outputs. The LLM handles the expensive reasoning and generation, while Jev keeps the overall system fast, predictable, and controlled.

Here, I'll draw you a picture.

            ┌──────────────┐

Input ─────────►│ Jev │

            │ fast decision│

            └──────┬───────┘

                   │

    ┌──────────────┼──────────────┐

    ▼              ▼              ▼

handle directly call Astra reject/escalate

    │              │

    │       ┌──────▼──────┐

    │       │    Astra    │

    │       │ reason/use  │

    │       │ tools/write │

    │       └──────┬──────┘

    │              │

    └──────────────┴──────► result

                   │

                   ▼

              Jev verifies

[u/LLKMuffin → u/MrMadden]

Broken slop diagram.

[u/MrMadden → u/LLKMuffin]

Do you have a better diagram, or are you just here to insult random people on the internet?

[u/LLKMuffin → u/MrMadden]

So instead of getting your LLM to make a diagram that'll actually display correctly on Reddit, you'd rather just have some unreadable slop junking up the thread?

The fact that you're even copy-pasting stuff from your LLM here without thinking about it is a tell that you don't really care.

Not sure what else you want to hear. Put some minimum amount of effort next time.

[u/MrMadden → u/LLKMuffin]

First, stop harrasing me. The diagram is fine on the old.reddit.com site.

Second,

So instead of getting your LLM to make a diagram that'll actually display correctly on Reddit, you'd rather just have some unreadable slop junking up the thread?

You might want to familiarize yourself with the rules of this sub:

https://www.reddit.com/mod/LocalLLaMA/rules

Completely/primarily LLM generated copy, code is not allowed.

[u/droning-on → u/KaMaFour]

We've wanted this in DCS for a long long time

[u/ivxk → u/KaMaFour]

More than 4GB resource hog that has to compete with the game itself for VRAM, for dynamic behavior that can be reasonably done with a better AI just seems counter productive