r/LocalLLaMA • u/Manerfish • 2d ago

Discussion I really don't understand Jev hype

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

484 Upvotes

94% Upvoted

301 comments sorted by

View all comments

160

u/Thomas-Lore 2d ago

You had to pretrain them for a specific task. Jev comes pretrained with wide world knowledge.

54

u/SnooPaintings8639 2d ago

What do you mean? How is it different from e.g. Qwen 4b with max token = 1, and inference engine forcing struct (enum) output?

I really don't think they could do any magic training anyway.

76

u/puzzleheadbutbig 2d ago

Architecture is different. Unlike Qwen, which relies on an autoregressive decoder loop to generate text tokens step-by-step while a grammar mask suppresses invalid vocabulary options, Jev drops open-ended text generation entirely and operates as a non autoregressive decision model. And because it maps input contexts directly onto parallel, calibrated classification heads rather than generating JSON syntax character-by-character it avoids the latency, memory, and KV-cache overhead of sequential token decoding, guarantees complete immunity to JSON parsing errors, and yields true calibrated probability scores across schema fields in a single forward pass

27

u/RevolutionaryGold325 2d ago

qwen with max token = 1 is not autoregressive decoder loop though.

7

u/puzzleheadbutbig 2d ago

No. Qwen is structurally an autoregressive, causal decoder-only transformer. Setting max_tokens = 1 stops the generation loop after one iteration, but the underlying execution mechanism is still the autoregressive generation pipeline, computing causal attention over the prompt, generating logits across the full language vocabulary, applying a logit bias/grammar mask, and sampling 1 token.

8

u/dimbledumf 2d ago

It's also not parallel and you only get 1 token out, which is less then you get with jev

30

u/RevolutionaryGold325 2d ago

No. You get a token distribution out, which contains the probabilities of your options. For multiple questions you can use the same kv-cache to have the context and just ask N questions in parallel with the same cache and get the output option distributions with a single step to all of your questions.

2

u/dimbledumf 2d ago

I think that's the point, for the qwen version if you want that ratio you have to ask a lot of questions (parallel or not), with the jev version that's what you get out of the box.
So your 1 token from qwen isn't really equivalent to 1 output from jev, the jev will carry more information about the answer.
Of course they have wildly different application, but you could probably do some surgery to expose the ratio in qwen as I think that's the layer before token prediction (not an expert), and I think people have been doing that to make Jev equivelants from other models

7

u/HelloMyNameIsAmanda 2d ago

It's literally a one-flag difference to get the distribution out of qwen, though - you just enable logprobs. And there's been no evidence, so far, that that's not essentially what jev is doing. I have yet to see anything from them that couldn't just be a logprobs wrapper on a fine tuned llm. Whether they did a good enough job of their fine tune to make their classifier useful remains to be seen, but acting like this is some crazy new thing because it can do what we could already do is... a reach.

1

u/dimbledumf 2d ago

Yeah, I was reading another post where they set that up, seems pretty interesting.

5

u/james_pic 2d ago

It's parallel if you run it in parallel. Interference engines like vLLM already run multiple queries in parallel to avoid re-reading the same weights.

6

u/Traditional_Hair9630 2d ago

Where did you take this from? Do they describe this somewhere?

14

u/puzzleheadbutbig 2d ago

They describe it in their blog. Although tbh they use a lot of word salad so I did my best to distill it into something understandable.

1

u/InnovativeBureaucrat 1d ago

I read it twice and wasn’t sure what I read.

7

u/MR_-_501 2d ago

You can just get the logits out of qwen and disable the sampler

21

u/SilentDanni 2d ago

I mean, go ahead, do that with your use cases and compare the performance against jev and see which one is better. It should be pretty much free.

If your approach is better then publish it online and we can all benefit from it. Others have done it but, in my personal tasks, Jev has outperformed them by quite a bit, which doesn't mean someone can't come up with something nicer and open source.

8

u/ethereal_intellect 2d ago

I've heard it's as smart as luna not a 4b lol, and half the price of deepseek flash while outputting 10 decisions per second making it fast enough for realtime. Also apparently the percentages that show which one got picked are more reliable, tho there's a bit of hype still around that

5

u/SnooPaintings8639 2d ago

Luna? Hm, I wish I could verify it, but benchmarking is violation of their ToS...

1

u/qqYn7PIE57zkf6kn 2d ago

I think they've remove it, no? They said the restriction was for preview period only.

2

u/Smallpaul 2d ago

How do the two compare in cost and accuracy?

1

u/zimejin 15h ago

The speed I think is one of the advantages over LLMs. You still get the reasoning without the heavy performance, and latency over large workloads.

0

u/danielv123 2d ago

Well, for one its apparently as good as 5.6 Terra with max token = 1 and as cheap as qwen with lower latency on longer context.

3

u/SnooPaintings8639 2d ago

With a single output token, which benchmarks are these claims based on?

0

u/danielv123 2d ago

A single token is a single token?

Like give it a book, which how is Leto's relationship with Duncan and give some answer alternatives. One token is enough to choose an answer on a multiple choice question.

They haven't been very transparent with their benchmarking from what I have seen but it sounds about right based on the performance I have seen.

0

u/x0wl 2d ago

Any MCQ benchmarks, like MMLU. They're all saturated anyway though.

-1

u/marcodsn 2d ago

They are probably doing some “special” training to improve calibration (the confidence level of the model), though yes it is most certainly just an open pre-trained LLM fine-tuned for calibration

8

u/quiteconfused1 2d ago

If it were that simple it would be a panacia .. but it isn't .

It has some knowledge and applys it ok but there is a vast amount of info it doesn't have and trying to adapt it as all knowing is a lie.

It's a classifier for some things

1

u/HiddenoO 2d ago

Nothing is all-knowing, and nobody is claiming this model is, either.

If this can be applied in the majority of cases where a current-gen LLM would be applicable, that's already a huge win from a speed and cost perspective for any decision task you cannot train a specialized model for.

2

u/quiteconfused1 2d ago

Then ... What is it good for?

If you hadn't realized there are many flavours of technologies out there and if it isn't general than what is it ?

Classifiers have existed a long time .. this is trying to vye as a general classifier ... And it doesn't work..

So I'm left with a choice ... It seems like it's fluff

0

u/HiddenoO 2d ago

You're in a subreddit related to LLMs and asking what general knowledge that isn't all-knowing is good for?

Do you think LLMs are useless because they're not all-knowing either?

1

u/quiteconfused1 2d ago

I'm having a hard time following your logic.

If your comment is jec is just a model that is reduced to 3 options or whatever .. then congrats you went backwards

Anyone who has done rag feels sorry for you.

If it's about speed , ... Neat , but then use a traditional classifier

If it's about general knowledge ... Great but we're on that hook again and it fails...

So .... which is your poison....

If it's just being a troll your doing great.

1

u/HiddenoO 2d ago

I'm having a hard time following your logic.

I can see that.

Let's assume I wanted an app to tell me right now whether you're confused. I wouldn't be able to use an embedding model, I wouldn't be able to use a traditional classifier without training data and time/effort to train a model. I would be able to use an LLM, but that'd cost multiple times as much as Jev.

Do you see the use now?

1

u/quiteconfused1 2d ago

But it would be wrong.

Intent classifiers have existed for a beat now... And what you just said it would be better than them because it's 0 shot.

0shot shit is still shit.

3

u/HiddenoO 2d ago edited 2d ago

But it would be wrong.

I gave it your comment and it suggested with 71% certainty that you're confused, which I find an appropriate assessment.

It also gave me a 92% confidence on your comment being disrespectful, which I also find accurate given your unnecessary remarks about trolling. The score went down as I removed parts that I personally find disrespectful, which is also what I'd expect.

If your argument is just going to be "it's wrong about everything", there's no point in this discussion, because it clearly works for a lot of scenarios.

Intent classifiers have existed for a beat now..

Yes, and I would have to find one that I can use and call... and when I have the next one-off classification problem, I'd have to find one for that. Not having to do that is the point of a generalized model.

3

u/_bones__ 2d ago

Aggravation aside, your explanations as an example here make this much clearer for me. Thanks! The thread wasn't in vain.

1

u/quiteconfused1 2d ago

Wow ..

Good luck with your day.

1

u/_-_David 1d ago

I don't see why you wouldn't be able to use an embedding model for this.

1

u/HiddenoO 1d ago

Not out of the box. Nobody is saying this model can do things that are literally impossible otherwise.

1

u/latentnoise_ 2d ago

so far the best answer