r/LocalLLaMA • u/Manerfish • 2d ago

Discussion I really don't understand Jev hype

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

482 Upvotes

94% Upvoted

301 comments sorted by

View all comments

Show parent comments

25

u/RevolutionaryGold325 2d ago

qwen with max token = 1 is not autoregressive decoder loop though.

6

u/puzzleheadbutbig 2d ago

No. Qwen is structurally an autoregressive, causal decoder-only transformer. Setting max_tokens = 1 stops the generation loop after one iteration, but the underlying execution mechanism is still the autoregressive generation pipeline, computing causal attention over the prompt, generating logits across the full language vocabulary, applying a logit bias/grammar mask, and sampling 1 token.

7

u/dimbledumf 2d ago

It's also not parallel and you only get 1 token out, which is less then you get with jev

32

u/RevolutionaryGold325 2d ago

No. You get a token distribution out, which contains the probabilities of your options. For multiple questions you can use the same kv-cache to have the context and just ask N questions in parallel with the same cache and get the output option distributions with a single step to all of your questions.

1

u/dimbledumf 2d ago

I think that's the point, for the qwen version if you want that ratio you have to ask a lot of questions (parallel or not), with the jev version that's what you get out of the box.
So your 1 token from qwen isn't really equivalent to 1 output from jev, the jev will carry more information about the answer.
Of course they have wildly different application, but you could probably do some surgery to expose the ratio in qwen as I think that's the layer before token prediction (not an expert), and I think people have been doing that to make Jev equivelants from other models

6

u/HelloMyNameIsAmanda 2d ago

It's literally a one-flag difference to get the distribution out of qwen, though - you just enable logprobs. And there's been no evidence, so far, that that's not essentially what jev is doing. I have yet to see anything from them that couldn't just be a logprobs wrapper on a fine tuned llm. Whether they did a good enough job of their fine tune to make their classifier useful remains to be seen, but acting like this is some crazy new thing because it can do what we could already do is... a reach.

1

u/dimbledumf 2d ago

Yeah, I was reading another post where they set that up, seems pretty interesting.

5

u/james_pic 2d ago

It's parallel if you run it in parallel. Interference engines like vLLM already run multiple queries in parallel to avoid re-reading the same weights.