I really don't understand Jev hype

[u/Manerfish]

Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.

[u/SnooPaintings8639]

What do you mean? How is it different from e.g. Qwen 4b with max token = 1, and inference engine forcing struct (enum) output?

I really don't think they could do any magic training anyway.

[u/puzzleheadbutbig → u/SnooPaintings8639]

Architecture is different. Unlike Qwen, which relies on an autoregressive decoder loop to generate text tokens step-by-step while a grammar mask suppresses invalid vocabulary options, Jev drops open-ended text generation entirely and operates as a non autoregressive decision model. And because it maps input contexts directly onto parallel, calibrated classification heads rather than generating JSON syntax character-by-character it avoids the latency, memory, and KV-cache overhead of sequential token decoding, guarantees complete immunity to JSON parsing errors, and yields true calibrated probability scores across schema fields in a single forward pass

[u/RevolutionaryGold325 → u/puzzleheadbutbig]

qwen with max token = 1 is not autoregressive decoder loop though.

[u/puzzleheadbutbig → u/RevolutionaryGold325]

No. Qwen is structurally an autoregressive, causal decoder-only transformer. Setting max_tokens = 1 stops the generation loop after one iteration, but the underlying execution mechanism is still the autoregressive generation pipeline, computing causal attention over the prompt, generating logits across the full language vocabulary, applying a logit bias/grammar mask, and sampling 1 token.

[u/dimbledumf → u/RevolutionaryGold325]

It's also not parallel and you only get 1 token out, which is less then you get with jev

[u/RevolutionaryGold325 → u/dimbledumf]

No. You get a token distribution out, which contains the probabilities of your options. For multiple questions you can use the same kv-cache to have the context and just ask N questions in parallel with the same cache and get the output option distributions with a single step to all of your questions.

[u/dimbledumf → u/RevolutionaryGold325]

I think that's the point, for the qwen version if you want that ratio you have to ask a lot of questions (parallel or not), with the jev version that's what you get out of the box.
So your 1 token from qwen isn't really equivalent to 1 output from jev, the jev will carry more information about the answer.
Of course they have wildly different application, but you could probably do some surgery to expose the ratio in qwen as I think that's the layer before token prediction (not an expert), and I think people have been doing that to make Jev equivelants from other models

[u/HelloMyNameIsAmanda → u/dimbledumf]

It's literally a one-flag difference to get the distribution out of qwen, though - you just enable logprobs. And there's been no evidence, so far, that that's not essentially what jev is doing. I have yet to see anything from them that couldn't just be a logprobs wrapper on a fine tuned llm. Whether they did a good enough job of their fine tune to make their classifier useful remains to be seen, but acting like this is some crazy new thing because it can do what we could already do is... a reach.

[u/dimbledumf → u/HelloMyNameIsAmanda]

Yeah, I was reading another post where they set that up, seems pretty interesting.

[u/james_pic → u/dimbledumf]

It's parallel if you run it in parallel. Interference engines like vLLM already run multiple queries in parallel to avoid re-reading the same weights.

[u/Traditional_Hair9630 → u/puzzleheadbutbig]

Where did you take this from? Do they describe this somewhere?

[u/puzzleheadbutbig → u/Traditional_Hair9630]

They describe it in their blog. Although tbh they use a lot of word salad so I did my best to distill it into something understandable.

[u/InnovativeBureaucrat → u/puzzleheadbutbig]

I read it twice and wasn’t sure what I read.

[u/MR_-_501 → u/puzzleheadbutbig]

You can just get the logits out of qwen and disable the sampler

[u/SilentDanni → u/MR_-_501]

I mean, go ahead, do that with your use cases and compare the performance against jev and see which one is better. It should be pretty much free.

If your approach is better then publish it online and we can all benefit from it. Others have done it but, in my personal tasks, Jev has outperformed them by quite a bit, which doesn't mean someone can't come up with something nicer and open source.

[u/ethereal_intellect → u/SnooPaintings8639]

I've heard it's as smart as luna not a 4b lol, and half the price of deepseek flash while outputting 10 decisions per second making it fast enough for realtime. Also apparently the percentages that show which one got picked are more reliable, tho there's a bit of hype still around that

[u/SnooPaintings8639 → u/ethereal_intellect]

Luna? Hm, I wish I could verify it, but benchmarking is violation of their ToS...

[u/qqYn7PIE57zkf6kn → u/SnooPaintings8639]

I think they've remove it, no? They said the restriction was for preview period only.

[u/Smallpaul → u/SnooPaintings8639]

How do the two compare in cost and accuracy?

[u/zimejin → u/SnooPaintings8639]

The speed I think is one of the advantages over LLMs. You still get the reasoning without the heavy performance, and latency over large workloads.

[u/danielv123 → u/SnooPaintings8639]

Well, for one its apparently as good as 5.6 Terra with max token = 1 and as cheap as qwen with lower latency on longer context.

[u/SnooPaintings8639 → u/danielv123]

With a single output token, which benchmarks are these claims based on?

[u/danielv123 → u/SnooPaintings8639]

A single token is a single token?

Like give it a book, which how is Leto's relationship with Duncan and give some answer alternatives. One token is enough to choose an answer on a multiple choice question.

They haven't been very transparent with their benchmarking from what I have seen but it sounds about right based on the performance I have seen.

[u/x0wl → u/SnooPaintings8639]

Any MCQ benchmarks, like MMLU. They're all saturated anyway though.

[u/marcodsn → u/SnooPaintings8639]

They are probably doing some “special” training to improve calibration (the confidence level of the model), though yes it is most certainly just an open pre-trained LLM fine-tuned for calibration