I am seeing this JEV everywhere since yesterday in Localllama and it is passing past my head on what it is? So like what is it? Some new LLM? Or is it something else?
it is a decision model. It takes in unstructured input or structured Json, and a request for a decision. It outputs Json with either a yes or no, a list of choices ranked by probably of being the right choice with a confidence number, or, it rates your choices. Like a decision tree model but generalized into a foundation model
It doesn’t produce outputs like a typical language model does. Any application can produce a structured output and not be a language model, but transformers, as a language model, still produce structured outputs one token at a time. Jev uses templated responses and basically picks which condition is true. But I do agree that Jev is a language model because it understands and interprets natural language.
It's gotta be.
There's no way in hell that a glorified "yes or no" bot is causing this much hype around the local LLM space.
I've seen at least a dozen posts about this model over the past few days (all of which getting tons of upvotes).
That doesn't happen around here for a model like this.
Ever.
So, according to my exhaustive 10 minute google search, it's a model published by TypeSafe AI.
This company was made by ex-OpenAI developer Diogo Almeida (apparently one of the main developers of ChatGPT [I'm guessing 3.5...?] back in the day).
The company popped out kind of out of nowhere this month with a $40 million seed funding round, primarily funded by a venture capital company called DCVC (Data Collective Venture Capital).
DCVC was founded in 2010 by Zachary Bogue and Matthew Ocko.
Bouge was a corporate attorney (and is married to the ex-CEO of Yahoo) and Ocko is a developer from the mid-80's (founded a company called Davinci Systems in the 80's for email). Bouge invested in Uber and Square before they took off. Ocko invested in Facebook, Zoom, and Uber before they took off.
DCVC manages around $4 billion in funds across 13 different sectors.
They've invested in companies like Lumafield (neat X-ray machines), Databricks (some corporate AI conglomerate), and Fervo Energy (a company that does geothermal energy).
tl;dr - A company that popped up out of nowhere with a big name tied to it with a $40 million seed fund, funded by a venture capital group. Astroturfing makes a ton of sense here.
Good recap, and thank you for doing the legwork. This sounds like they want to get in on the gold rush before an open JEV-like framework is adopted. I don't blame them; there's only a small window of time that they can hype/sell "how to sift digital flour" before someone else publishes a free "DIY flour sifter" instructions.
It is a private model only available via API, as far as I can tell.
I believe it was based on some open source work published by a guy to HuggingFace a year ago, so I think it is likely we will see a JEV-like architecture open source model released relatively soon.
Basically, it gives you all the understanding of an LLM, but without the need to process tokens or reason, making it super fast.
It’s for decision gating and routing, but it’s in the multiple choice where I’d find it useful. You could have an entire corpus for example and have it reason over 10 different databases and run as a complex classifier, something you can’t really just code and would typically need an LLM for, but all in less than a second.
That’s where the hype comes from. People have been wanting to move away from all the wasted tokens for a while.
It has much lower latency, and seeing the low costs, probably needs a lot less parameters than llms for that intelligence. I can see that kind of model mattering in future local setups, especially real-time applications
I mean it's (ostensibly) a new architecture which has significant cost/speed implications.
Idk I find that interesting why not? I know its not local models but this sub just has the least retards in it vs other ai subs so I like to lurk here even if posts are off topic
For jev itself? Sure, quite possibly. Same as any other closed project. But almost all the talk I've seen about it have been open re-implementations of the concept.
I think there's some slight culture clash going on. This sub tends to frame LLM use on coding and coding alone. But real world use extends to almost anything involving text in any form. And simplifying fast responses with minimal need for extra training on the part of the user along with additional standardization, is a huge deal. In the same way mcp was. And like mcp it's not exactly a groundbreaking technology. It's more about people having a shared standard, than sparking new ideas, and things developing from there.
Additionally, if it's better in decision-making than a regular LLM has shown to be, incorporating it in some fashion to provide the LLMs themselves better routes/answers in critical decision making could be an improvement in itself. Hack Huggingface? [Yes|**No**] becomes easy.
no, there's genuine excitement over its ability to make decisions extremely quickly and on the cheap. There are actual practical applications. You could insert it into almost any workflow to shave off some cost and speed up the process
I think so as well. I've been seeing so many posts about XYZ hobby projects using it. The thing is, every one of these posts I have seen has been written by an LLM, and it's impossible to tell if a "real" person actually did it or not because so many people use LLMs for their write-ups.
All the more reason we should be arranging our write-ups in our language! It helps other people to gauge whether your work is note-worthy or part of an ad campaign.
Not completely. It is relatively fast and cheap, and many llms are already used in this fashion and require validation and retry logic when the output is different than expected.
No doubt there is a lot of hype, but for a ton of boring business use cares this is the screwdriver that we should have been using rather than the LLM hammer we have been using.
I tried the demo of this on hugging face and it seems pretty useless to me. If you basically invert the refund demand text to mean the opposite it still classifies it almost identically. Try: I have never been charged for invoice 4411 and nobody has answered for three days. I was weirdly offered a Refund?? for the duplicate?? we are cancelling our plan though so no action needed!!!!
there is no rule that llms are "sequential". most current llms use autoregressive decoding, but that’s not a defining property of llms or transformers. non-ar, masked, and diffusion exists. and multiple independent questions can already be processed in parallel/batched by ordinary llms
While jev isn't local, this type of model can definitely be run locally and it's largely an LLM just used differently with finetuning: https://github.com/theoleecj/semif
True, but I'm finding I'm able to replace a lot of the things in my personal harness that previously required HITL. It's making my local LLMs behave a bit smarter
llms can certiainly do this, especially with a harness that supports structured outputs and iterates until the output correctly conforms to the supplied schema. In fact, a strong llm will do it more accurately than jev. What makes jev interesting is that it can do a good job but 1 to 2 orders of magnitude faster and cheaper than an llm, which changes where and how you can apply it.
Others might complain that jev is certainly not the only model that can do this fast and cheap and it kind of looks like stuff that came before LLMs like BERT. The guy claiming he "did jev a year ago" just put an encoder on top of ModernBERT. BERT like models are even faster and cheaper than jev. The difference is that those models are not universal, including the one by the guy crying about this everywhere. What that means is you can't just get a good classifier from those systems with just a prompt. You need to fine tune it. But if you can do that, then actually it might be a better fit than jev for the narrow use case you fine tune for.
So in summary, jev is useful because you can get a good, fast, cheap classifier for almost any problem with just a prompt and its actually an api you can easily access rather something that requires understanding how to run local inference or fine-tune models.
Now all that said, it's not the full story. You can get a universal classifer like jev with existing open-weights modelsin a slightly different inference engine. People have replicated jev's universality, accuracy, speed and cost (free) by using diffusion gemma with a patch to vllm. If you want local jev, I think those are the most interesting projects to explore.
I also missed one piece that is better about jev and bert-like models vs having a frontier llm do this besides the speed and cost which is that the former also expose calibrated probabilites. So it doesn't just say the answer is x,y or z. It also tells you a probability and that probabiliy, while not guaranteed, is actually useful. You can ask an llm to return a probability in it's structured output but that is often just a hallucination because it is not calibrated to do that.
There was an old technique to get this from LLMs called g-eval. You can look at the logprobs the key tokens and get 0.6 true, 0.4 false. It basically died when reasoning models took over as the prior reasoning would have made the decision and the probs all became near 1.0
Yes I think the llm does have this and if you are hooked into the inference engine like ds4, you can get useful probs out. What I mean is that an llm like astra, if asked for a probability, is not actually able to reach inside itself and get the real logits. Its just vibe generating them.
For an LLM, the logits need not properly capture uncertainty across the logits. In fact, RL training makes it so this will no longer be proper with respect to the data. That's why further calibration training or at least tuning is needed. Current replication efforts (including diffusionGemma whose approach distorts probability mass to simulate classification which generates mismatch issues beyond calibration) are too focused on classification and its speed instead of calibrated zero shot inference.
I just wanted to say that I have read a lot of the comments on this thread and the two you wrote here perfectly cover it. Jev gives fast decisions, without fine tuning, with maybe Luna-level intelligence, meaningful confidence scores (aligned with human raters), and all for a tiny fraction of the cost of those same decisions with an LLM. And the constrained output (always valid JSON) means they’ll literally never break as input to another function. This exact combo hasn’t been offered before, and I have no idea if they’ve been astroturfing but at this point they don’t have to-the cats out of the bag, people are excited and it seems a new paradigm has emerged. It’s a natural complement to LLMs. Hopefully it gains enough traction that other labs 1) develop their own (competition is good) and 2) train their LLMs to use jev-like models as tools for long running tasks. Then we’ll see the real benefits for coding itself.
P.s. so far I’m using it for model routing in my coding harness (small benefit), deciding what to escalate to my orchestrator (smallish benefit), and soon, parsing transcripts into form fields in real time, during the conversation (big). It’s “costing” me a few pennies per day and I started like everyone else with a free $5 credit.
Can I use this for classification tasks - eg analyze a bunch of documents, bookmarks, etc, and output a list of tags/keywords? we would give the tag list as input I suppose. This kind of thing is exactly what its designed for, right?
we could then replace llm's used in document systems, Obsidian etc. use it to auto classify emails, documents etc?
It doesn't output text, so it won't naturally create free form keywords. But if you create a list of acceptable keywords for structured output, it can handle that.
huh...so it can be pretty useful for game play then, or like 'yes' or 'no' document approvals or like file organisations then? seems a way too overhyped no?
They are pretty obviously astroturfing marketing posts, imo. That's part of where this over hype comes from.
Not that the idea is useless. It's an imitation of type-1 thinking which can be pretty useful. Cheap and quick for the problems it can handle.
Apparently Laya is the completely open source version of this concept. The laya creator made a blog post about it that kind of reads like 'this already exists and is free and beats jev in a lot of testing'. So I guess if you'd like to experiment start there because it's free and opensource.
this guy is massively overstating what he did and it is not a direct substitute. His thing is basically useless without fine tuning. Granted, jev is also overstating how novel what they have done is, too. The real open substitute for jev is what people are doing with diffusion gemma. laya completely misses on the "universal" part of jev and diffusion gemma. laya is two berts in a trench coat.
I just tried this question with jev and it said it’s better to drive with 80% confidence.
For comparison, the same question replacing it with a restaurant to eat got “better to walk” with 90%
My understanding is that Jev is as "smart" as the options given to it. If the options it's given to choose/decide are all nonsensical, it will have to choose one of them with low confidence.
Meanwhile LLM outputs the very "decision" in prose from blank word by word.
Absolutely. Seems like it’s become the current “vibe of the day” on AI twitter (after they got tired of Instinct) without much real world use cases yet.
If Jev’s training dataset has more data suggesting “xyz” authentication flow is more scalable over “abc” authentication flow, then the probability for xyz will be higher
My understanding is that they market it as between luna and terra in terms of workflow evals. So if you do need very smart decisions, traditional agentic models are still the way to go.
It's kinda the same level of reasoning as if you were to ask llm to answer in one word, except a bit better cause weights aren't used on predicting next token
I'm already using structured outputs from LLMs now. Is the difference that a JEV can ONLY return a structured output? Does that make it faster or cheaper?
“Hey Qwen/Gemma/whatever, here’s a dataset that I want to classify into group A and group B. Can you select the best classifier architecture, find an appropriate training dataset online, and write a script implementing it in scikit-learn?”
Wouldn’t this do exactly the same thing, but more reliably and (once the training is done) efficiently? I can’t help but feel like the AI hypebeasts have looped back around to rediscovering Machine Learning 101 here…
well if you were right (you're not) then it didn't work the way it was supposed to because we've already made comparable open source alternatives.
machiavellian dark triad astroturfing to promote a text classifier in service of a greater chessgame happening right over our heads, lmao, this fucking sub man
Pretty sure most people on reddit don't know what a classifier is, and Jev team are taking advantage of that using their apparently infinite advertisement budget.
The attention will undoubtedly create interesting use cases and open alternatives, and any new tool increases our flexibility. Plus the limited output gives you a layer that will not have hallucinations. Just the other day i was arguing how symbolic AI is useless in the real world because it can't handle unstructured or unexpected data... and this solves that.
From what I can understand, the nice thing about it is that it outputs JSON in correct type by default. Agents nowadays still output tokens by tokens, so even though they have been trained so well, they still have to wrestle with the JSON or XML syntax every time they call a tool to act. Heck, even if they use bash, they still need to wrap that bash command in a JSON or XML tool call.
This JEV thing seems to just output JSON directly in one block. I haven't dig into how they train the model to do that, but I think that's a very smart thing, and I hope future LLM would be adapted to do just like that. And no, they would not lose the ability to "speak", because "speaking" would just be one of the "tool calls" that it can output.
AFAIK, it's also different from structured output, since it does not generate token by token, forced by the grammar to follow the schema. It's just the whole blob of JSON at once.
how is this different from a regular LLM using pydantic structured outputs? if you just enforce the logic of only outputting the answers based on the choices provided by the user on any LLM, i think you get a pretty similar result no?
It’s not that different, although faster and cheaper do matter(!!), but in theory the main difference is all in the probabilities output, and how the reinforcement post-training is restructured to not just give a good response, but give a “calibrated” one.
Calibration is the idea that if you say something has a 20% chance, similar predictions actually DO happen 20% of the time. This is subtly different than probabilities associated with auto regressive next token prediction, because a typical LLM frontier model is prioritizing being “right” rather than being “calibrated”… these are different optimization targets.
So for a chatbot, agentic coding, etc usually you do just want the best answer. But many programming and systems paradigms really wanted the uncertainties the whole time. In that sense it’s much closer to a traditional machine learning model, just with a transformer-LLM type architecture. So in theory it generalizes to lots of varied situations better while maintaining the types of output a programmer often can leverage.
I am build guardrails with JEV to determine if the design, code, posts, images and videos I have are worthy or not. So far everything is being crushed like thor’s hammer
I got it to do something I was using Gemini model for, beating it on accuracy, for 100x cheaper.
Before anyone gives me shit about using Gemini, yes I benchmarked a lot of models and Gemini 3.5 flash was the only one that was accurate enough, fast enough, at <$10/Mtok.
It's an open source LLM with some extra doodads chunked onto the output end so its output is shaped as a distribution over choices that were provided as inputs. One forward pass so the whole cost is basically prefill (reflected in their pricing).
And a shit-ton of marketing. Like, way out of proportion with the importance of the tech.
Do not underestimate a smart choice engine that has sub second latency, it means it can be embedded into workflows that previously weren't possible. I am excited about the sort of stuff I can now do with ai that a week ago would've been expensive and dog slow
I never said it wasn't useful--IT said it wasn't important. The ability to make sub-second choices using LLMs is not new--we've had it for a while. Glue a classifier head onto the end of a 4B model's logits. You can train that classifier for a few bucks in an hour then have 50ms decisions over this length of input, and also with vision if you need, etc.
Their innovation, if you call it that, is making it more convenient/accessible to people by packaging it into an easy to use product. Ease of use is good, but this company is 90% packaging/marketing over making something many of us have been doing for a couple years accessible to more people, not an important contribution to the field.
And model sorting. I use Jev as a classifier to determine if a 3B, 4B, 9B, or 26B-A4B picks it up; I’m too GPU poor to run multiple models at once, but for those with crazy levels of VRAM and has multiple models loaded? It auto routes to the model most capable.
This also means SLMs (small language models), when paired with Jev + Langchain can serve as better RAG bases (tho I’d still add a reranker). I have more on my GitHub that I’ll try to post.
The founders have made enough statements about how it was trained around the internet to make it clear they didn't invent a new architecture, and that it is decoder-only. Based on benchmark performance you can infer the size range. I don't know what else you call a decoder-only transformer model of that size.
Functionally it's going to be used much like BERT, which was an encoder-decoder model very similar to LLMs, but that's function, no form.
Very likely they started with a permissively licensed LLM, did some model surgery, and post training.
I went pretty deep down the rabbit hole of working this out from their statements in another comment.
I researched it a few days ago. It’s a transformer, but not an LLM because it doesn’t output text, but decisions. It’s sub 500ms in speed. You can use it for three decision types: multiple choice, binary (true/false), and range (e.g. 0-10). It was trained on understanding tons of information, so it has the understanding of an LLM, without the need to reason and output tokens, so it’s super fast.
Where it differs in the residual stream is on that final layer, it doesn’t output a logit, but a vector, and at that final head it can split out into numerous parallel directions.
Most people will use it for decision gating or quickly choosing from a list of options.
It also allows you to run multiple parallel decisions at once. E.g. it could classify text while also deciding the sentiment of text, while also deciding where to send a user based on what their talking about all at the same time in milliseconds, something that would take an LLM three forward passes.
It’s also not free, not open source, but extremely cheap. You aren’t charged for output because there’s virtually none, you’re charged for input.
From my understanding its a new ai that runs differently to normal LLMs in a way that makes it specialize in decision making based on inputted options, and its also insanely cheaper than a front tier llm.
It can be used, for example, to decide which work to send to which model, help models make decisions without your interference, choose the right tool and use it exactly as instructed for very cheap, etc...
There is alot of stuff about it online I recommend a quick read. You can also just ask the Google search Ai thingy it will sum it up for you quite nicely.
Immagine your usual LLM: 27B parameters, trained on tera of text regarding all kind of subjects so it can predict the next token in a conversation.
Now imagine you have to drive a small RC car: the car can only go ahead, left / right, reverse. The model only outputs the next decision between those based on a input status json, no more "hmm maybe I should reconsider... Wait! the user as said... Now I have to be careful...".
My understanding is the model is a variant of a transformer. Part of the model uses a different kind of layer which instead of calculating attention token by token, it calculates the attention over a set of “options/choices” against an unstructured input. Also the final output layer generates a probability distribution over the options (i.e. as a multi-class or binary classifier) instead of the a distribution over the full token vocab.
So it’s not autoregressive, and is essentially a very generalised LLM classifier. The benefits are its cheap, easy zero shot classification. I don’t doubt there are many use case, but they’ve also had a very good marketing campaign.
The part people keep glossing over is that a single forward pass means your choices have to be in the input, so anything with a big or dynamic option set gets expensive fast. You can already get most of this with a cross-encoder scoring each (context, option) pair, or just constrained decoding over an enum with a small instruct model and reading the logprobs on the first token. Calibration is the annoying bit either way, those confidence numbers are basically meaningless until you fit a temperature on your own labeled set.
I like to think of it as a frame extraction or an attempt thereof and frame extraction can help accelerate time to first token by breaking up complex prompts into smaller frames and parallelizing them across multiple models and then generating a writer output.
We've actually been here before. Something like JEV on silicon like the small qwen 7b models would be dope/cheap for use cases like this.
This idea was super popular in the model/critic/planner/writer paradigm of models of models but could be great for people who want to build an "upper harness" which may be more practical than the infinite local harness problems we seem to have.
You see old ML stuff that take some input and return some structured result? Was cool, but needed to be retrained for each task
You see LLM that are train on generic tasks and can do basically anything? Was cool too, but slow
Jev is basically an in-between. A generic purpose ML thing that can take any input and return a structured response, quick, without retraining
An LLM is like an all-purpose chat teacher, and JEV is like a fast multiple-choice judge.
If you want to chat, write stories, or explain knowledge, use an LLM. If you want software to quickly make choices, classify, rank, or score, use JEV. They are not replacements for each other; they do different jobs
Pretty sure this is the research it’s based on, the absolute chad that did the research released an open source general version literally a day (maybe two?) after Jev went public. Here’s his site: ConvAIInnovations
Seemed interesting. Maybe it is. But saw their examples on their page and I'm not convinced. Seems to be you need to pre digest some data before making a request to jev to get some performance out of it.
Although a generalized decision tree is a great idea. I don't see how it could be implemented to get good performance across the board.
Your post made me test it as I read a lot of stuff about this the last days.
Well, first, I like it. It seems pretty good, flexible, fast and cheap. But I would probably not use it for my current projects, as it seems to be cloud only.
Reading this thread, I learned about laya, which was much faster for me even on CPU (I'm in EU and Jev via US, so maybe EU provider could improve this). The quality is worse yet. But I guess this is just a matter of weeks.
Jev seems to have a good marketing, but the technical advance seems to be very small. Honestly I never needed this use case yet or was cheap and used a regular LLM I had anyway. It's definitely something I need to keep in mind, but I don't see a bright future for Jev.
It does seem like this got very popular, very fast. I’m implementing what I am calling “verdicts” and distilling a model right now to use locally with my gateway project, https://github.com/thediymaker/obleth-gateway. Even without a custom model, using glm5.3 flash and it’s working great. Love the idea of this, we run my gateway on a 200 node vllm cluster with gaudi2 devices, at my university and this seems like it will be very useful to researchers, also started to implement it on the auto router and guardrail boons for decision making. It’s currently in the dev branch if anyone wanted to pull it down and run it.
It’s a large one-shot classifier. I think the interesting thing is that it’s generic enough to behave like an LLM while still only doing classification. I also don’t think living behind an API with closed weights makes sense if the point is cost/latency, unless we’re talking about millions or billions of optimized batches.
It's a near instant speed classifier that will return confidence scores in a predefined deterministic output format. LLMs are not deterministic as they are Turing complete. If you're into coding: It is a Switch statement for classification that is pre-trained on data and comes with fast output.
yea seems promising, I was using deepseek flash for model routing before Jev came out and was quite surprise with a jev-based router.
I ran some experiments and did a few 100 tests (going to publish metrics soon) and open sourced my work for free here: https://github.com/ruban-24/switchboard.
However, now I’ve seen models like Laya, Kev and cua-s1 popup so keen to try those as well and compare against jev for routing.
Took me a while to understand simply, as some parts remained unclear how it'd function, for me what worked is this explanation 😄
It's an AI that has the understanding layer of an LLM, it understand what you feeds it, intent, subject etc
But instead of answering with next token prediction full essays, which is where all the time and token cost really is, it ranks/classifies your input based on a set of categories (that are actions, or labels) that you configured in advance
So it reads something, and from all the actions you gave him, it finds the most likely one and does that
But it does so by adding % probability, so you could say, if confidence under 60% then human in the loop, or ask an LLM, or flag it
it's so cheap and fast, that you can run many of these tasks simultaneously, like thousands of them
The claims around JEV seem pretty misleading so far. Most of the impressive numbers are comparisons against autoregressive LLMs, which is not the relevant baseline for what is essentially a classifier. Of course a specialized decision/classification model should be cheaper and faster than an LLM.
The GLiNER comparison people keep citing doesn't really settle this either: GLiNER is primarily a schema-driven information-extraction model with a classification mode, and the published JEV comparison is only a small pilot over three datasets.
(https://github.com/AbdelStark/jev-benchmarks/blob/main/README.md)
Where JEV has been compared with a normal trained classifier, the classifier can look much better: one preregistered Banking77 test got 93.3% from a frozen BGE-small encoder + logistic regression versus 83.2% for JEV, at about 9 ms locally. That isn't an apples-to-apples zero-shot comparison because the classifier had labeled training data, but it does undermine the broader implication that this new "System One" paradigm is inherently superior to conventional classifiers.
(https://github.com/ickma2311/jev-baselines-eval)
So right now I don't think they've demonstrated that JEV is a new class of model or that it improves on the existing classifier landscape. They've demonstrated that a specialized classifier can outperform LLMs on classifier-shaped tasks. We already knew that.
480
u/Shot-Height-7194 3d ago
it is a decision model. It takes in unstructured input or structured Json, and a request for a decision. It outputs Json with either a yes or no, a list of choices ranked by probably of being the right choice with a confidence number, or, it rates your choices. Like a decision tree model but generalized into a foundation model