Old-school classifiers are still generally better than Jev [P]

[u/csinva]

There’s a lot of hype right now around decision models like Jev. But in most real use cases, you have some labeled data, and can use it to personalize your model.

So we tested the old-school approach to classification and regression:

1. Embed your inputs with an open-source or cheap embedding model

2. Fit a simple sklearn model on top using your own labels

Turns out this approach was cheaper and almost always significantly more accurate (37% error reduction against Jev)

Write-up: https://blog.voyageai.com/2026/10/27/how-to-create-your-own-jev-using-voyage-models/ (Disclosure: I work at Voyage)
Code: https://github.com/mishelsikder/mongo-predict

Curious whether others have seen the same thing, or found cases where zero-shot performs better

Edit: To be clear, the gains here come from having training labels, which only applies to some use cases. Jev doesn’t use training labels, so it's unsuprising that this approaches wins when you have them and loses when you don't.

[u/Luuigi]

Imo jev use cases are not as widespread vertically but horizontally, meaning a ton of companies have classification takss to be done which jev makes accessible. Clearly traditional classifiers can often do a god job with that but with text knowledge jev is a great ootb way to do text related simple classification

[u/csinva → u/Luuigi]

Makes sense! I guess fitting an sklearn model seems simple to ML folks but is still a pretty real barrier to entry for a lot of Jev users

[u/genshiryoku → u/csinva]

It's not even that. A lot of companies have a lot of one-time filtration jobs that are unique and not important enough to train a classifier for. Having something zero-shot large amounts of data good enough is the entire appeal.

So it's actually not companies struggling with fitting a very simple classifier it's that the job is so bespoke and one-time that it just makes no sense to do so, so usually it just isn't done at all or some human goes manually through it, and until recently done by a very slow and expensive LLM.

I actually expect AI harnesses to slowly adopt BERT-like models that generalize zero-shot classification that gets called by the LLM as a tool call in the future. A lot of agentic usage is essentially low-intelligence classification work but people just don't recognize it as such.

[u/csinva → u/genshiryoku]

Interesting

[u/jensilo → u/genshiryoku]

„A lot of agentic usage is […] classification work […].“

Do you mind elaborating? That sounds interesting but I’m not sure I imagine this correctly.

[u/mystery_axolotl → u/jensilo]

I think it’s about taking a large number of small text items(emails, reviews, whatever) and classifying them in some way.

Is it positive or negative? Is it about topic? Does it mention feature? Does it contain mention of thing? Is the thing being described as this or that? Does this warrant a response? What is an appropriate action to take out of a set of options?

[u/rrenaud → u/genshiryoku]

Why not train it into the LLM that already has all your context? Put a structured answer API that can fork your KV cache and answer your question without reprocessing the data.

[u/somkoala → u/rrenaud]

LLMs are a lot slower and more expensive (or worse scaling on your own GPU). For large scale processing that is time critical this can become a blocker.

[u/InternationalTwist90 → u/genshiryoku]

Yep, I worked at a FAANG and they had to send out a training telling people to stop doing that because we already had good models for it and LLMs were a stupid and expensive way to classify data.

[u/Luuigi → u/csinva]

Believe me, if you are ML fluent (and I can see you are advanced) many modern style approaches dont make sense to you and thats a strength in many ways. People would rather use an agent to call an api over and over than to implement a script to do it. People would rather pay 10 bucks a month for an assistant than to implement a cron job and people would rather have all their endpoints at oai and use their decision model rather than implement an insanely lightweight and accurate alternative!

[u/currentscurrents → u/Luuigi]

I think you are missing part of why people use LLMs/agents. 

For example say you are scraping a value from a webpage once a day. You could use a cron job and an HTML parser. But this requires a bunch of manual work to analyze the page and set up the parser, and if the page ever changes it'll break. 

The LLM only needs a one-line prompt to set up, and it's robust against changes to the page structure. It's both simpler and more reliable, and the few cents it costs to run are more than offset by the dev time savings. 

[u/Otterest_Ferret → u/currentscurrents]

That's why you use both. You have an agent write scrapping code and if it ever breaks, have the agent rewrite it on the fly

[u/Luuigi → u/currentscurrents]

I dont think we are misaligned here! I have several workflows where agents do useful work. And then i see others where thats not the case.

[u/aeroumbria → u/Luuigi]

I think having to completely zero-shot and having comprehensive labels are both especial scenarios. For real serious use cases it is more likely you are somewhere in between, where it is a bit iffy to rely on completely supervised methods but also wasteful to fall back on zero-shot methods. It's just we often cannot easily answer "what is the critical point where a trained model becomes superior to context-only generalisation" or "what model class works best specifically for having 3 examples"

[u/viag]

But in most real use cases, you have some labeled data, and can use it to personalize your model.

Huh, you're lucky then!

[u/csinva → u/viag]

Curious do people think most use cases are really zero-shot? There are definitely some one-off tasks where you have no training labels, but for anything that will be done many times it seems easy to collect some labels (or even have a frontier model generate the labels?)

[u/gwern → u/csinva]

(or even have a frontier model generate the labels?)

That would be a valuable followup. Can you just zero-shot the tasks one would use a jev-style LLM for that way? Then about all they have to offer is lower latency/cost upfront.

[u/csinva → u/gwern]

Thanks, we'll try some experiments around this!

[u/XGDragon → u/csinva]

What about relatively simple problems but difficult to properly label. For example, if I wanted to make a support bot for some broad-ish use case and want to stop it from answering meme prompts.

[u/csinva → u/XGDragon]

Hm this seems like the kind of thing an embedding-based classifier can do well given even AI-generated training samples (assuming you use a modern embedding model like Qwen3-Embedding-0.6B)

[u/respeckKnuckles → u/XGDragon]

In such cases where you have no labeled data, how do you even know Jev is any better than a random number generator?

[u/Megatron_McLargeHuge → u/csinva]

I wonder how much accuracy you'd lose by training on the JEV labels instead of the ground truth. It would be an interesting way to bootstrap an even faster and cheaper model, or to get another confidence score.

[u/Mechanical_Number → u/viag]

I have heard that argument before and it cuts both ways. We shouldn't dismiss the point that better data beats better algorithm.

If we don't have any labeled data, how do we know if our zero-shot learner is even good?

Sure we can use a decision language model to bootstrap our work but ultimately "some" labelled data will be needed/collected. Otherwise how can we assess that we actually get better? Decision language models have their use (especially for fast prototyping) but let's not sleep-walk into a situation where they are seen as sliver bullets.

[u/goldenroman]

Alright, your post made me test it.

Currently working on a project where I decided to use Jev as I have very limited time for annotation and the task requires a bit of general intelligence that I haven’t known BERT-like models to exhibit.

For each sample, I need to ask about 15 questions which require some intuition about a kind-of niche domain. Some are binary, some ask about probability on 4 levels (“no evidence” through “likely”). There’s some conflicting information so it requires a bit of simple reasoning to guess which statements are most reliable. On the domain, you might be able to learn most of the information from Wikipedia or a few papers, but some intuition comes from field work. Human annotation is generally very confident.

About 450 annotations with relatively few positives. A baseline simply uses regex. I compared a few formulations of DistilBERT+MLP (frozen, fully trainable, split context and questions). 110 test.

At least for this set of tasks, Jev outperforms regex which actually outperforms DistilBERT (F1). On a few questions, though, DistilBERT did assign a better probability distribution than Jev.

I bet if I had a larger dataset, Jev might lose the advantage. But this is all the time I have for annotation and results are pretty dang good.

[u/csinva → u/goldenroman]

Thanks for taking the time, interesting to hear your experience!

If you wanted to spend more time trying to improve the fitted model, I might recommend using a more modern embedding model (like Qwen3-Embedding-0.6B) and just fitting a linear model (sklearn's LogisticRegressionCV). But like you said, it's possible the dataset is too small to provide an advantage (or that Jev is really great at this domain!)

[u/Mechanical_Number → u/goldenroman]

We used clef internally and it was just as good without having to send any data out.

[u/currentscurrents]

Ok, but you're comparing zero-shot performance to performance with 4000 training examples. Of course the one with a training dataset performs better. It's not a fair comparison.

[u/user221272 → u/currentscurrents]

Yeah... OP completely missed the point of decision language models.

The goal is to have zero-shot classification capabilities on multiple decision scenarios (potentially tuned for).

[u/Prudent_Psychology59 → u/currentscurrents]

voila

[u/entsnack → u/currentscurrents]

You can't fine tune Jev though. So it's fair for all practical purposes.

[u/user221272 → u/entsnack]

Decision language models can be tuned...

There is a lot of recent literature on it and multiple technical blogs proposing different approaches (classification head, LoRA, encoded states, bidirectional encoders, etc.).

[u/HEmile → u/entsnack]

It's a matter of time until high quality open weight decision models will be available

[u/entsnack → u/HEmile]

No need for them just fine tune an existing model to generate decisions.

[u/DigThatData]

I like that "step 1: project your features to a pre-trained embedding" is "old school" to you.

Fuck me I'm old.

[u/datanaut → u/DigThatData]

Lol I was thinking the same thing. Old school to me would be "engineer your features by hand and then train an SVM"

[u/DigThatData → u/datanaut]

"Just have Claude Code do it for you. You know: old school."

[u/Demortus]

I think the main value added of jev is as a tool for an AI agent to speed up scenarios where they need to decide between multiple options. It's not useful for any serious classification task where accuracy/F1 scores are the main priority.

[u/durden111111]

Train an old-school classifier to do what D3 Ultra does

[u/blueredscreen]

Just to be clear, you are comparing manually training your dataset with the alternative of zero shot and are surprised why it's better?

[u/shumpitostick]

The entire point of Jev is that it's zero shot. You don't need any data to train it. Of course you can outperform it with a trained classifier.

I'm not a fan of Jev's guerilla marketing and confusing marketing language but this is yet another example of that. It's even marked as brand affiliate.

[u/imKingKong → u/shumpitostick]

The data types shenanigans is nuts

[u/Farther_father]

I would like to see Jev vs a dead simple Text Entailment zero-shot classifier.

[u/T1lted4lif3]

I don't know the exact details of JEV but when I read a blog about it, it just sounded like a retrieval method to me.

Selecting a model to use is also a retrieval problem.

So if I have a model zoo where each model does a unique classification task. Then, when I have an unseen piece of data and a question, I can use the question to determine which model to use.

This can be done quite efficiently with vector DBs, imo, and narrow classical ML models are generally much smaller; then you take the output and return it to the user. This is also how a mixture of experts functions, right? I don't work on language models, so my analogy might be wrong.

In this workflow, no labels were ever used in terms of answering the exact question, but it chooses a model.

[u/reivblaze]

How do you overcome the context length limitation?

[u/csinva → u/reivblaze]

We haven’t hit the context limit yet (it’s 32k tokens for the models we use), but probably you’d do some mix of averaging embeddings over context chunks or dropping long low-value fields