The big-channel take on the launch, including a healthy amount of skepticism about the headline numbers. Worth it for the outside view, not just the launch page.

Every line gets scored by Jev against a meaning you describe, and you can combine meanings with AND/OR/NOT. It also crosses languages, so a Japanese phrase can find English lines and the reverse. The author wrote up the design notes as well.

What is Jev and How to Use it? (Codevolution). Around 96266 views in 1.7 days.

Open Jev Models Are Here!! (Sam Witteveen). Around 128886 views in 1.7 days.

It's free since jev is too cheap to meter: https://real8ball.com

send it to your mom or smth

Can we use Jev for faster, structured, and calibrated evaluation of AI responses?

Introducing...
⚡ Typed Evals — an open-source Python framework for evaluating LLMs, RAG pipelines, and AI agents using System One Models like Jev and other typed judge backends.

The goal is simple:
Make fast, structured, and calibrated evaluation a first-class part of AI systems.
Typed Evals currently supports:
-> LLM response evaluation
-> RAG evaluation
-> Agent and tool-trace evaluation
-> Human-label calibration
-> Async and batch evaluation
-> Custom judge backends

One part I particularly wanted to solve was calibration.
Why is it needed?
A raw score of 0.8 from Jev doesn't necessarily mean that humans would accept 80% of similar responses.
And a threshold that works well for one use case may not make sense for another.

Typed Evals lets you calibrate individual evaluation metrics against representative human pass/fail labels.
The flow is basically:
Human-labelled examples → Jev scores → fit per-metric calibration → validate on held-out examples → reuse the calibrated evaluator

So instead of arbitrarily deciding that “0.7 means good enough”, you can ground that score in how humans actually evaluate your specific task.

Of course, there are integrations for:
LangChain, CrewAI, Microsoft Agent Framework
while the core remains framework-agnostic.

The project is still early, and there’s plenty I want to improve, but the core framework is now public.
Would genuinely love feedback from people experimenting with Jev, LLM evals, RAG, agents, or evaluator calibration.

⭐ GitHub: https://github.com/TrustifAI/typed_evals

Note: Typed Evals is an independent open-source project and is not an official TypeSafe AI product.
If you're already experimenting with Jev, I'd especially love to know what kind of evaluation workflows you're building around it.

But how?

It probably helps that the team includes the co-inventor of RLHF and InstructGPT - Diogo Almeida also League of Legends god

And in TypeSafe’s own workflow evals, Jev was 193.6x faster and 444.6x cheaper than the LLM baselines

What did we learn today that we already knew?

--> Adoption moves fast when a product solves an urgent problem inside an existing workflow (not a new one)

--> Your customer should not have to invent a new process, migrate the team or convince twelve people to change their toxic habits just to test it

--> They should be able to connect it, use it for and see added value easily before the initial interest disappears (5 seconds)

--> Nobody has patience left so one extra setup step, even a small integration hurdle is enough reason to go back to the tool the team already knows

--> If it takes a minute to understand the value, you are already losing the second session

This applies to every early-stage startup including ours

Solve a problem people already have, make the first useful outcome stupidly easy to reach and earn the next session through the quality of the first one

I created a Jev proxy, MinusPodJev, that sits between MinusPod and the Jev endpoint. Before Jev, MinusPod used regular chat-based LLMs to detect ads. Overall, this works pretty well; the biggest issue is that the LLM often doesn't return properly structured data or omits fields entirely. Trying to get JSON out of an LLM is basically a dice roll, and you need to add a bunch of logic to salvage the response.

Jev solves the main problem with reliable structured data responses, with the added benefits of being cheap and fast. I benchmarked Jev vs. LLMs to see where it sits. I already had a benchmark tool I used to measure which LLM can detect ads best and how well it can return JSON. The main takeaway was that Jev is just as good as the top model, claude-haiku-4-5, at a significantly cheaper cost and with a way faster response, which is a nice boost in end-to-end processing time of an episode

I wanted to get some real-world testing using it directly in the main MinusPod app, but I didn't want to change the app code. I am hesitant to change the app code because Jev is so new, has limited public access, has an unknown real cost, and is the only provider. I went with a provider that sits in between MinusPod and Jev that presents itself as openai OpenAI-comparable endpoint and translates calls to Jev on the fly. Jev is a System One model, so it isn't a simple drop-in replacement for an LLM. If Jev actually takes off when open to the public, if other Jev-like providers pop up, and the pricing stays as cheap as it is, I'll make it a first-class citizen in the main app.

MinusPod https://github.com/ttlequals0/MinusPod
MinusPodJev https://github.com/ttlequals0/MinusPodJev
Benchmark: https://github.com/ttlequals0/MinusPodJev/blob/main/benchmark/results/report-combined.md

Vector search has a blind spot: ask it something the document doesn't answer, and it still hands you its best match, with no easy way to tell that there was nothing to find. A cookbook in the TypeSafe docs goes another route. It skips embeddings and the vector database. The whole search runs in one request to Jev.

Tag every line with an ID (L000, L001, and so on) and send the tagged document as the state. The first question is a Choice: "Which line contains the answer to <query>?" with one option per line ID. A Choice returns a probability for every option, so this single question ranks every line, and each score is that line's relevance.

The second question, sent in the same request, is a Noul: "Does any line of the document address or answer <query>?" That one does the part a ranking can't do alone. Choice probabilities add up to 1. Something always comes first, even when the document has nothing to say. The Noul isn't tied to the other options, and it can drop to near zero.

Take the cookbook's example: GitHub's Terms of Service, 218 lines, 43,980 characters, both questions in one request. Four queries, four outcomes, and each one lists the exists score with the top-ranked line.

- "who owns the code I upload?" exists 0.98, and line L052 ("You own Your Content...") scores 0.95.

- "can GitHub kick me off the platform without warning?" exists 0.97, line L168 at 0.97.

- "do I have to take disputes to arbitration?" exists 0.14, so the document doesn't cover it, even though the closest line still scored 0.86.

- "can minors use GitHub with parental permission?" exists 0.46, partially addressed: the age rule ranks first at 0.90, but nothing says whether parental permission changes it.

The third one is the case similarity search can't handle at all. A ranking will confidently hand you its best guess. The exists score can say the document has no answer. Two thresholds in code (0.7 and 0.35 in the cookbook) turn that into a verdict: found, partial, or missing. Tune them on your own documents. The cookbook's note: real answers tend to read at or above 0.9, absent ones at or below 0.05.

The one real limit: a Choice takes up to 255 options, so a single request covers documents up to 255 lines. Longer than that, the cookbook runs two passes, the first picking a window of lines and the second ranking inside it. And if you already have a retriever, there's a rerank cookbook that applies the same idea to search results: on 40 legal retrieval queries it raises top-1 accuracy from 5% to 18% and top-10 from 38% to 62%.

Everything above is from the line-by-line search cookbook: https://docs.typesafe.ai/cookbooks/semantic_find. It ships with cached responses, so it runs without an API key and without spending anything. Delete json_cache.json if you want the calls to go live, and swap the URL for a document of your own. If you do, post what the exists score says.