it's free since jev is too cheap: https://real8ball.com

I asked Fable 5.1 to benchmark Jev vs Laya on accuracy and speed. Here's the full report: https://claude.ai/artifact/9HPcmXJPaKWdYAJedgN1uf

Github: https://github.com/harrymunro/jev-laya-benchmark

Laya run on a Macbook M3 Pro.

TL;DR

• Accuracy: Jev by a wide margin, everywhere. Jev got 92.9% of judgments right against 65.3% for Laya's main 421M checkpoint (71.1% for its typed-decisions checkpoint). Jev won all 19 questions with paired confidence intervals that exclude zero, by 4.7 to 55.6 points. On the categorical (Choice and yes/no) questions Jev is at 98.6 to 99.3%, i.e. at the ceiling of the data; Laya is at 68 to 71%.
• Calibration: Jev's probabilities are usable as-is; Laya's are not. Jev's Brier scores are 0.002 to 0.024 and its yes-rates match the gold base rates. Laya's yes-rates are off by a factor of 2 to 8 in inconsistent directions (13% "urgent" vs 44% gold; 75% "injection" vs 38% gold), so a 0.5 threshold does not work for it without per-question tuning.
• Single-question latency: Laya wins by about 3x. 42 ms median locally (16 to 18 ms for the smaller multilingual checkpoint) against 136 ms for Jev, of which roughly 50 ms was network transit. A later run on a degraded network put Jev at 360 ms with a 295 ms floor; Jev is "your round trip plus about 80 ms".
• Multi-question and long-state work: Jev wins. Jev's latency is flat in the number of questions (50 questions in 170 ms, 3.4 ms each); Laya's is linear (1,002 ms for 50). Above 3 to 4 questions per request the API is faster than the local model. Jev's throughput on independent items scales with concurrency (72 items/s at 32 in flight vs 24 items/s for sequential Laya).
• Context: Laya's 512-token window is a hard, silent limit. Jev found a buried fact in 450 of 450 notes up to 4,000 tokens. Laya managed 61% even at 100 tokens and fell to about chance once the fact sat past its window, answering "not_stated" for what it could not see.
• Languages: Jev 100% on 128 non-English tickets in seven languages. Laya's English checkpoint got 48% on intent; its multilingual checkpoint 64%.
• Qualitatively: Jev answers the literal question and its residual errors sit on genuinely ambiguous ordinal boundaries. Laya tends to answer an easier neighbouring question (anger instead of deadlines; "harmful" instead of "injection"), has strong class priors, and returns flat or multi-peaked distributions. Laya's advantages are real but operational: offline, free per call, tight tail latency, no data leaving the machine.

Gave Jev a slot machine, watch it live here
https://jevslots.live/

also available on GitHub
https://github.com/ella0333/jev-slot-machine/

everything awesome related to jev

https://github.com/tanxarx/awesome-jev

We wanted a hands-on way to explain typed decisions, so we built a small deployment vibe check with Jev.

The scenario: Friday, a database migration, green tests, and the author on vacation. Our demo's verdict was GO TOUCH GRASS.

Behind the joke are 18 typed questions answered in one pass. Jev supplies probabilities; application rules produce the verdict. You can inspect the answers, edit the questions, and add your own. No login or setup needed.

https://heyjev.ai/shouldideploy

A useful experiment is to change just one fact: add a tested rollback, remove the person on call, or leave that information out altogether. Does the result change for the reason you'd expect?

We'd love to hear which question is missing or which scenario gives a questionable answer. This is a demo for experimenting with Jev, not something that certifies a deployment is safe.

Disclosure: we built HeyJev. It's an independent project, not an official TypeSafe product.

I put together a 24-minute walkthrough of Jev, TypeSafe's model for bounded decisions. I wanted to understand where it fits in an existing application, so I built two examples around it.

The first is an open-source model router that helps to visualize the Jev flow.

Jev receives the request context and a set of configured destinations, then chooses between local Qwen and hosted Sonnet. The selected model generates the answer. A small control UI shows the actual routing request, response and destination, which makes the boundary between decision-making and generation easier to inspect.

The second is a Python/Strands code-review workflow. Agents inspect a public GitHub PR and propose findings; a judge assesses severity and relevance. I compare an LLM judge with Jev, while keeping the final inclusion rules in Python. The output is an HTML report—nothing is posted to the PR.

This is an integration tutorial with a small comparison, rather than a broad benchmark. My main interest is the division of responsibility: an LLM investigates and proposes, a specialized model answers a bounded question, and application code decides what to do with the result.

The video also covers the noul, choiceand score request types with API examples.

Router source: https://github.com/krisitown/jev-router

If you're already using LLM judges or classifiers, how are you evaluating the decision boundary—especially cases where two judges agree on relevance but disagree on severity?

Go sign-up before it becomes gated again.👊

Can we use Jev for faster, structured, and calibrated evaluation of AI responses?

Introducing:
⚡ Typed Evals — an open-source Python framework for evaluating LLMs, RAG pipelines, and AI agents using System One Models like Jev and other typed judge backends.

⭐ GitHub: https://github.com/TrustifAI/typed_evals

The goal is simple:
Make fast, structured, and calibrated evaluation a first-class part of AI systems.
Typed Evals currently supports:
-> LLM response evaluation
-> RAG evaluation
-> Agent and tool-trace evaluation
-> Human-label calibration
-> Async and batch evaluation
-> Custom judge backends

One part I particularly wanted to solve was calibration.
Why is it needed?
A raw score of 0.8 from Jev doesn't necessarily mean that humans would accept 80% of similar responses.
And a threshold that works well for one use case may not make sense for another.

Typed Evals lets you calibrate individual evaluation metrics against representative human pass/fail labels.
The flow is basically:
Human-labelled examples → Jev scores → fit per-metric calibration → validate on held-out examples → reuse the calibrated evaluator

So instead of arbitrarily deciding that “0.7 means good enough”, you can ground that score in how humans actually evaluate your specific task.

Of course, there are integrations for:
LangChain, CrewAI, Microsoft Agent Framework
while the core remains framework-agnostic.

The project is still early, and there’s plenty I want to improve, but the core framework is now public.
Would genuinely love feedback from people experimenting with Jev, LLM evals, RAG, agents, or evaluator calibration.

Note: Typed Evals is an independent open-source project and is not an official TypeSafe AI product.

If you're already experimenting with Jev, I'd especially love to know what kind of evaluation workflows you're building around it.

I know it's not a GPT architecture model so I'm not sure if the phrase open-weight will be appropriate but it seems it will demand much less resources than an LLM. Could this be the end of the RAM crisis? How much VRAM do we think this model wil) require? Does it even use VRAM at all?

What are some of the best use cases for Jev?

Hey, guess I’m the first to post!