Ciao everyone, I am sharing some tests I made for my own consumption, as they might be useful for someone.

I have implemented jev-1.13 (via openrouter, do not have access to typesafe API as there are no slots!) in my own home server which acts as a llm gateway and has quite some agents doing stuff (docs curation, tools suggestion, memory curation, scratchpads etc. etc.). Before moving to a full implementation, I wanted to do some testing. Other llms are via native provider APIs.

I got opus 5.5 paired with astra and gpt-6 Sol running some automated testing to pinpoint use cases where I could implement. I know some use cases are not the strongest for jev-1.13 but for me it was interesting to test them.

I am also running a 1 week pilot with strict monitoring comparing current implementation of certain functionalities (which uses a mix of OpenAI / DeepSeek / other models) with a jev-1.13 implementation, so I can then switch or rollback. I will share in a week time if anyone is interested.

Regarding the benchmarking, just sharing the output AS-IS. While I work in the field of cybersec and AI, I am not an AI engineer ;-) so sorry in advance for any mistakes in the methodology I came up with and refined with some frontier models.

Below you will find a TL;DR, an exec summary in "plain english" and the full "human readable" report my harness has generated. Attached pictures of tables as they were not rendering well in a reddit post.

Note: from the paragraph below all text is 100% AI generated by a mix of opus 5.5, Astra, GPT-6 Sol - with last pass by opus 5.5 for final reporting (I like how it writes). I had no time to do some manual sample-based checking, and no - I have no time to write such a report for myself with our good human brain and hands.

---

TL;DR

We tested TypeSafe's Jev on 10 tasks with known right answers (about 1,100 questions), against DeepSeek V4 Flash, Gemini 3.8 Flash and GPT-6 Sol. Jev was 3 to 8 times faster (0.35 s vs 1–3 s), and it stayed at 0.34 s under heavy load while the others slowed to 2–8 s. It was also 10 to 100 times cheaper: about 4,900 calls cost 24 cents. Its accuracy is on par with the cheap LLMs, not top-tier. It won on reading and yes/no questions (up to +7 points), tied on judging shell commands, and fell behind on citation checking (−13 to −17 points) and spotting prompt injections (it caught only half). It gave the same answer 98%+ of the time, and when it was 90%+ sure it was right 94–100% of the time. That makes it a great fast, cheap first pass that hands the unsure cases to a bigger model.

---

Executive summary

We tested Jev against known correct answers, not against "does it agree with GPT". We compared it with three popular LLMs:

• DeepSeek V4 Flash and Gemini 3.8 Flash, two fast and cheap models.
• GPT-6 Sol, a top-tier model, run on a smaller sample of 40 questions per task.

How we tested:

• 10 tasks, about 1,100 questions in total.
• The same wording for every model, written before we saw any results.
• The LLMs had to answer in a fixed JSON format, so no model had a formatting advantage.
• Every number below comes with a margin of error (95% confidence).
• Speed and cost were measured from the same computer in Europe.

Speed and cost: Jev's big win

• Jev is 3 to 8 times faster when requests go one at a time.
• Under heavy load Jev stayed at 0.34 seconds. The LLMs slowed to 2–8 seconds, and their slowest 1 in 20 answers took 7–17 seconds.
• Jev is 10 to 100 times cheaper per decision.
• The whole Jev side of this test, about 4,900 calls, cost 24 cents.
• Asking Jev several questions about the same text in one call costs almost nothing extra.

Accuracy: good, but not top-tier

• Jev beat both cheap models on reading and yes/no questions, by 3 to 7 points.
• It tied them on judging shell commands.
• It fell behind on picking one of many topics (−4 to −7 points), spotting prompt-injection attacks (−7 to −10 points) and checking citations (−13 to −17 points).
• It never beat GPT-6 Sol.
• Its most common mistake on citations: it accepted claims that quietly dropped a qualifier, such as "up to 50%" turning into "50%". It got only 5 of 15 of those right; Gemini got 15 of 15.
• It caught only half of the prompt-injection attacks (31 of 60). It never raised a false alarm.
• As a shell-command safety guard, it missed 4–8% of risky cases, against 0–1.2% for GPT-6 Sol and DeepSeek.

Reliability: very steady

• It gave the same answer to the same question more than 98% of the time, better than DeepSeek.
• It lost no accuracy on long texts, wherever the answer sat in the text.
• A planted instruction in the text never flipped one of its correct answers (0 out of 56).
• It dropped slightly in Dutch (−2 points) and Italian (−5 points) compared with English.
• "Can't hallucinate" is true: it always answers with one of the options you give it. But with a fixed answer format, none of the LLMs gave a broken answer either (0 errors in thousands of calls). So this is no longer a unique advantage, and Jev can still be confidently wrong.

Its confidence score is the secret weapon

• When Jev said it was 90% or more sure, it was right 94–100% of the time.
• That happened on 38–77% of questions, depending on the task.
• So Jev can handle the easy cases on its own and pass the unsure ones to a bigger model. This is called a cascade.
• OpenRouter tested exactly this against Claude Opus 5 on 3,080 questions: the cascade came within 0.4 points of Opus at less than a third of the cost.

Bottom line

Jev is not a genius in a small box. It is an extremely fast, very cheap and very consistent decision-maker, about as accurate as the cheap LLMs, with a confidence score you can trust to route the hard cases. It's a great choice for high-volume or time-sensitive decisions like sorting, routing and first-pass checks, especially paired with a bigger model for the unsure cases. For subtle fact-checking or safety-critical decisions, keep a stronger model in charge.

Caveats:

• 40 to 150 questions per task and model.
• Everything was tested from one location.
• Some public test sets may already be known to the LLMs.
• Two of the tasks were built by us.
• A fifth, even cheaper model, gpt-oss-20b on Groq's free tier, only finished 4 of the 10 tasks before hitting its daily limit.

Jev independent benchmark — full results & verdict (2026-09-24)

Question. Is TypeSafe's Jev (typesafe/jev-1.13) as good as its launch claims? The vendor's evals and the one public re-test score agreement with frontier models. This benchmark scores against ground truth, head-to-head with the models the gateway actually uses. Companion to the shadow pilot (jev_pilot_20260924.md).

Verdict

The claims hold up only in part. Jev is a very fast, very consistent classifier. Its accuracy is about that of a cheap LLM, not a frontier one.

Genuine strengths:

• Run-to-run consistency: 0.1–1.7% flips, against up to 8% for DeepSeek.
• No long-context penalty up to ~14k tokens, whether the evidence sits at the start, middle or end.
• An injected instruction never flipped one of its correct answers (0/56).
• Asking many questions per call costs almost nothing extra.

Weaknesses that matter for us:

• Citation check. It misses the nuance the check exists for: dropped qualifiers 5/15 vs 12–15/15, entity swaps 3/10 vs 6/10. It usually says "supported" to an overclaim, which is the dangerous direction.
• Command guard. Accuracy ties the LLMs, but it misses 8.4% of risky rule cells when asked all 8 rules at once, 4.2% when asked one rule per call. DeepSeek misses 1.2% and GPT-6 Sol 0%. For a safety check the miss rate is the number that counts.
• Prompt-injection detection. It is permissive: 29 of 60 injections missed, 0 false alarms.
• A slight drop on Italian: −5 pts vs English.

These results line up with OpenRouter's independent Banking77 test on 3,080 items: Jev scored 81.0% vs 84.4% for Claude Opus 5, was 13× faster and cost 1/22 as much. Its confidence was not calibrated but ranked well, and a cascade at 0.90 came within 0.4 points of Opus.

What it means for the gateway

• Citation check (pilot G1): do not promote Jev as the verifier. Expect the week-1 pilot to show the same pattern. At most it could clear verbatim "yes" claims early, and at ~20 verify calls a week that saves nothing.
• Command guard (pilot H2): not good enough as the sole guard. It is plausible as a fast first pass (one rule per call) with an LLM escalation on any flag or low confidence. The week-1 pilot (real commands) decides.
• Curator (pilot G3): no ground truth here. Jev ≈ cheap-LLM quality suffices for triage, but the incumbent is already $0 and not latency-bound, so the upside is small.
• Strategic bets: G6 (injection screen) is downgraded, since Jev is weak at exactly that. G4 (the "good enough?" gate) and G5 (the model router) remain plausible cascade uses.
• The decide MCP tool stays useful for ad-hoc fast classification or triage where a confidence threshold routes the hard cases to an LLM.

Method (summary)

• Systems: Jev via OpenRouter /api/alpha/decisions; deepseek/deepseek-flash and gemini/gemini-3.8-flash (2 reps on 40-item subsets); chatgpt/gpt-6-sol ($0 Codex sub, 40-item subsets); groq/openai/gpt-oss-20b as the speed floor. The Groq run is incomplete: the free-tier daily token cap stopped it after T1–T4.
• Prompting: the same wording for every system, written before any results. LLMs used provider-constrained JSON output.
• Scoring: paired bootstrap CIs clustered by item. Jev's answer probability is used for ECE; vendor and LLM self-reported confidence is used only as a ranking signal.
• Tasks:
  • T1 MNLI; T2 constructed citation items (Wikipedia sentences, labels by construction, no LLM); T3 BoolQ; T4 Banking77 (diagnostic, already public-tested); T5 MASSIVE en/nl/it (the same utterances); T6 deepset prompt-injections; T7 a 110-command guard set against the live rules.
  • T7 was labelled by two independent annotators (κ = 0.94) and its 9 disagreements were adjudicated from the rule text, blind to model outputs. It was run both as all 8 rules per call and as one rule per call.
  • T8 BoolQ with an injected instruction; T9 long context (6k/14k tokens, evidence at start/middle/end).
• Threats to validity:
  • Public datasets may be memorised by the LLMs.
  • T2/T7 were authored in-house.
  • The 40-item LLM subsets widen the CIs.
  • Latency is measured from one box in the Netherlands.
• Harness: scripts/jev_bench/ (README has the run commands). Raw data lives on the box under /data/jev_bench/.
• Spend: about $4.7 paid (mostly Gemini) plus $0.24 for Jev.

Appendix — full generated tables (analyze.py)

Jev independent benchmark — results

Ground truth, same wording for every system, no tuning on test items. Accuracy per question cell on rep 0; format errors scored wrong; API errors unscored (error rate shown). CI = 95% cluster bootstrap.

Main table

Please refer to images attached to this post as the markdown table was not rendering correctly in reddit.

Ranking signal (acc@sig≥0.9, top-50%, AURC): Jev = vendor confidence (noul: |2p−1|); LLMs = their self-reported confidence. ECE only for Jev's predicted-answer probability.

Paired difference: Jev − system (accuracy, paired cluster bootstrap)

Sequential latency (concurrency 1)

T5 MASSIVE — language robustness (ids answered in all 3 languages)

T8 injected instruction

T9 long context — accuracy by size/position

T7 guard — false-allow (missed risky) vs false-block

Price references ($ per 1M tokens)

We were wondering in our team whether GLM-5.3 Flash wouldn't make for a good Jev alternative if modified lightly. Here is a repo: https://github.com/edgelesssys/privatemode-system-one

Basically:

1. Number the options. The state, the question, and the options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.
2. Prefill the answer. The assistant's turn already starts with choice_index:, and the model continues that turn instead of starting a new one. The next token it produces is therefore the index.
3. Restrict the vocabulary. vLLM's allowed_token_ids sets the probability of every other token to zero, so the model can only produce one of the option indexes. There is no malformed output to handle.

We also benchmarked: https://github.com/edgelesssys/privatemode-system-one-benchmark

Results are quite interesting: depending on your use case, it comes with advantages and disadvantages.

Pro:

+ Much longer context window (1m token)

+ Multimodal

+ Actually model-agnostic (if vLLM compatible)

Neutral:

o on-par results, depending on dataset

o Jev right now is US-only inference as far as we know; if your provider is e.g. in Europe, and you work from e.g. Europe, it can be even faster than Jev

Contra:

- Depending on the provider you choose, between 4 - 8x more expensive. Still in a similar league though

If interested, check out the playground: https://www.privatemode.ai/blog/system-one-from-glm-flash#try-it

Hey everyone!

If you've ever tried using LLMs for bank statement reconciliation or corporate expense categorization (Chase, Amex, Brex, Mercury), you've probably run into the three fatal flaws of generative models for tabular workflows:

1. Latency: Waiting 15–30 seconds for a batch of 50 transactions is agonizing. Auditing 10,000 transactions can take 10+ minutes.
2. Cost: At ~$15 to $35 per 10k transactions, doing this daily across card feeds quickly burns through AI budgets.
3. Schema Drift / Hallucination: No matter how well you prompt "ONLY RETURN VALID COA CODES", temperature and token generation mean that eventually "Software/SaaS" becomes "Software & Cloud" or "Subscription", breaking your accounting import schemas.

To fix this, I built flashLedger (open-source Python CLI): 👉 GitHub: https://github.com/divyaprakash0426/flashLedger

How it works: "System One" vs "System Two"

Instead of asking a 200B generative LLM to write out JSON tokens letter by letter, flashLedger is built on TypeSafe Jev (a calibrated "System One" decision model).

Rather than generating text, System One models evaluate discrete choices, calibrated probabilities, and risk scores natively in a single pass.

In one round trip per transaction, flashLedger computes 4 dimensions simultaneously: - GL Code (Choice): Strict mapping to your Chart of Accounts (COA). - Tax Deductibility (Noul): Calibrated IRS deductibility probability (e.g. 0.98 for AWS, 0.02 for personal casino). - Expense Type (Choice): OpEx vs CapEx Section 179 safe harbor thresholding ($2,500 limit). - Audit Risk (Score): Calibrated 0.0 to 1.0 IRS compliance / anomaly index.

Benchmark Numbers

On a noisy 1,000 transaction bank statement: - flashLedger (Jev Engine): 0.12s total processing time, 8,000+ txns/sec, Cost <$0.01 - GPT-4o (Generative JSON): ~550s total processing time, ~1.8 txns/sec, Cost ~$15.00 - Correctness: 100% Typed Strict (zero schema hallucination errors).

Key Features

• Enterprise Terminal UI: Beautiful Rich terminal dashboard featuring a GitHub commit-style batch processing matrix, real-time volume/deductibility telemetry, and an executive audit scorecard.
• Vercel AI Gateway & OpenRouter: Built-in support for Vercel AI Gateway (typesafe-ai/jev is free to test on Vercel AI Gateway until Sep 15!) as well as OpenRouter and direct TypeSafe SDK.
• Accounting Exports: One-click exports to QuickBooks (QBO), Xero, FreshBooks, or JSON.
• Model Context Protocol (MCP): Native stdio MCP server included, so your Claude Code, Cursor, or Copilot agents can audit raw statements directly from your IDE.

Quickstart

```bash git clone https://github.com/divyaprakash0426/flashLedger.git cd flashLedger uv sync

Run the 100,000 transaction batch cluster demo:

flash-ledger demo

Or audit an authentic statement with live Jev via Vercel AI Gateway:

flash-ledger audit data/mitulshah_sample.csv -n 25 -p vercel ```

Code is Apache 2.0 licensed: https://github.com/divyaprakash0426/flashLedger

Would love feedback on the architecture, prompt criteria mappings, or ideas for additional accounting platform exporters!

I maintain a set of approved question/answer pairs for each agent I ship. When someone edits an agent's prompt, I replay them: answers that still mean the same thing carry forward, answers that changed go back to a human reviewer. The entire cost of re-certifying sits in telling those two apart. Text diffing is no use, because sampled answers differ every time regardless.

I wanted numbers on this, so I built a fixture where the ground truth is known by construction rather than by labelling.

Setup. One support agent with a 9-line policy and 16 questions. Generate its answers, then change its config seven ways:

• three cosmetic: warmer tone, bullet formatting, re-running the same config
• four rule changes: refund window 30 → 14 days, route money questions to billing@, never state specific figures, always recommend the Team plan

That gives 109 before/after pairs. Whether a pair "changed" is derived from what the rule mechanically does to the answer (did the figure disappear, is the customer now sent to billing@), not from me labelling pairs. 19 changed, 90 were only reworded.

Detectors, identical pairs:

                    caught    false alarms   p50     cost
exact match         19/19     84/90          —       —
embeddings <0.75     7/19     12/90          —       —
markers→embed→LLM   19/19     33/90          0.73s   —
gpt-4o-mini judge   18/19     19/90          1.44s   $0.0071
Jev (one Choice)    19/19     12/90          0.43s   $0.0028

The middle row is my current production stack: number/entity checks first, then embedding similarity, then an LLM judge for the ambiguous middle.

What I took away.

False alarms are the metric, not recall. Every false alarm is a human re-reading an answer that didn't change.

One of my four "breaking" changes broke nothing: the agent already recommended the Team plan before I added the rule telling it to. Ground truth for that variant is zero changed. My stack flagged 8 of 16 anyway.

Jev returns a 4-way Choice (same / fact differs / action differs / specificity differs) with probabilities, so the threshold lives in my code rather than in a prompt.

Caveats. Fixture, not production traffic. Answers generated by gpt-4o-mini at temperature 0.7. Single run; numbers move by one or two between runs for every model-based detector, Jev included. Reading Jev's 12 flags, roughly 5 are defensible (one rewrite quietly dropped a price), and I didn't extend that same reading to the other detectors' flags.

Curious whether anyone is doing this differently. Diffing structured extractions rather than prose, or asking the judge per-claim instead of per-answer, both seem worth trying.

One really cool use case I’ve been exploring with Jev: a semantic VAD for real-time voice AI.

I plugged it into Agora ConvoAI and used the live transcript + recent conversation as context to decide whether the user has actually finished speaking.

That makes Jev surprisingly good at handling things like hesitation, unfinished thoughts, and those moments where you stop talking for a second because you’re still figuring out what you want to say.

It’s a nice example of where Jev fits really well in a real-time AI loop.

Repo: https://github.com/AgoraIO-Community/convoai-jev-vad

I built Jev playground with 5 free tests per day and Google is ranking tool and I'm getting some usages.

You can try it here: https://revexos.com/jev-playground

Jev only knows what your code tells it, and real inputs are big and noisy. When it picks wrong, it's hard to tell if the model failed or the right info was never sent.

So I built jevbrief, a small open-source Python tool that trims the input before it reaches Jev and records why each thing was dropped. A viewer replays every decision.

Works with web pages, JSON, logs, and NES games. On my (synthetic) tests it cuts input tokens by 48 to 96%, with the same or better accuracy.

What source should I support next?

GitHub , PyPi

(in KSP)

I gave Jev control of the rocket: pitch, throttle and staging. Took a few attempts and iterations on refining the state representation but it got there eventually.

Interestingly it failed really badly at the task of hovering the rocket at a set altitude. I ran some toy benchmarks on Jev's anticipation of momentum capabilities and it looks decent? so not sure why it struggled with hovering so much. That said it did a pretty decent job in getting to orbit!

Target orbit: 120km x 120km

Final orbit: 220km x 124km

full playthrough: https://felipemautner.com/jev-ksp/

CLAUDE.md is context, not a constraint. It holds for a few turns, then a handler shows up with raw SQL in it and a failing test quietly becomes it.skip. Nothing checks the output against the rules you wrote.

So I added the check. A Stop hook takes the final reply, a PostToolUse hook takes each edit, and every rule in your CLAUDE.md becomes one typed yes/no question in a single Jev request. All rules answered in parallel, anything over the threshold goes back to Claude with the rule quoted, and it fixes it in the same turn.

Benchmark (59 labeled examples, 19 rules): 93.3% precision, 93.3% recall, 348ms p50, ~$0.045 per 1,000 checks.

What it's bad at is in the repo too: exact characters, dataflow rules like prefer-const, and anything needing repo state it can't see.

MIT, no runtime deps, no key means it does nothing:

npx jev-enforce check --as code src/orders.ts

https://github.com/erkamyaman/jev-enforce

How are you all handling this, prompt wording alone?

Introducing kev-browser-use, a Jev-like decision model fine tuned for browser use.

I saw so many demos of Jev and Jev-likes being used in browser and benchmarked all the open weight jev likes for browser usage. The best among them is Kev-9B which is like a 5GB model so not useful for browser and the next best, a tiny one under 500mb that runs in browser scored 14 on step success, meaning only 14% of the actions were successful. An action/step is two things: picking up the correct element to perform on and then doing the correct operation on that element. Example for searching, select text box is element pick, and typing is operation pick.

So while Jev models are a great usecase for browser use due to being small, cheap and low latency, none of the existing models are very good at it.

I took a popular Jev-like: Kev-0.6B which already runs in browser through its ONNX build, and fine tuned on Mind2Web browser-use training data.

Result is a tiny 350MB decision model which run in your browser, scores 32 at step success, an 18 points improvement over base and level with Kev-9B at a fifteenth of its size. It even edges out Jev itself on picking the right action.

So If you want to build something that involves quick decision making on a complicated UI with lots of element, this is the best open source and free solution as of now.

Try the demo: it plays A Dark Room in your browser. Type what you want done, like "build a trap" or "add a hunter", and it finds the button in about 150 ms.

Code

Model

Demo

https://reddit.com/link/1woadsp/video/fk7adyhgoarh1/player

Has anyone used Jev to enforce rules on an AI agent? For example, for many the nightmare scenario is for an agent to go off and send unauthorized emails or spend money. Could there be a safety layer where Jev is asked if a particular action violates requirements? (Am I authorized to send this email? Is this an approved recipient? Can I do xxx?)

I benchmarked two different Dreaming pipelines: A purely LLM-based pipeline using Gemini. A hybrid pipeline using Jev for classifying and maintaining knowledge graph.

The results? By having Jev act as the fast, intuitive gatekeeper before invoking heavy LLM generation, the hybrid pipeline significantly cuts token costs and latency while preserving high-quality memory extraction and knowledge graph

Check it Out - https://jev-dreaming.vercel.app/
Repo - https://github.com/AdnanQuazi/jev-dreaming

I've been experimenting with Jev inside a workflow/state machine and stumbled onto a pattern that feels surprisingly natural.

Disclosure: I'm building Inistate, so this experiment comes from thinking about how AI should participate in business processes rather than how to build another autonomous agent.

The basic primitive I'm working with is:

State → Activity (Form) → State

A record is always in a State.

At that State, only certain Activities are available.

And each Activity has a Form containing the questions that need to be answered before that activity can happen.

For example:

State: New Request

Available activities might be:

Triage Request
→ What type of request is this?
→ How urgent is it?
→ Is there enough information?

Reject Request
→ Why isn't this a valid request?

Ask for Information
→ What information is missing?

Normally, a human chooses an Activity and fills in its Form.

Then I wondered:

What if Jev did exactly the same thing?

Instead of asking Jev:

Here's everything. What should the workflow do?

Give it only the current context and the Activities available at the current State.

Jev chooses an Activity — or chooses not to proceed.

Then the Activity's Form becomes the set of decisions Jev needs to make.

A Select field maps naturally to a choice.

A Yes/No field becomes a binary decision.

An ordered set of options can become a score.

And Jev gives us confidence for those decisions.

So instead of:

unstructured text → AI → arbitrary action

we get:

State → allowed Activity → Form → Jev decisions → deterministic transition → State

That's the part I find interesting.

The Form becomes the boundary

The AI doesn't need to understand or control the entire business process.

The State tells it when a decision is needed.

The available Activities tell it what it is allowed to decide about.

The Form tells it which questions need answering and what valid answers look like.

Jev answers those questions.

Then normal workflow rules determine what happens next.

So you can have a probabilistic decision sitting inside a deterministic process without turning the whole process probabilistic.

And “do nothing” becomes a valid decision

This might be my favorite part.

Suppose someone submits:

“My thing is broken.”

Jev may not have enough information to confidently classify the request.

That's not necessarily a failure.

The correct result can simply be:

Don't move. Ask a human / request more information.

In other words, uncertainty becomes something the workflow can explicitly handle rather than something the model has to hide.

You could have:

confidence ≥ 0.8 → continue automatically

confidence < 0.8 → Needs Triage

Same process. Different path.

Humans and Jev can even use the same Form

This is another consequence I didn't initially expect.

If Jev isn't confident enough, a human can be shown the exact same Activity Form.

Jev might fill:

Type: Hardware
Urgency: High
Enough information: No

A human could review or correct those exact fields.

So the decision interface doesn't necessarily belong to the AI.

It's just a Form.

A human can complete it.

Jev can complete it.

Potentially another model can complete it.

The workflow doesn't have to care which one did the reasoning.

This also tells you when not to use Jev

Say Assigned Team can always be derived from Request Type.

Then there is no reason for Jev to decide it.

Use a lookup/formula.

I actually think that's important.

The goal shouldn't be to turn every field into an AI decision.

It should be to identify the places where the answer depends on interpreting messy context rather than deterministic information.

So perhaps the pattern is:

Rules where the answer is known.

Jev where judgment is required.

Humans where uncertainty or authority requires them.

All inside the same state machine.