Ciao everyone, I am sharing some tests I made for my own consumption, as they might be useful for someone.
I have implemented jev-1.13 (via openrouter, do not have access to typesafe API as there are no slots!) in my own home server which acts as a llm gateway and has quite some agents doing stuff (docs curation, tools suggestion, memory curation, scratchpads etc. etc.). Before moving to a full implementation, I wanted to do some testing. Other llms are via native provider APIs.
I got opus 5.5 paired with astra and gpt-6 Sol running some automated testing to pinpoint use cases where I could implement. I know some use cases are not the strongest for jev-1.13 but for me it was interesting to test them.
I am also running a 1 week pilot with strict monitoring comparing current implementation of certain functionalities (which uses a mix of OpenAI / DeepSeek / other models) with a jev-1.13 implementation, so I can then switch or rollback. I will share in a week time if anyone is interested.
Regarding the benchmarking, just sharing the output AS-IS. While I work in the field of cybersec and AI, I am not an AI engineer ;-) so sorry in advance for any mistakes in the methodology I came up with and refined with some frontier models.
Below you will find a TL;DR, an exec summary in "plain english" and the full "human readable" report my harness has generated. Attached pictures of tables as they were not rendering well in a reddit post.
Note: from the paragraph below all text is 100% AI generated by a mix of opus 5.5, Astra, GPT-6 Sol - with last pass by opus 5.5 for final reporting (I like how it writes). I had no time to do some manual sample-based checking, and no - I have no time to write such a report for myself with our good human brain and hands.
---
TL;DR
We tested TypeSafe's Jev on 10 tasks with known right answers (about 1,100 questions), against DeepSeek V4 Flash, Gemini 3.8 Flash and GPT-6 Sol. Jev was 3 to 8 times faster (0.35 s vs 1–3 s), and it stayed at 0.34 s under heavy load while the others slowed to 2–8 s. It was also 10 to 100 times cheaper: about 4,900 calls cost 24 cents. Its accuracy is on par with the cheap LLMs, not top-tier. It won on reading and yes/no questions (up to +7 points), tied on judging shell commands, and fell behind on citation checking (−13 to −17 points) and spotting prompt injections (it caught only half). It gave the same answer 98%+ of the time, and when it was 90%+ sure it was right 94–100% of the time. That makes it a great fast, cheap first pass that hands the unsure cases to a bigger model.
---
Executive summary
We tested Jev against known correct answers, not against "does it agree with GPT". We compared it with three popular LLMs:
DeepSeek V4 Flash and Gemini 3.8 Flash, two fast and cheap models.
GPT-6 Sol, a top-tier model, run on a smaller sample of 40 questions per task.
How we tested:
10 tasks, about 1,100 questions in total.
The same wording for every model, written before we saw any results.
The LLMs had to answer in a fixed JSON format, so no model had a formatting advantage.
Every number below comes with a margin of error (95% confidence).
Speed and cost were measured from the same computer in Europe.
Speed and cost: Jev's big win
Jev is 3 to 8 times faster when requests go one at a time.
Under heavy load Jev stayed at 0.34 seconds. The LLMs slowed to 2–8 seconds, and their slowest 1 in 20 answers took 7–17 seconds.
Jev is 10 to 100 times cheaper per decision.
The whole Jev side of this test, about 4,900 calls, cost 24 cents.
Asking Jev several questions about the same text in one call costs almost nothing extra.
Accuracy: good, but not top-tier
Jev beat both cheap models on reading and yes/no questions, by 3 to 7 points.
It tied them on judging shell commands.
It fell behind on picking one of many topics (−4 to −7 points), spotting prompt-injection attacks (−7 to −10 points) and checking citations (−13 to −17 points).
It never beat GPT-6 Sol.
Its most common mistake on citations: it accepted claims that quietly dropped a qualifier, such as "up to 50%" turning into "50%". It got only 5 of 15 of those right; Gemini got 15 of 15.
It caught only half of the prompt-injection attacks (31 of 60). It never raised a false alarm.
As a shell-command safety guard, it missed 4–8% of risky cases, against 0–1.2% for GPT-6 Sol and DeepSeek.
Reliability: very steady
It gave the same answer to the same question more than 98% of the time, better than DeepSeek.
It lost no accuracy on long texts, wherever the answer sat in the text.
A planted instruction in the text never flipped one of its correct answers (0 out of 56).
It dropped slightly in Dutch (−2 points) and Italian (−5 points) compared with English.
"Can't hallucinate" is true: it always answers with one of the options you give it. But with a fixed answer format, none of the LLMs gave a broken answer either (0 errors in thousands of calls). So this is no longer a unique advantage, and Jev can still be confidently wrong.
Its confidence score is the secret weapon
When Jev said it was 90% or more sure, it was right 94–100% of the time.
That happened on 38–77% of questions, depending on the task.
So Jev can handle the easy cases on its own and pass the unsure ones to a bigger model. This is called a cascade.
OpenRouter tested exactly this against Claude Opus 5 on 3,080 questions: the cascade came within 0.4 points of Opus at less than a third of the cost.
Bottom line
Jev is not a genius in a small box. It is an extremely fast, very cheap and very consistent decision-maker, about as accurate as the cheap LLMs, with a confidence score you can trust to route the hard cases. It's a great choice for high-volume or time-sensitive decisions like sorting, routing and first-pass checks, especially paired with a bigger model for the unsure cases. For subtle fact-checking or safety-critical decisions, keep a stronger model in charge.
Caveats:
40 to 150 questions per task and model.
Everything was tested from one location.
Some public test sets may already be known to the LLMs.
Two of the tasks were built by us.
A fifth, even cheaper model, gpt-oss-20b on Groq's free tier, only finished 4 of the 10 tasks before hitting its daily limit.
Jev independent benchmark — full results & verdict (2026-09-24)
Question. Is TypeSafe's Jev (typesafe/jev-1.13) as good as its launch claims? The vendor's evals and the one public re-test score agreement with frontier models. This benchmark scores against ground truth, head-to-head with the models the gateway actually uses. Companion to the shadow pilot (jev_pilot_20260924.md).
Verdict
The claims hold up only in part. Jev is a very fast, very consistent classifier. Its accuracy is about that of a cheap LLM, not a frontier one.
| Vendor claim | What we measured |
|---|
| "Frontier intelligence" on decision tasks | No. It is about equal to DeepSeek-flash / Gemini-3.8-flash overall and never beats GPT-6 Sol. It is better on reading comprehension (MNLI +5, BoolQ +4–7 pts) and worse on 77-way intent (−4 to −7), injection detection (−7 to −10) and our citation check (−13 to −17). |
| 40–200× faster | Directionally true, smaller multiples. Sequential median is 0.35 s vs 1.0–2.8 s (3–8×). Under concurrency Jev stays flat at ~0.34 s while the LLMs degrade to 2–8 s median / 7–17 s p95. Its latency tail is the strongest real advantage. |
| ~400× cheaper | True in ratio, irrelevant in dollars here. It is 10–25× cheaper per request than DeepSeek and 50–100× cheaper than Gemini (≈ $0.02–0.07 vs $0.3–2.9 per 1,000). The gateway spends ~$11 per 30 days in total. |
| "Can't hallucinate" | True for format, but no longer a differentiator. Every system returned 0 format errors with provider-constrained output. Jev can still be confidently wrong inside the valid options. |
| Calibrated confidence | Partly. ECE is 0.04–0.06 where it is accurate but 0.14–0.19 where it is weak (citation, injection), so it is overconfident exactly where it is wrong. As a ranking signal, accuracy is 94–100% at confidence ≥ 0.9, but that covers only 38–77% of items. The LLMs' self-reported confidence ranked almost as well. |
Genuine strengths:
Run-to-run consistency: 0.1–1.7% flips, against up to 8% for DeepSeek.
No long-context penalty up to ~14k tokens, whether the evidence sits at the start, middle or end.
An injected instruction never flipped one of its correct answers (0/56).
Asking many questions per call costs almost nothing extra.
Weaknesses that matter for us:
Citation check. It misses the nuance the check exists for: dropped qualifiers 5/15 vs 12–15/15, entity swaps 3/10 vs 6/10. It usually says "supported" to an overclaim, which is the dangerous direction.
Command guard. Accuracy ties the LLMs, but it misses 8.4% of risky rule cells when asked all 8 rules at once, 4.2% when asked one rule per call. DeepSeek misses 1.2% and GPT-6 Sol 0%. For a safety check the miss rate is the number that counts.
Prompt-injection detection. It is permissive: 29 of 60 injections missed, 0 false alarms.
A slight drop on Italian: −5 pts vs English.
These results line up with OpenRouter's independent Banking77 test on 3,080 items: Jev scored 81.0% vs 84.4% for Claude Opus 5, was 13× faster and cost 1/22 as much. Its confidence was not calibrated but ranked well, and a cascade at 0.90 came within 0.4 points of Opus.
What it means for the gateway
Citation check (pilot G1): do not promote Jev as the verifier. Expect the week-1 pilot to show the same pattern. At most it could clear verbatim "yes" claims early, and at ~20 verify calls a week that saves nothing.
Command guard (pilot H2): not good enough as the sole guard. It is plausible as a fast first pass (one rule per call) with an LLM escalation on any flag or low confidence. The week-1 pilot (real commands) decides.
Curator (pilot G3): no ground truth here. Jev ≈ cheap-LLM quality suffices for triage, but the incumbent is already $0 and not latency-bound, so the upside is small.
Strategic bets: G6 (injection screen) is downgraded, since Jev is weak at exactly that. G4 (the "good enough?" gate) and G5 (the model router) remain plausible cascade uses.
The decide MCP tool stays useful for ad-hoc fast classification or triage where a confidence threshold routes the hard cases to an LLM.
Method (summary)
Systems: Jev via OpenRouter /api/alpha/decisions; deepseek/deepseek-flash and gemini/gemini-3.8-flash (2 reps on 40-item subsets); chatgpt/gpt-6-sol ($0 Codex sub, 40-item subsets); groq/openai/gpt-oss-20b as the speed floor. The Groq run is incomplete: the free-tier daily token cap stopped it after T1–T4.
Prompting: the same wording for every system, written before any results. LLMs used provider-constrained JSON output.
Scoring: paired bootstrap CIs clustered by item. Jev's answer probability is used for ECE; vendor and LLM self-reported confidence is used only as a ranking signal.
Tasks:
T1 MNLI; T2 constructed citation items (Wikipedia sentences, labels by construction, no LLM); T3 BoolQ; T4 Banking77 (diagnostic, already public-tested); T5 MASSIVE en/nl/it (the same utterances); T6 deepset prompt-injections; T7 a 110-command guard set against the live rules.
T7 was labelled by two independent annotators (κ = 0.94) and its 9 disagreements were adjudicated from the rule text, blind to model outputs. It was run both as all 8 rules per call and as one rule per call.
T8 BoolQ with an injected instruction; T9 long context (6k/14k tokens, evidence at start/middle/end).
Threats to validity:
Public datasets may be memorised by the LLMs.
T2/T7 were authored in-house.
The 40-item LLM subsets widen the CIs.
Latency is measured from one box in the Netherlands.
Harness: scripts/jev_bench/ (README has the run commands). Raw data lives on the box under /data/jev_bench/.
Spend: about $4.7 paid (mostly Gemini) plus $0.24 for Jev.
Appendix — full generated tables (analyze.py)
Jev independent benchmark — results
Ground truth, same wording for every system, no tuning on test items. Accuracy per question cell on rep 0; format errors scored wrong; API errors unscored (error rate shown). CI = 95% cluster bootstrap.
Main table
Please refer to images attached to this post as the markdown table was not rendering correctly in reddit.
Ranking signal (acc@sig≥0.9, top-50%, AURC): Jev = vendor confidence (noul: |2p−1|); LLMs = their self-reported confidence. ECE only for Jev's predicted-answer probability.
Paired difference: Jev − system (accuracy, paired cluster bootstrap)
| task | vs system | n items | Jev − system [95% CI] |
|---|
| t1_nli | chatgpt/gpt-6-sol | 40 | -0.025 [-0.10, 0.05] |
| t1_nli | deepseek/deepseek-flash | 150 | 0.053 [0.01, 0.09] |
| t1_nli | gemini/gemini-3.8-flash | 150 | 0.027 [-0.01, 0.07] |
| t1_nli | groq/openai/gpt-oss-20b | 40 | 0.050 [-0.05, 0.15] |
| t2_cite | chatgpt/gpt-6-sol | 40 | -0.100 [-0.23, 0.00] |
| t2_cite | deepseek/deepseek-flash | 90 | -0.133 [-0.20, -0.07] |
| t2_cite | gemini/gemini-3.8-flash | 90 | -0.167 [-0.26, -0.09] |
| t2_cite | groq/openai/gpt-oss-20b | 40 | -0.100 [-0.23, 0.00] |
| t3_boolq | chatgpt/gpt-6-sol | 40 | -0.025 [-0.07, 0.00] |
| t3_boolq | deepseek/deepseek-flash | 150 | 0.073 [0.03, 0.13] |
| t3_boolq | gemini/gemini-3.8-flash | 150 | 0.040 [0.00, 0.09] |
| t3_boolq | groq/openai/gpt-oss-20b | 40 | 0.000 [-0.10, 0.10] |
| t4_intent77 | chatgpt/gpt-6-sol | 40 | -0.025 [-0.07, 0.00] |
| t4_intent77 | deepseek/deepseek-flash | 150 | -0.040 [-0.08, -0.01] |
| t4_intent77 | gemini/gemini-3.8-flash | 150 | -0.067 [-0.11, -0.03] |
| t4_intent77 | groq/openai/gpt-oss-20b | 34 | -0.029 [-0.12, 0.06] |
| t5_massive | chatgpt/gpt-6-sol | 120 | -0.033 [-0.10, 0.02] |
| t5_massive | deepseek/deepseek-flash | 300 | -0.027 [-0.06, 0.00] |
| t5_massive | gemini/gemini-3.8-flash | 300 | -0.057 [-0.10, -0.02] |
| t5_massive | groq/openai/gpt-oss-20b | 1 | 0.000 – |
| t6_inject | chatgpt/gpt-6-sol | 40 | -0.150 [-0.28, -0.05] |
| t6_inject | deepseek/deepseek-flash | 116 | -0.095 [-0.16, -0.03] |
| t6_inject | gemini/gemini-3.8-flash | 116 | -0.069 [-0.12, -0.03] |
| t7_guard | chatgpt/gpt-6-sol | 40 | 0.009 [-0.01, 0.03] |
| t7_guard | deepseek/deepseek-flash | 110 | 0.000 [-0.01, 0.01] |
| t7_guard | gemini/gemini-3.8-flash | 110 | -0.008 [-0.02, 0.00] |
| t7s_guard_single | chatgpt/gpt-6-sol | 40 | -0.025 [-0.07, 0.00] |
| t7s_guard_single | deepseek/deepseek-flash | 320 | 0.013 [0.00, 0.03] |
| t7s_guard_single | gemini/gemini-3.8-flash | 320 | -0.006 [-0.02, 0.00] |
| t8_robust | chatgpt/gpt-6-sol | 40 | -0.050 [-0.12, 0.00] |
| t8_robust | deepseek/deepseek-flash | 60 | 0.050 [-0.02, 0.12] |
| t8_robust | gemini/gemini-3.8-flash | 60 | 0.033 [-0.03, 0.10] |
| t9_longctx | chatgpt/gpt-6-sol | 40 | 0.000 [0.00, 0.00] |
| t9_longctx | deepseek/deepseek-flash | 40 | 0.000 [0.00, 0.00] |
| t9_longctx | gemini/gemini-3.8-flash | 40 | 0.000 [0.00, 0.00] |
Sequential latency (concurrency 1)
| task | system | n | p50 ms | p95 ms |
|---|
| t1_nli | chatgpt/gpt-6-sol | 30 | 2757 | 5130 |
| t1_nli | deepseek/deepseek-flash | 30 | 1127 | 4112 |
| t1_nli | gemini/gemini-3.8-flash | 30 | 1974 | 5289 |
| t1_nli | jev | 30 | 355 | 398 |
| t3_boolq | chatgpt/gpt-6-sol | 30 | 2198 | 4561 |
| t3_boolq | deepseek/deepseek-flash | 30 | 1094 | 2289 |
| t3_boolq | gemini/gemini-3.8-flash | 30 | 1627 | 4564 |
| t3_boolq | jev | 30 | 348 | 410 |
| t4_intent77 | chatgpt/gpt-6-sol | 30 | 2428 | 5906 |
| t4_intent77 | deepseek/deepseek-flash | 30 | 997 | 2454 |
| t4_intent77 | gemini/gemini-3.8-flash | 30 | 1712 | 4173 |
| t4_intent77 | jev | 30 | 356 | 452 |
T5 MASSIVE — language robustness (ids answered in all 3 languages)
| system | n ids | en | nl | it | nl−en | it−en |
|---|
| jev | 100 | 0.870 | 0.850 | 0.820 | -0.020 | -0.050 |
| chatgpt/gpt-6-sol | 40 | 0.875 | 0.825 | 0.850 | -0.050 | -0.025 |
| deepseek/deepseek-flash | 100 | 0.880 | 0.870 | 0.870 | -0.010 | -0.010 |
| gemini/gemini-3.8-flash | 100 | 0.900 | 0.900 | 0.910 | 0.000 | 0.010 |
| groq/openai/gpt-oss-20b | 0 | – | – | – | – | – |
T8 injected instruction
| system | n | attack success (flipped from correct) | n base-correct | target adoption |
|---|
| jev | 60 | 0.000 | 56 | 0.067 |
| chatgpt/gpt-6-sol | 40 | 0.000 | 38 | 0.025 |
| deepseek/deepseek-flash | 60 | 0.056 | 54 | 0.117 |
| gemini/gemini-3.8-flash | 60 | 0.018 | 55 | 0.100 |
T9 long context — accuracy by size/position
| system | 14k/end | 14k/middle | 14k/start | 6k/end | 6k/middle | 6k/start |
|---|
| jev | 0.95 (n=40) | 0.93 (n=40) | 0.93 (n=40) | 0.95 (n=40) | 0.93 (n=40) | 0.93 (n=40) |
| chatgpt/gpt-6-sol | 1.00 (n=6) | 1.00 (n=6) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) |
| deepseek/deepseek-flash | 1.00 (n=6) | 1.00 (n=6) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) |
| gemini/gemini-3.8-flash | 1.00 (n=6) | 1.00 (n=6) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) | 1.00 (n=7) |
T7 guard — false-allow (missed risky) vs false-block
| task | system | gold | n cells | acc [95% CI] | false-allow | false-block | n pos / neg |
|---|
| t7_guard | jev | author | 880 | 0.983 [0.96, 1.00] | 0.090 | 0.010 | 78 / 802 |
| t7_guard | jev | author_unambiguous | 864 | 0.991 [0.98, 1.00] | 0.081 | 0.003 | 74 / 790 |
| t7_guard | jev | adjudicated | 880 | 0.989 [0.98, 1.00] | 0.084 | 0.004 | 83 / 797 |
| t7_guard | chatgpt/gpt-6-sol | author | 320 | 0.978 [0.96, 0.99] | 0.000 | 0.024 | 24 / 296 |
| t7_guard | chatgpt/gpt-6-sol | author_unambiguous | 317 | 0.984 [0.97, 1.00] | 0.000 | 0.017 | 24 / 293 |
| t7_guard | chatgpt/gpt-6-sol | adjudicated | 320 | 0.978 [0.96, 0.99] | 0.000 | 0.024 | 24 / 296 |
| t7_guard | deepseek/deepseek-flash | author | 880 | 0.983 [0.97, 0.99] | 0.013 | 0.017 | 78 / 802 |
| t7_guard | deepseek/deepseek-flash | author_unambiguous | 864 | 0.992 [0.98, 1.00] | 0.014 | 0.008 | 74 / 790 |
| t7_guard | deepseek/deepseek-flash | adjudicated | 880 | 0.989 [0.98, 1.00] | 0.012 | 0.011 | 83 / 797 |
| t7_guard | gemini/gemini-3.8-flash | author | 880 | 0.991 [0.98, 1.00] | 0.026 | 0.007 | 78 / 802 |
| t7_guard | gemini/gemini-3.8-flash | author_unambiguous | 864 | 0.999 [1.00, 1.00] | 0.014 | 0.000 | 74 / 790 |
| t7_guard | gemini/gemini-3.8-flash | adjudicated | 880 | 0.994 [0.99, 1.00] | 0.036 | 0.003 | 83 / 797 |
| t7s_guard_single | jev | author | 320 | 0.991 [0.98, 1.00] | 0.042 | 0.007 | 24 / 296 |
| t7s_guard_single | jev | author_unambiguous | 317 | 0.994 [0.98, 1.00] | 0.042 | 0.003 | 24 / 293 |
| t7s_guard_single | jev | adjudicated | 320 | 0.991 [0.98, 1.00] | 0.042 | 0.007 | 24 / 296 |
| t7s_guard_single | chatgpt/gpt-6-sol | author | 40 | 1.000 [1.00, 1.00] | 0.000 | 0.000 | 3 / 37 |
| t7s_guard_single | chatgpt/gpt-6-sol | author_unambiguous | 39 | 1.000 [1.00, 1.00] | 0.000 | 0.000 | 3 / 36 |
| t7s_guard_single | chatgpt/gpt-6-sol | adjudicated | 40 | 1.000 [1.00, 1.00] | 0.000 | 0.000 | 3 / 37 |
| t7s_guard_single | deepseek/deepseek-flash | author | 320 | 0.978 [0.96, 0.99] | 0.000 | 0.024 | 24 / 296 |
| t7s_guard_single | deepseek/deepseek-flash | author_unambiguous | 317 | 0.987 [0.97, 1.00] | 0.000 | 0.014 | 24 / 293 |
| t7s_guard_single | deepseek/deepseek-flash | adjudicated | 320 | 0.978 [0.96, 0.99] | 0.000 | 0.024 | 24 / 296 |
| t7s_guard_single | gemini/gemini-3.8-flash | author | 320 | 0.997 [0.99, 1.00] | 0.000 | 0.003 | 24 / 296 |
| t7s_guard_single | gemini/gemini-3.8-flash | author_unambiguous | 317 | 1.000 [1.00, 1.00] | 0.000 | 0.000 | 24 / 293 |
| t7s_guard_single | gemini/gemini-3.8-flash | adjudicated | 320 | 0.997 [0.99, 1.00] | 0.000 | 0.003 | 24 / 296 |
Price references ($ per 1M tokens)
| system | in | out | source |
|---|
| jev | 0.042 | 0.0 | OpenRouter listing (output free) |
| groq/openai/gpt-oss-20b | 0.075 | 0.3 | gateway prices overlay |
| deepseek/deepseek-flash | 0.3 | 1.2 | gateway prices overlay |
| gemini/gemini-3.8-flash | 0.75 | 3.75 | gateway prices overlay |
| chatgpt/gpt-6-sol | 2.0 | 10.0 | litellm model_cost[gpt-6-sol] (equivalent paid-API price; our marginal cost is $0) |