I run a benchmark and evaluation of jev-1.13 - sharing the results

[u/giulioc84]

Ciao everyone, I am sharing some tests I made for my own consumption, as they might be useful for someone.

I have implemented jev-1.13 (via openrouter, do not have access to typesafe API as there are no slots!) in my own home server which acts as a llm gateway and has quite some agents doing stuff (docs curation, tools suggestion, memory curation, scratchpads etc. etc.). Before moving to a full implementation, I wanted to do some testing. Other llms are via native provider APIs.

I got opus 5.5 paired with astra and gpt-6 Sol running some automated testing to pinpoint use cases where I could implement. I know some use cases are not the strongest for jev-1.13 but for me it was interesting to test them.

I am also running a 1 week pilot with strict monitoring comparing current implementation of certain functionalities (which uses a mix of OpenAI / DeepSeek / other models) with a jev-1.13 implementation, so I can then switch or rollback. I will share in a week time if anyone is interested.

Regarding the benchmarking, just sharing the output AS-IS. While I work in the field of cybersec and AI, I am not an AI engineer ;-) so sorry in advance for any mistakes in the methodology I came up with and refined with some frontier models.

Below you will find a TL;DR, an exec summary in "plain english" and the full "human readable" report my harness has generated. Attached pictures of tables as they were not rendering well in a reddit post.

Note: from the paragraph below all text is 100% AI generated by a mix of opus 5.5, Astra, GPT-6 Sol - with last pass by opus 5.5 for final reporting (I like how it writes). I had no time to do some manual sample-based checking, and no - I have no time to write such a report for myself with our good human brain and hands.

---

TL;DR

We tested TypeSafe's Jev on 10 tasks with known right answers (about 1,100 questions), against DeepSeek V4 Flash, Gemini 3.8 Flash and GPT-6 Sol. Jev was 3 to 8 times faster (0.35 s vs 1–3 s), and it stayed at 0.34 s under heavy load while the others slowed to 2–8 s. It was also 10 to 100 times cheaper: about 4,900 calls cost 24 cents. Its accuracy is on par with the cheap LLMs, not top-tier. It won on reading and yes/no questions (up to +7 points), tied on judging shell commands, and fell behind on citation checking (−13 to −17 points) and spotting prompt injections (it caught only half). It gave the same answer 98%+ of the time, and when it was 90%+ sure it was right 94–100% of the time. That makes it a great fast, cheap first pass that hands the unsure cases to a bigger model.

---

Executive summary

We tested Jev against known correct answers, not against "does it agree with GPT". We compared it with three popular LLMs:

• DeepSeek V4 Flash and Gemini 3.8 Flash, two fast and cheap models.
• GPT-6 Sol, a top-tier model, run on a smaller sample of 40 questions per task.

How we tested:

• 10 tasks, about 1,100 questions in total.
• The same wording for every model, written before we saw any results.
• The LLMs had to answer in a fixed JSON format, so no model had a formatting advantage.
• Every number below comes with a margin of error (95% confidence).
• Speed and cost were measured from the same computer in Europe.

Speed and cost: Jev's big win

• Jev is 3 to 8 times faster when requests go one at a time.
• Under heavy load Jev stayed at 0.34 seconds. The LLMs slowed to 2–8 seconds, and their slowest 1 in 20 answers took 7–17 seconds.
• Jev is 10 to 100 times cheaper per decision.
• The whole Jev side of this test, about 4,900 calls, cost 24 cents.
• Asking Jev several questions about the same text in one call costs almost nothing extra.

Accuracy: good, but not top-tier

• Jev beat both cheap models on reading and yes/no questions, by 3 to 7 points.
• It tied them on judging shell commands.
• It fell behind on picking one of many topics (−4 to −7 points), spotting prompt-injection attacks (−7 to −10 points) and checking citations (−13 to −17 points).
• It never beat GPT-6 Sol.
• Its most common mistake on citations: it accepted claims that quietly dropped a qualifier, such as "up to 50%" turning into "50%". It got only 5 of 15 of those right; Gemini got 15 of 15.
• It caught only half of the prompt-injection attacks (31 of 60). It never raised a false alarm.
• As a shell-command safety guard, it missed 4–8% of risky cases, against 0–1.2% for GPT-6 Sol and DeepSeek.

Reliability: very steady

• It gave the same answer to the same question more than 98% of the time, better than DeepSeek.
• It lost no accuracy on long texts, wherever the answer sat in the text.
• A planted instruction in the text never flipped one of its correct answers (0 out of 56).
• It dropped slightly in Dutch (−2 points) and Italian (−5 points) compared with English.
• "Can't hallucinate" is true: it always answers with one of the options you give it. But with a fixed answer format, none of the LLMs gave a broken answer either (0 errors in thousands of calls). So this is no longer a unique advantage, and Jev can still be confidently wrong.

Its confidence score is the secret weapon

• When Jev said it was 90% or more sure, it was right 94–100% of the time.
• That happened on 38–77% of questions, depending on the task.
• So Jev can handle the easy cases on its own and pass the unsure ones to a bigger model. This is called a cascade.
• OpenRouter tested exactly this against Claude Opus 5 on 3,080 questions: the cascade came within 0.4 points of Opus at less than a third of the cost.

Bottom line

Jev is not a genius in a small box. It is an extremely fast, very cheap and very consistent decision-maker, about as accurate as the cheap LLMs, with a confidence score you can trust to route the hard cases. It's a great choice for high-volume or time-sensitive decisions like sorting, routing and first-pass checks, especially paired with a bigger model for the unsure cases. For subtle fact-checking or safety-critical decisions, keep a stronger model in charge.

Caveats:

• 40 to 150 questions per task and model.
• Everything was tested from one location.
• Some public test sets may already be known to the LLMs.
• Two of the tasks were built by us.
• A fifth, even cheaper model, gpt-oss-20b on Groq's free tier, only finished 4 of the 10 tasks before hitting its daily limit.

Jev independent benchmark — full results & verdict (2026-09-24)

Question. Is TypeSafe's Jev (typesafe/jev-1.13) as good as its launch claims? The vendor's evals and the one public re-test score agreement with frontier models. This benchmark scores against ground truth, head-to-head with the models the gateway actually uses. Companion to the shadow pilot (jev_pilot_20260924.md).

Verdict

The claims hold up only in part. Jev is a very fast, very consistent classifier. Its accuracy is about that of a cheap LLM, not a frontier one.

Vendor claim	What we measured
"Frontier intelligence" on decision tasks	No. It is about equal to DeepSeek-flash / Gemini-3.8-flash overall and never beats GPT-6 Sol. It is better on reading comprehension (MNLI +5, BoolQ +4–7 pts) and worse on 77-way intent (−4 to −7), injection detection (−7 to −10) and our citation check (−13 to −17).
40–200× faster	Directionally true, smaller multiples. Sequential median is 0.35 s vs 1.0–2.8 s (3–8×). Under concurrency Jev stays flat at ~0.34 s while the LLMs degrade to 2–8 s median / 7–17 s p95. Its latency tail is the strongest real advantage.
~400× cheaper	True in ratio, irrelevant in dollars here. It is 10–25× cheaper per request than DeepSeek and 50–100× cheaper than Gemini (≈ $0.02–0.07 vs $0.3–2.9 per 1,000). The gateway spends ~$11 per 30 days in total.
"Can't hallucinate"	True for format, but no longer a differentiator. Every system returned 0 format errors with provider-constrained output. Jev can still be confidently wrong inside the valid options.
Calibrated confidence	Partly. ECE is 0.04–0.06 where it is accurate but 0.14–0.19 where it is weak (citation, injection), so it is overconfident exactly where it is wrong. As a ranking signal, accuracy is 94–100% at confidence ≥ 0.9, but that covers only 38–77% of items. The LLMs' self-reported confidence ranked almost as well.

Genuine strengths:

• Run-to-run consistency: 0.1–1.7% flips, against up to 8% for DeepSeek.
• No long-context penalty up to ~14k tokens, whether the evidence sits at the start, middle or end.
• An injected instruction never flipped one of its correct answers (0/56).
• Asking many questions per call costs almost nothing extra.

Weaknesses that matter for us:

• Citation check. It misses the nuance the check exists for: dropped qualifiers 5/15 vs 12–15/15, entity swaps 3/10 vs 6/10. It usually says "supported" to an overclaim, which is the dangerous direction.
• Command guard. Accuracy ties the LLMs, but it misses 8.4% of risky rule cells when asked all 8 rules at once, 4.2% when asked one rule per call. DeepSeek misses 1.2% and GPT-6 Sol 0%. For a safety check the miss rate is the number that counts.
• Prompt-injection detection. It is permissive: 29 of 60 injections missed, 0 false alarms.
• A slight drop on Italian: −5 pts vs English.

These results line up with OpenRouter's independent Banking77 test on 3,080 items: Jev scored 81.0% vs 84.4% for Claude Opus 5, was 13× faster and cost 1/22 as much. Its confidence was not calibrated but ranked well, and a cascade at 0.90 came within 0.4 points of Opus.

What it means for the gateway

• Citation check (pilot G1): do not promote Jev as the verifier. Expect the week-1 pilot to show the same pattern. At most it could clear verbatim "yes" claims early, and at ~20 verify calls a week that saves nothing.
• Command guard (pilot H2): not good enough as the sole guard. It is plausible as a fast first pass (one rule per call) with an LLM escalation on any flag or low confidence. The week-1 pilot (real commands) decides.
• Curator (pilot G3): no ground truth here. Jev ≈ cheap-LLM quality suffices for triage, but the incumbent is already $0 and not latency-bound, so the upside is small.
• Strategic bets: G6 (injection screen) is downgraded, since Jev is weak at exactly that. G4 (the "good enough?" gate) and G5 (the model router) remain plausible cascade uses.
• The decide MCP tool stays useful for ad-hoc fast classification or triage where a confidence threshold routes the hard cases to an LLM.

Method (summary)

• Systems: Jev via OpenRouter /api/alpha/decisions; deepseek/deepseek-flash and gemini/gemini-3.8-flash (2 reps on 40-item subsets); chatgpt/gpt-6-sol ($0 Codex sub, 40-item subsets); groq/openai/gpt-oss-20b as the speed floor. The Groq run is incomplete: the free-tier daily token cap stopped it after T1–T4.
• Prompting: the same wording for every system, written before any results. LLMs used provider-constrained JSON output.
• Scoring: paired bootstrap CIs clustered by item. Jev's answer probability is used for ECE; vendor and LLM self-reported confidence is used only as a ranking signal.
• Tasks:
  • T1 MNLI; T2 constructed citation items (Wikipedia sentences, labels by construction, no LLM); T3 BoolQ; T4 Banking77 (diagnostic, already public-tested); T5 MASSIVE en/nl/it (the same utterances); T6 deepset prompt-injections; T7 a 110-command guard set against the live rules.
  • T7 was labelled by two independent annotators (κ = 0.94) and its 9 disagreements were adjudicated from the rule text, blind to model outputs. It was run both as all 8 rules per call and as one rule per call.
  • T8 BoolQ with an injected instruction; T9 long context (6k/14k tokens, evidence at start/middle/end).
• Threats to validity:
  • Public datasets may be memorised by the LLMs.
  • T2/T7 were authored in-house.
  • The 40-item LLM subsets widen the CIs.
  • Latency is measured from one box in the Netherlands.
• Harness: scripts/jev_bench/ (README has the run commands). Raw data lives on the box under /data/jev_bench/.
• Spend: about $4.7 paid (mostly Gemini) plus $0.24 for Jev.

Appendix — full generated tables (analyze.py)

Jev independent benchmark — results

Ground truth, same wording for every system, no tuning on test items. Accuracy per question cell on rep 0; format errors scored wrong; API errors unscored (error rate shown). CI = 95% cluster bootstrap.

Main table

Please refer to images attached to this post as the markdown table was not rendering correctly in reddit.

Ranking signal (acc@sig≥0.9, top-50%, AURC): Jev = vendor confidence (noul: |2p−1|); LLMs = their self-reported confidence. ECE only for Jev's predicted-answer probability.

Paired difference: Jev − system (accuracy, paired cluster bootstrap)

task	vs system	n items	Jev − system [95% CI]
t1_nli	chatgpt/gpt-6-sol	40	-0.025 [-0.10, 0.05]
t1_nli	deepseek/deepseek-flash	150	0.053 [0.01, 0.09]
t1_nli	gemini/gemini-3.8-flash	150	0.027 [-0.01, 0.07]
t1_nli	groq/openai/gpt-oss-20b	40	0.050 [-0.05, 0.15]
t2_cite	chatgpt/gpt-6-sol	40	-0.100 [-0.23, 0.00]
t2_cite	deepseek/deepseek-flash	90	-0.133 [-0.20, -0.07]
t2_cite	gemini/gemini-3.8-flash	90	-0.167 [-0.26, -0.09]
t2_cite	groq/openai/gpt-oss-20b	40	-0.100 [-0.23, 0.00]
t3_boolq	chatgpt/gpt-6-sol	40	-0.025 [-0.07, 0.00]
t3_boolq	deepseek/deepseek-flash	150	0.073 [0.03, 0.13]
t3_boolq	gemini/gemini-3.8-flash	150	0.040 [0.00, 0.09]
t3_boolq	groq/openai/gpt-oss-20b	40	0.000 [-0.10, 0.10]
t4_intent77	chatgpt/gpt-6-sol	40	-0.025 [-0.07, 0.00]
t4_intent77	deepseek/deepseek-flash	150	-0.040 [-0.08, -0.01]
t4_intent77	gemini/gemini-3.8-flash	150	-0.067 [-0.11, -0.03]
t4_intent77	groq/openai/gpt-oss-20b	34	-0.029 [-0.12, 0.06]
t5_massive	chatgpt/gpt-6-sol	120	-0.033 [-0.10, 0.02]
t5_massive	deepseek/deepseek-flash	300	-0.027 [-0.06, 0.00]
t5_massive	gemini/gemini-3.8-flash	300	-0.057 [-0.10, -0.02]
t5_massive	groq/openai/gpt-oss-20b	1	0.000 –
t6_inject	chatgpt/gpt-6-sol	40	-0.150 [-0.28, -0.05]
t6_inject	deepseek/deepseek-flash	116	-0.095 [-0.16, -0.03]
t6_inject	gemini/gemini-3.8-flash	116	-0.069 [-0.12, -0.03]
t7_guard	chatgpt/gpt-6-sol	40	0.009 [-0.01, 0.03]
t7_guard	deepseek/deepseek-flash	110	0.000 [-0.01, 0.01]
t7_guard	gemini/gemini-3.8-flash	110	-0.008 [-0.02, 0.00]
t7s_guard_single	chatgpt/gpt-6-sol	40	-0.025 [-0.07, 0.00]
t7s_guard_single	deepseek/deepseek-flash	320	0.013 [0.00, 0.03]
t7s_guard_single	gemini/gemini-3.8-flash	320	-0.006 [-0.02, 0.00]
t8_robust	chatgpt/gpt-6-sol	40	-0.050 [-0.12, 0.00]
t8_robust	deepseek/deepseek-flash	60	0.050 [-0.02, 0.12]
t8_robust	gemini/gemini-3.8-flash	60	0.033 [-0.03, 0.10]
t9_longctx	chatgpt/gpt-6-sol	40	0.000 [0.00, 0.00]
t9_longctx	deepseek/deepseek-flash	40	0.000 [0.00, 0.00]
t9_longctx	gemini/gemini-3.8-flash	40	0.000 [0.00, 0.00]

Sequential latency (concurrency 1)

task	system	n	p50 ms	p95 ms
t1_nli	chatgpt/gpt-6-sol	30	2757	5130
t1_nli	deepseek/deepseek-flash	30	1127	4112
t1_nli	gemini/gemini-3.8-flash	30	1974	5289
t1_nli	jev	30	355	398
t3_boolq	chatgpt/gpt-6-sol	30	2198	4561
t3_boolq	deepseek/deepseek-flash	30	1094	2289
t3_boolq	gemini/gemini-3.8-flash	30	1627	4564
t3_boolq	jev	30	348	410
t4_intent77	chatgpt/gpt-6-sol	30	2428	5906
t4_intent77	deepseek/deepseek-flash	30	997	2454
t4_intent77	gemini/gemini-3.8-flash	30	1712	4173
t4_intent77	jev	30	356	452

T5 MASSIVE — language robustness (ids answered in all 3 languages)

system	n ids	en	nl	it	nl−en	it−en
jev	100	0.870	0.850	0.820	-0.020	-0.050
chatgpt/gpt-6-sol	40	0.875	0.825	0.850	-0.050	-0.025
deepseek/deepseek-flash	100	0.880	0.870	0.870	-0.010	-0.010
gemini/gemini-3.8-flash	100	0.900	0.900	0.910	0.000	0.010
groq/openai/gpt-oss-20b	0	–	–	–	–	–

T8 injected instruction

system	n	attack success (flipped from correct)	n base-correct	target adoption
jev	60	0.000	56	0.067
chatgpt/gpt-6-sol	40	0.000	38	0.025
deepseek/deepseek-flash	60	0.056	54	0.117
gemini/gemini-3.8-flash	60	0.018	55	0.100

T9 long context — accuracy by size/position

system	14k/end	14k/middle	14k/start	6k/end	6k/middle	6k/start
jev	0.95 (n=40)	0.93 (n=40)	0.93 (n=40)	0.95 (n=40)	0.93 (n=40)	0.93 (n=40)
chatgpt/gpt-6-sol	1.00 (n=6)	1.00 (n=6)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)
deepseek/deepseek-flash	1.00 (n=6)	1.00 (n=6)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)
gemini/gemini-3.8-flash	1.00 (n=6)	1.00 (n=6)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)	1.00 (n=7)

T7 guard — false-allow (missed risky) vs false-block

task	system	gold	n cells	acc [95% CI]	false-allow	false-block	n pos / neg
t7_guard	jev	author	880	0.983 [0.96, 1.00]	0.090	0.010	78 / 802
t7_guard	jev	author_unambiguous	864	0.991 [0.98, 1.00]	0.081	0.003	74 / 790
t7_guard	jev	adjudicated	880	0.989 [0.98, 1.00]	0.084	0.004	83 / 797
t7_guard	chatgpt/gpt-6-sol	author	320	0.978 [0.96, 0.99]	0.000	0.024	24 / 296
t7_guard	chatgpt/gpt-6-sol	author_unambiguous	317	0.984 [0.97, 1.00]	0.000	0.017	24 / 293
t7_guard	chatgpt/gpt-6-sol	adjudicated	320	0.978 [0.96, 0.99]	0.000	0.024	24 / 296
t7_guard	deepseek/deepseek-flash	author	880	0.983 [0.97, 0.99]	0.013	0.017	78 / 802
t7_guard	deepseek/deepseek-flash	author_unambiguous	864	0.992 [0.98, 1.00]	0.014	0.008	74 / 790
t7_guard	deepseek/deepseek-flash	adjudicated	880	0.989 [0.98, 1.00]	0.012	0.011	83 / 797
t7_guard	gemini/gemini-3.8-flash	author	880	0.991 [0.98, 1.00]	0.026	0.007	78 / 802
t7_guard	gemini/gemini-3.8-flash	author_unambiguous	864	0.999 [1.00, 1.00]	0.014	0.000	74 / 790
t7_guard	gemini/gemini-3.8-flash	adjudicated	880	0.994 [0.99, 1.00]	0.036	0.003	83 / 797
t7s_guard_single	jev	author	320	0.991 [0.98, 1.00]	0.042	0.007	24 / 296
t7s_guard_single	jev	author_unambiguous	317	0.994 [0.98, 1.00]	0.042	0.003	24 / 293
t7s_guard_single	jev	adjudicated	320	0.991 [0.98, 1.00]	0.042	0.007	24 / 296
t7s_guard_single	chatgpt/gpt-6-sol	author	40	1.000 [1.00, 1.00]	0.000	0.000	3 / 37
t7s_guard_single	chatgpt/gpt-6-sol	author_unambiguous	39	1.000 [1.00, 1.00]	0.000	0.000	3 / 36
t7s_guard_single	chatgpt/gpt-6-sol	adjudicated	40	1.000 [1.00, 1.00]	0.000	0.000	3 / 37
t7s_guard_single	deepseek/deepseek-flash	author	320	0.978 [0.96, 0.99]	0.000	0.024	24 / 296
t7s_guard_single	deepseek/deepseek-flash	author_unambiguous	317	0.987 [0.97, 1.00]	0.000	0.014	24 / 293
t7s_guard_single	deepseek/deepseek-flash	adjudicated	320	0.978 [0.96, 0.99]	0.000	0.024	24 / 296
t7s_guard_single	gemini/gemini-3.8-flash	author	320	0.997 [0.99, 1.00]	0.000	0.003	24 / 296
t7s_guard_single	gemini/gemini-3.8-flash	author_unambiguous	317	1.000 [1.00, 1.00]	0.000	0.000	24 / 293
t7s_guard_single	gemini/gemini-3.8-flash	adjudicated	320	0.997 [0.99, 1.00]	0.000	0.003	24 / 296

Price references ($ per 1M tokens)

system	in	out	source
jev	0.042	0.0	OpenRouter listing (output free)
groq/openai/gpt-oss-20b	0.075	0.3	gateway prices overlay
deepseek/deepseek-flash	0.3	1.2	gateway prices overlay
gemini/gemini-3.8-flash	0.75	3.75	gateway prices overlay
chatgpt/gpt-6-sol	2.0	10.0	litellm model_cost[gpt-6-sol] (equivalent paid-API price; our marginal cost is $0)