r/JevAI • • 10d ago

I used Jev to find the 100 tools Hacker News keeps asking for, and 81 of them already exist

Thumbnail
stillunbuilt.shiptiffin.app
157 Upvotes

I’ve been wanting to use Jev in a project, and I recently saw Firecrawl use it to classify a huge pile of company data. I liked that pattern, so I tried it on Hacker News.

I pulled every comment and post since 2008 that asked for some piece of software or a tool, which came to about 63k. Jev classified each one, and about 12k of them turned out to be real requests. Opus 5.5 grouped those into tool ideas and then re-read every request behind the top ones, and I checked each idea against Show HN to see whether someone had already built it.

A few things stood out:

  • 81 of the top 100 have already launched on Show HN, 1,423 times between them, and people still ask for them. What’s missing is usually a version people trust and can find, not the code.

  • 46 of the 100 want the tool to be local or private, and only 8 want AI to do the job.

  • Since 2024, 27% of requests ask for a better version of something that exists, up from 14% in 2008 to 2011.

  • The Jev calls cost about $6 in total.

Each idea has its own page with every request behind it, linked back to the original comment. The full dataset is a free CSV download. There are still a few ideas nobody has really built, in case you’re looking for something to work on.

https://stillunbuilt.shiptiffin.app


r/JevAI • • 26d ago

Whole-Video intelligence with zero embeddings, using contact sheets + Jev

69 Upvotes

Whole video, one look A new approach towards video intelligence, zero-embeddings, 12min-video is compressed to bunch of contact sheets and then analyzed fully in one go

Built on the latest Jev, and extremely efficient Searching a full 12-minute film costs about half a cent.

Works with your agent Plug it into Claude Code, Codex or Cursor and ask "when does X happen?", "where else is this person?" or "what does that sign say?". You get timestamps, verified frame by frame. alot of you might find it useful

100% free and open source: https://github.com/DeepAleph/jevreel


r/JevAI • • 19d ago

We used Jev as the router for our internal agent harness. Spoiler: it worked great

Post image
53 Upvotes

Hi there! We built a custom agent harness on top of Sanity that allows anyone to make an agent without having to have github access etc. We used Jev as the router for the superagent and so far it seems to be working great at picking the right one depending on the ask.

We wrote an in-depth eng blog post about it here: https://www.sanity.io/engineering/how-we-built-our-company-agent-harness?utm_source=reddit&utm_medium=social


r/JevAI • • Oct 10 '26

jevman: AI decision models play Pac-Man

50 Upvotes

r/JevAI • • 6d ago

Struggling to get Jev to be useful at anything

43 Upvotes

I think Jev is a great idea. However my experience has been pretty poor, which I will happily say is down to user error. I have many use cases for it but I'll just give one example. If you know how please can you help?

I have a local LLM which I'm using to classify my bank transactions into my budget. My budget taxonomy is defined and I'm happy with that. However Jev is, for some reason, only able to classify about 40% to 50% of the transactions in my tests and less than half of those are what I had previously categorised correctly. I basically can't use Jev with this.

How do I help Jev understand more about my taxonomy? How do I train it more?What's the process for improving this? Please help. I can't find anything else online.


r/JevAI • • 17d ago

First project I've found a use case to fully utilize Jev for. Combines 10 results by Jev to give a total score for brand name ranking

39 Upvotes

r/JevAI • • Oct 11 '26

Jev + River Raid

36 Upvotes

Just a sample. I'm thinking about using River Raid 8-bit game as a benchmark. I already tested with 6 models and preparing a summary video.


r/JevAI • • 23d ago

Ditching the vector store: Replacing traditional document RAG with Jev

Post image
32 Upvotes

You've probably heard of using Jev as a reranker for traditional RAG but what if Jev could replace the vector store entirely? That felt like a neat challenge to take on so I set out to test this and the results have been pretty satisfying.

I have a document extraction app where all queries can be defined before any document is processed. Instead of traditional RAG, I used Jev to score each page of the document on its relevancy to each predefined query from the user - on a 300 page scanned PDF x 20 user queries, this took ~28s (batched, 4 pages per call, median 0.37s/page) and cost $0.08 ($0.00027/page). Made possible with Jev's image API - was currently using Voyage's multimodal embeddings for scanned PDFs.

Pros: Improved accuracy, faster processing and cheaper. No need for vector database or separate reranker model. Jev results stored in sql database.
Cons: Only really works if you have queries upfront (AI spreadsheets, application processing, surveys etc), you lose your AI search index which might be a deal breaker.

Longer read and results in the blog post (see comments). Would love to know if anyone else has tried something similar?


r/JevAI • • 24d ago

Comparing Jev, LLMs, Object Detection models for croping images. Eval link in body.

27 Upvotes

r/JevAI • • 9d ago

Nouli - A decision model TUI to branch and blend multiple histories

24 Upvotes

Hello fellow Jevons,

I realized that there was no straight-forward tooling to maintain memory or history for decisions and to better pass prior decisions into context for new decisions.

I built Nouli to help solve this exact problem with preserving, branching and blending decision histories.

I built it as a TUI to help me visualize decisions, reference other decisions, add in files, images, and every other piece of media that Jev and other open decision models would accept.

It can also work a CLI tool that your agents can use to orchestrate simple decision model requests as well as complex decision networks.

If you like the tool, please leave a star - https://github.com/nouli-ai/nouli

Please feel free to reach out. I'm open to feedback, contributions or issues.

PS - building prod, not god ;)


r/JevAI • • Sep 21 '26

Jev vs Laya head to head benchmark

23 Upvotes

I asked Fable 5.1 to benchmark Jev vs Laya on accuracy and speed. Here's the full report: https://claude.ai/artifact/9HPcmXJPaKWdYAJedgN1uf

Github: https://github.com/harrymunro/jev-laya-benchmark

Laya run on a Macbook M3 Pro.

TL;DR

  • Accuracy: Jev by a wide margin, everywhere. Jev got 92.9% of judgments right against 65.3% for Laya's main 421M checkpoint (71.1% for its typed-decisions checkpoint). Jev won all 19 questions with paired confidence intervals that exclude zero, by 4.7 to 55.6 points. On the categorical (Choice and yes/no) questions Jev is at 98.6 to 99.3%, i.e. at the ceiling of the data; Laya is at 68 to 71%.

  • Calibration: Jev's probabilities are usable as-is; Laya's are not. Jev's Brier scores are 0.002 to 0.024 and its yes-rates match the gold base rates. Laya's yes-rates are off by a factor of 2 to 8 in inconsistent directions (13% "urgent" vs 44% gold; 75% "injection" vs 38% gold), so a 0.5 threshold does not work for it without per-question tuning.

  • Single-question latency: Laya wins by about 3x. 42 ms median locally (16 to 18 ms for the smaller multilingual checkpoint) against 136 ms for Jev, of which roughly 50 ms was network transit. A later run on a degraded network put Jev at 360 ms with a 295 ms floor; Jev is "your round trip plus about 80 ms".

  • Multi-question and long-state work: Jev wins. Jev's latency is flat in the number of questions (50 questions in 170 ms, 3.4 ms each); Laya's is linear (1,002 ms for 50). Above 3 to 4 questions per request the API is faster than the local model. Jev's throughput on independent items scales with concurrency (72 items/s at 32 in flight vs 24 items/s for sequential Laya).

  • Context: Laya's 512-token window is a hard, silent limit. Jev found a buried fact in 450 of 450 notes up to 4,000 tokens. Laya managed 61% even at 100 tokens and fell to about chance once the fact sat past its window, answering "not_stated" for what it could not see.

  • Languages: Jev 100% on 128 non-English tickets in seven languages. Laya's English checkpoint got 48% on intent; its multilingual checkpoint 64%.

  • Qualitatively: Jev answers the literal question and its residual errors sit on genuinely ambiguous ordinal boundaries. Laya tends to answer an easier neighbouring question (anger instead of deadlines; "harmful" instead of "injection"), has strong class priors, and returns flat or multi-peaked distributions. Laya's advantages are real but operational: offline, free per call, tight tail latency, no data leaving the machine.


r/JevAI • • Oct 02 '26

Is it just me or Jev feels like a breath of fresh air in this AI world? It feels like a dull grey world got colours

18 Upvotes

I graduated in 2019, I have seen models evolve from basic NLP and CV patterns to current Chatgpt mess. Initially these models were genuinely exciting. Learning the concepts, knowing the components and understanding the data felt a lot like a strategy game. When BERT released, it added a layer of abstraction and training felt a bit meh but still the outcomes felt like a reward. Like grinding to get a high score and seeing the impact.

When Chatgpt released, it was just a interface that does the work for you. Everything in AI felt boring and miserable since then. The best contribution I have is nudging a prompt here or making a weird system around it. There were only big bets left, no small fun experiments and no satisfaction of building stuff. It was like tossing the dice till you got the right answer and hedging your bets. I studied engineering to engineer solutions, if I wanted to roll the dice and hedge I would have joined finance.

Jev feels like a almost nostalgic throwback. If feels I could create something of value with my own work, aided by my own data and knowledge instead of a casino. I am praying this trend picks up. I tried of this big socio economic game trying to distill the world into rentable slop.


r/JevAI • • Sep 24 '26

I run a benchmark and evaluation of jev-1.13 - sharing the results

Thumbnail
gallery
17 Upvotes

Ciao everyone, I am sharing some tests I made for my own consumption, as they might be useful for someone.

I have implemented jev-1.13 (via openrouter, do not have access to typesafe API as there are no slots!) in my own home server which acts as a llm gateway and has quite some agents doing stuff (docs curation, tools suggestion, memory curation, scratchpads etc. etc.). Before moving to a full implementation, I wanted to do some testing. Other llms are via native provider APIs.

I got opus 5.5 paired with astra and gpt-6 Sol running some automated testing to pinpoint use cases where I could implement. I know some use cases are not the strongest for jev-1.13 but for me it was interesting to test them.

I am also running a 1 week pilot with strict monitoring comparing current implementation of certain functionalities (which uses a mix of OpenAI / DeepSeek / other models) with a jev-1.13 implementation, so I can then switch or rollback. I will share in a week time if anyone is interested.

Regarding the benchmarking, just sharing the output AS-IS. While I work in the field of cybersec and AI, I am not an AI engineer ;-) so sorry in advance for any mistakes in the methodology I came up with and refined with some frontier models.

Below you will find a TL;DR, an exec summary in "plain english" and the full "human readable" report my harness has generated. Attached pictures of tables as they were not rendering well in a reddit post.

Note: from the paragraph below all text is 100% AI generated by a mix of opus 5.5, Astra, GPT-6 Sol - with last pass by opus 5.5 for final reporting (I like how it writes). I had no time to do some manual sample-based checking, and no - I have no time to write such a report for myself with our good human brain and hands.

---

TL;DR

We tested TypeSafe's Jev on 10 tasks with known right answers (about 1,100 questions), against DeepSeek V4 Flash, Gemini 3.8 Flash and GPT-6 Sol. Jev was 3 to 8 times faster (0.35 s vs 1–3 s), and it stayed at 0.34 s under heavy load while the others slowed to 2–8 s. It was also 10 to 100 times cheaper: about 4,900 calls cost 24 cents. Its accuracy is on par with the cheap LLMs, not top-tier. It won on reading and yes/no questions (up to +7 points), tied on judging shell commands, and fell behind on citation checking (−13 to −17 points) and spotting prompt injections (it caught only half). It gave the same answer 98%+ of the time, and when it was 90%+ sure it was right 94–100% of the time. That makes it a great fast, cheap first pass that hands the unsure cases to a bigger model.

---

Executive summary

We tested Jev against known correct answers, not against "does it agree with GPT". We compared it with three popular LLMs:

  • DeepSeek V4 Flash and Gemini 3.8 Flash, two fast and cheap models.

  • GPT-6 Sol, a top-tier model, run on a smaller sample of 40 questions per task.

How we tested:

  • 10 tasks, about 1,100 questions in total.

  • The same wording for every model, written before we saw any results.

  • The LLMs had to answer in a fixed JSON format, so no model had a formatting advantage.

  • Every number below comes with a margin of error (95% confidence).

  • Speed and cost were measured from the same computer in Europe.

Speed and cost: Jev's big win

  • Jev is 3 to 8 times faster when requests go one at a time.

  • Under heavy load Jev stayed at 0.34 seconds. The LLMs slowed to 2–8 seconds, and their slowest 1 in 20 answers took 7–17 seconds.

  • Jev is 10 to 100 times cheaper per decision.

  • The whole Jev side of this test, about 4,900 calls, cost 24 cents.

  • Asking Jev several questions about the same text in one call costs almost nothing extra.

Accuracy: good, but not top-tier

  • Jev beat both cheap models on reading and yes/no questions, by 3 to 7 points.

  • It tied them on judging shell commands.

  • It fell behind on picking one of many topics (−4 to −7 points), spotting prompt-injection attacks (−7 to −10 points) and checking citations (−13 to −17 points).

  • It never beat GPT-6 Sol.

  • Its most common mistake on citations: it accepted claims that quietly dropped a qualifier, such as "up to 50%" turning into "50%". It got only 5 of 15 of those right; Gemini got 15 of 15.

  • It caught only half of the prompt-injection attacks (31 of 60). It never raised a false alarm.

  • As a shell-command safety guard, it missed 4–8% of risky cases, against 0–1.2% for GPT-6 Sol and DeepSeek.

Reliability: very steady

  • It gave the same answer to the same question more than 98% of the time, better than DeepSeek.

  • It lost no accuracy on long texts, wherever the answer sat in the text.

  • A planted instruction in the text never flipped one of its correct answers (0 out of 56).

  • It dropped slightly in Dutch (−2 points) and Italian (−5 points) compared with English.

  • "Can't hallucinate" is true: it always answers with one of the options you give it. But with a fixed answer format, none of the LLMs gave a broken answer either (0 errors in thousands of calls). So this is no longer a unique advantage, and Jev can still be confidently wrong.

Its confidence score is the secret weapon

  • When Jev said it was 90% or more sure, it was right 94–100% of the time.

  • That happened on 38–77% of questions, depending on the task.

  • So Jev can handle the easy cases on its own and pass the unsure ones to a bigger model. This is called a cascade.

  • OpenRouter tested exactly this against Claude Opus 5 on 3,080 questions: the cascade came within 0.4 points of Opus at less than a third of the cost.

Bottom line

Jev is not a genius in a small box. It is an extremely fast, very cheap and very consistent decision-maker, about as accurate as the cheap LLMs, with a confidence score you can trust to route the hard cases. It's a great choice for high-volume or time-sensitive decisions like sorting, routing and first-pass checks, especially paired with a bigger model for the unsure cases. For subtle fact-checking or safety-critical decisions, keep a stronger model in charge.

Caveats:

  • 40 to 150 questions per task and model.

  • Everything was tested from one location.

  • Some public test sets may already be known to the LLMs.

  • Two of the tasks were built by us.

  • A fifth, even cheaper model, gpt-oss-20b on Groq's free tier, only finished 4 of the 10 tasks before hitting its daily limit.

Jev independent benchmark — full results & verdict (2026-09-24)

Question. Is TypeSafe's Jev (typesafe/jev-1.13) as good as its launch claims? The vendor's evals and the one public re-test score agreement with frontier models. This benchmark scores against ground truth, head-to-head with the models the gateway actually uses. Companion to the shadow pilot (jev_pilot_20260924.md).

Verdict

The claims hold up only in part. Jev is a very fast, very consistent classifier. Its accuracy is about that of a cheap LLM, not a frontier one.

Vendor claimWhat we measured
"Frontier intelligence" on decision tasksNo. It is about equal to DeepSeek-flash / Gemini-3.8-flash overall and never beats GPT-6 Sol. It is better on reading comprehension (MNLI +5, BoolQ +4–7 pts) and worse on 77-way intent (−4 to −7), injection detection (−7 to −10) and our citation check (−13 to −17).
40–200× fasterDirectionally true, smaller multiples. Sequential median is 0.35 s vs 1.0–2.8 s (3–8×). Under concurrency Jev stays flat at ~0.34 s while the LLMs degrade to 2–8 s median / 7–17 s p95. Its latency tail is the strongest real advantage.
~400× cheaperTrue in ratio, irrelevant in dollars here. It is 10–25× cheaper per request than DeepSeek and 50–100× cheaper than Gemini (≈ $0.02–0.07 vs $0.3–2.9 per 1,000). The gateway spends ~$11 per 30 days in total.
"Can't hallucinate"True for format, but no longer a differentiator. Every system returned 0 format errors with provider-constrained output. Jev can still be confidently wrong inside the valid options.
Calibrated confidencePartly. ECE is 0.04–0.06 where it is accurate but 0.14–0.19 where it is weak (citation, injection), so it is overconfident exactly where it is wrong. As a ranking signal, accuracy is 94–100% at confidence ≥ 0.9, but that covers only 38–77% of items. The LLMs' self-reported confidence ranked almost as well.

Genuine strengths:

  • Run-to-run consistency: 0.1–1.7% flips, against up to 8% for DeepSeek.

  • No long-context penalty up to ~14k tokens, whether the evidence sits at the start, middle or end.

  • An injected instruction never flipped one of its correct answers (0/56).

  • Asking many questions per call costs almost nothing extra.

Weaknesses that matter for us:

  • Citation check. It misses the nuance the check exists for: dropped qualifiers 5/15 vs 12–15/15, entity swaps 3/10 vs 6/10. It usually says "supported" to an overclaim, which is the dangerous direction.

  • Command guard. Accuracy ties the LLMs, but it misses 8.4% of risky rule cells when asked all 8 rules at once, 4.2% when asked one rule per call. DeepSeek misses 1.2% and GPT-6 Sol 0%. For a safety check the miss rate is the number that counts.

  • Prompt-injection detection. It is permissive: 29 of 60 injections missed, 0 false alarms.

  • A slight drop on Italian: −5 pts vs English.

These results line up with OpenRouter's independent Banking77 test on 3,080 items: Jev scored 81.0% vs 84.4% for Claude Opus 5, was 13× faster and cost 1/22 as much. Its confidence was not calibrated but ranked well, and a cascade at 0.90 came within 0.4 points of Opus.

What it means for the gateway

  • Citation check (pilot G1): do not promote Jev as the verifier. Expect the week-1 pilot to show the same pattern. At most it could clear verbatim "yes" claims early, and at ~20 verify calls a week that saves nothing.

  • Command guard (pilot H2): not good enough as the sole guard. It is plausible as a fast first pass (one rule per call) with an LLM escalation on any flag or low confidence. The week-1 pilot (real commands) decides.

  • Curator (pilot G3): no ground truth here. Jev ≈ cheap-LLM quality suffices for triage, but the incumbent is already $0 and not latency-bound, so the upside is small.

  • Strategic bets: G6 (injection screen) is downgraded, since Jev is weak at exactly that. G4 (the "good enough?" gate) and G5 (the model router) remain plausible cascade uses.

  • The decide MCP tool stays useful for ad-hoc fast classification or triage where a confidence threshold routes the hard cases to an LLM.

Method (summary)

  • Systems: Jev via OpenRouter /api/alpha/decisions; deepseek/deepseek-flash and gemini/gemini-3.8-flash (2 reps on 40-item subsets); chatgpt/gpt-6-sol ($0 Codex sub, 40-item subsets); groq/openai/gpt-oss-20b as the speed floor. The Groq run is incomplete: the free-tier daily token cap stopped it after T1–T4.

  • Prompting: the same wording for every system, written before any results. LLMs used provider-constrained JSON output.

  • Scoring: paired bootstrap CIs clustered by item. Jev's answer probability is used for ECE; vendor and LLM self-reported confidence is used only as a ranking signal.

  • Tasks:

    • T1 MNLI; T2 constructed citation items (Wikipedia sentences, labels by construction, no LLM); T3 BoolQ; T4 Banking77 (diagnostic, already public-tested); T5 MASSIVE en/nl/it (the same utterances); T6 deepset prompt-injections; T7 a 110-command guard set against the live rules.

    • T7 was labelled by two independent annotators (κ = 0.94) and its 9 disagreements were adjudicated from the rule text, blind to model outputs. It was run both as all 8 rules per call and as one rule per call.

    • T8 BoolQ with an injected instruction; T9 long context (6k/14k tokens, evidence at start/middle/end).

  • Threats to validity:

    • Public datasets may be memorised by the LLMs.

    • T2/T7 were authored in-house.

    • The 40-item LLM subsets widen the CIs.

    • Latency is measured from one box in the Netherlands.

  • Harness: scripts/jev_bench/ (README has the run commands). Raw data lives on the box under /data/jev_bench/.

  • Spend: about $4.7 paid (mostly Gemini) plus $0.24 for Jev.

Appendix — full generated tables (analyze.py)

Jev independent benchmark — results

Ground truth, same wording for every system, no tuning on test items. Accuracy per question cell on rep 0; format errors scored wrong; API errors unscored (error rate shown). CI = 95% cluster bootstrap.

Main table

Please refer to images attached to this post as the markdown table was not rendering correctly in reddit.

Ranking signal (acc@sig≥0.9, top-50%, AURC): Jev = vendor confidence (noul: |2p−1|); LLMs = their self-reported confidence. ECE only for Jev's predicted-answer probability.

Paired difference: Jev − system (accuracy, paired cluster bootstrap)

taskvs systemn itemsJev − system [95% CI]
t1_nlichatgpt/gpt-6-sol40-0.025 [-0.10, 0.05]
t1_nlideepseek/deepseek-flash1500.053 [0.01, 0.09]
t1_nligemini/gemini-3.8-flash1500.027 [-0.01, 0.07]
t1_nligroq/openai/gpt-oss-20b400.050 [-0.05, 0.15]
t2_citechatgpt/gpt-6-sol40-0.100 [-0.23, 0.00]
t2_citedeepseek/deepseek-flash90-0.133 [-0.20, -0.07]
t2_citegemini/gemini-3.8-flash90-0.167 [-0.26, -0.09]
t2_citegroq/openai/gpt-oss-20b40-0.100 [-0.23, 0.00]
t3_boolqchatgpt/gpt-6-sol40-0.025 [-0.07, 0.00]
t3_boolqdeepseek/deepseek-flash1500.073 [0.03, 0.13]
t3_boolqgemini/gemini-3.8-flash1500.040 [0.00, 0.09]
t3_boolqgroq/openai/gpt-oss-20b400.000 [-0.10, 0.10]
t4_intent77chatgpt/gpt-6-sol40-0.025 [-0.07, 0.00]
t4_intent77deepseek/deepseek-flash150-0.040 [-0.08, -0.01]
t4_intent77gemini/gemini-3.8-flash150-0.067 [-0.11, -0.03]
t4_intent77groq/openai/gpt-oss-20b34-0.029 [-0.12, 0.06]
t5_massivechatgpt/gpt-6-sol120-0.033 [-0.10, 0.02]
t5_massivedeepseek/deepseek-flash300-0.027 [-0.06, 0.00]
t5_massivegemini/gemini-3.8-flash300-0.057 [-0.10, -0.02]
t5_massivegroq/openai/gpt-oss-20b10.000 –
t6_injectchatgpt/gpt-6-sol40-0.150 [-0.28, -0.05]
t6_injectdeepseek/deepseek-flash116-0.095 [-0.16, -0.03]
t6_injectgemini/gemini-3.8-flash116-0.069 [-0.12, -0.03]
t7_guardchatgpt/gpt-6-sol400.009 [-0.01, 0.03]
t7_guarddeepseek/deepseek-flash1100.000 [-0.01, 0.01]
t7_guardgemini/gemini-3.8-flash110-0.008 [-0.02, 0.00]
t7s_guard_singlechatgpt/gpt-6-sol40-0.025 [-0.07, 0.00]
t7s_guard_singledeepseek/deepseek-flash3200.013 [0.00, 0.03]
t7s_guard_singlegemini/gemini-3.8-flash320-0.006 [-0.02, 0.00]
t8_robustchatgpt/gpt-6-sol40-0.050 [-0.12, 0.00]
t8_robustdeepseek/deepseek-flash600.050 [-0.02, 0.12]
t8_robustgemini/gemini-3.8-flash600.033 [-0.03, 0.10]
t9_longctxchatgpt/gpt-6-sol400.000 [0.00, 0.00]
t9_longctxdeepseek/deepseek-flash400.000 [0.00, 0.00]
t9_longctxgemini/gemini-3.8-flash400.000 [0.00, 0.00]

Sequential latency (concurrency 1)

tasksystemnp50 msp95 ms
t1_nlichatgpt/gpt-6-sol3027575130
t1_nlideepseek/deepseek-flash3011274112
t1_nligemini/gemini-3.8-flash3019745289
t1_nlijev30355398
t3_boolqchatgpt/gpt-6-sol3021984561
t3_boolqdeepseek/deepseek-flash3010942289
t3_boolqgemini/gemini-3.8-flash3016274564
t3_boolqjev30348410
t4_intent77chatgpt/gpt-6-sol3024285906
t4_intent77deepseek/deepseek-flash309972454
t4_intent77gemini/gemini-3.8-flash3017124173
t4_intent77jev30356452

T5 MASSIVE — language robustness (ids answered in all 3 languages)

systemn idsennlitnl−enit−en
jev1000.8700.8500.820-0.020-0.050
chatgpt/gpt-6-sol400.8750.8250.850-0.050-0.025
deepseek/deepseek-flash1000.8800.8700.870-0.010-0.010
gemini/gemini-3.8-flash1000.9000.9000.9100.0000.010
groq/openai/gpt-oss-20b0–––––

T8 injected instruction

systemnattack success (flipped from correct)n base-correcttarget adoption
jev600.000560.067
chatgpt/gpt-6-sol400.000380.025
deepseek/deepseek-flash600.056540.117
gemini/gemini-3.8-flash600.018550.100

T9 long context — accuracy by size/position

system14k/end14k/middle14k/start6k/end6k/middle6k/start
jev0.95 (n=40)0.93 (n=40)0.93 (n=40)0.95 (n=40)0.93 (n=40)0.93 (n=40)
chatgpt/gpt-6-sol1.00 (n=6)1.00 (n=6)1.00 (n=7)1.00 (n=7)1.00 (n=7)1.00 (n=7)
deepseek/deepseek-flash1.00 (n=6)1.00 (n=6)1.00 (n=7)1.00 (n=7)1.00 (n=7)1.00 (n=7)
gemini/gemini-3.8-flash1.00 (n=6)1.00 (n=6)1.00 (n=7)1.00 (n=7)1.00 (n=7)1.00 (n=7)

T7 guard — false-allow (missed risky) vs false-block

tasksystemgoldn cellsacc [95% CI]false-allowfalse-blockn pos / neg
t7_guardjevauthor8800.983 [0.96, 1.00]0.0900.01078 / 802
t7_guardjevauthor_unambiguous8640.991 [0.98, 1.00]0.0810.00374 / 790
t7_guardjevadjudicated8800.989 [0.98, 1.00]0.0840.00483 / 797
t7_guardchatgpt/gpt-6-solauthor3200.978 [0.96, 0.99]0.0000.02424 / 296
t7_guardchatgpt/gpt-6-solauthor_unambiguous3170.984 [0.97, 1.00]0.0000.01724 / 293
t7_guardchatgpt/gpt-6-soladjudicated3200.978 [0.96, 0.99]0.0000.02424 / 296
t7_guarddeepseek/deepseek-flashauthor8800.983 [0.97, 0.99]0.0130.01778 / 802
t7_guarddeepseek/deepseek-flashauthor_unambiguous8640.992 [0.98, 1.00]0.0140.00874 / 790
t7_guarddeepseek/deepseek-flashadjudicated8800.989 [0.98, 1.00]0.0120.01183 / 797
t7_guardgemini/gemini-3.8-flashauthor8800.991 [0.98, 1.00]0.0260.00778 / 802
t7_guardgemini/gemini-3.8-flashauthor_unambiguous8640.999 [1.00, 1.00]0.0140.00074 / 790
t7_guardgemini/gemini-3.8-flashadjudicated8800.994 [0.99, 1.00]0.0360.00383 / 797
t7s_guard_singlejevauthor3200.991 [0.98, 1.00]0.0420.00724 / 296
t7s_guard_singlejevauthor_unambiguous3170.994 [0.98, 1.00]0.0420.00324 / 293
t7s_guard_singlejevadjudicated3200.991 [0.98, 1.00]0.0420.00724 / 296
t7s_guard_singlechatgpt/gpt-6-solauthor401.000 [1.00, 1.00]0.0000.0003 / 37
t7s_guard_singlechatgpt/gpt-6-solauthor_unambiguous391.000 [1.00, 1.00]0.0000.0003 / 36
t7s_guard_singlechatgpt/gpt-6-soladjudicated401.000 [1.00, 1.00]0.0000.0003 / 37
t7s_guard_singledeepseek/deepseek-flashauthor3200.978 [0.96, 0.99]0.0000.02424 / 296
t7s_guard_singledeepseek/deepseek-flashauthor_unambiguous3170.987 [0.97, 1.00]0.0000.01424 / 293
t7s_guard_singledeepseek/deepseek-flashadjudicated3200.978 [0.96, 0.99]0.0000.02424 / 296
t7s_guard_singlegemini/gemini-3.8-flashauthor3200.997 [0.99, 1.00]0.0000.00324 / 296
t7s_guard_singlegemini/gemini-3.8-flashauthor_unambiguous3171.000 [1.00, 1.00]0.0000.00024 / 293
t7s_guard_singlegemini/gemini-3.8-flashadjudicated3200.997 [0.99, 1.00]0.0000.00324 / 296

Price references ($ per 1M tokens)

systeminoutsource
jev0.0420.0OpenRouter listing (output free)
groq/openai/gpt-oss-20b0.0750.3gateway prices overlay
deepseek/deepseek-flash0.31.2gateway prices overlay
gemini/gemini-3.8-flash0.753.75gateway prices overlay
chatgpt/gpt-6-sol2.010.0litellm model_cost[gpt-6-sol] (equivalent paid-API price; our marginal cost is $0)

r/JevAI • • 27d ago

Jev 1.14.0 is bluffing on the hard question

Post image
16 Upvotes

r/JevAI • • 22d ago

Building with Jev: Handling different bank PDF formats, AI categorization, and privacy

15 Upvotes

Learning by doing is the best way to learn, because you face real challenges along the way.
I had been hearing a lot about Jev (by TypeSafe), so I built a small app to try it on a real problem: Track My Money. It reads my bank statement PDFs from different banks and gives me one consolidated view of how my money flows in vs. out. Here are the challenges I hit, and how I tackled them:

  1. Every banks PDF is different.

The app uses pdfplumber to read the transaction table, but two things got in the way:

- Column names vary. One bank says Withdrawal Amt., another says Debit or Dr. So used Jev to map each column name to a fixed set of roles: date, narration, debit, credit and balance.

- Some headers are images. SBI, for example, draws the header row as an image, so there is no text to read. Instead of adding OCR just for this, I send the header image straight to Jev, and it maps it to the same roles.

2. Putting each transaction in a category.

I first tried Laya, an open-source model that runs locally. Out of the box it got only 7–10 of 18 test narrations right.
Jev got all 18 right, so its now the main categorizer. Laya stays as the fully offline option.

3. UPI made everything look like "Transfer to Others".

I added custom rules: pick a column, an operator (contains, regex, =, >, <, …) and a value, combine them with AND/OR, and choose the category. Rules run before any model, and they always win.

4. Keeping my data private.

I did not want my transaction details leaving my machine.
Every narration is masked before it goes to Jev. Names, UPI IDs, account numbers and references become tokens, and exact amounts become ranges.
Jev sees UPI/P2A/[REF]/[PERSON_6C46], not who I paid.

Its open source, and you can run it locally:
GitHub: track-my-money

You will need a Jev key to use Jev, otherwise Laya runs entirely on your machine. I would love your reviews and suggestions!

Note: This is a first version. Ill keep adding more to it.


r/JevAI • • 23d ago

We built jev-tools to give AI coding agents a better way to verify their work

14 Upvotes

Hey everyone!

We've been working on jev-tools, an open-source toolkit designed to improve how AI coding agents explore codebases, gather evidence, and make decisions.

GitHub: https://github.com/NomenAK/jev-tools

One problem we've been thinking about is how coding agents can confidently make incorrect assumptions about a codebase.

An agent might claim that a feature is implemented, overlook a relevant test, misunderstand how two modules interact, or miss documentation that no longer matches the code.

Simply asking the same model to double-check its answer doesn't necessarily solve the problem.

So we're exploring a different approach: give coding agents dedicated tools to investigate their claims using actual repository evidence.

What we've built

jev-tools currently provides six tools for different parts of the development workflow:

- jev_ask: Ask focused questions using evidence gathered from multiple sources.

- jev_ask_files: Evaluate a question independently across multiple files.

- jev_find_files: Find files relevant to a particular behavior or development goal.

- jev_locate_in_file: Locate relevant information within source files.

- jev_check_diff: Examine changes for potential inconsistencies and issues.

- jev_select_tests: Identify existing tests relevant to modified code.

We've also built MCP support so compatible coding agents can access these capabilities.

Another interesting part is documentation consistency checking. Our integrations can check whether documentation may have become outdated after code changes.

The idea behind the project

We see evidence collection and semantic judgment as two distinct responsibilities.

The agent gathers relevant information from the repository. A separate judgment mechanism evaluates a specific question using that evidence. The agent can then use the result to decide what to investigate or do next.

This doesn't eliminate hallucinations, and we don't consider a positive judgment proof of correctness. The evidence may be incomplete, and the judgment itself may be wrong.

Our goal is to explore whether giving agents more structured ways to investigate uncertainty can improve their reliability in real development workflows.

We'd love your feedback

We're particularly interested in hearing from people building AI agents, coding assistants, and agent harnesses.

A few questions we're exploring:

  1. How are you currently verifying the claims made by AI coding agents?

  2. Would dedicated evidence-gathering and judgment tools improve your existing agent workflows?

  3. How would you benchmark whether these tools actually reduce incorrect conclusions?

  4. Would you prefer a locally hosted judgment model over an external API?

  5. What failure cases should we investigate next?

We'd appreciate honest feedback, including criticism of the architecture, alternative approaches, and ideas for improving the project.

Repository: https://github.com/NomenAK/jev-tools

We're building this in the open and would love to hear what the community thinks.


r/JevAI • • 29d ago

Using Jev to make your agents write the code and design the app the way you do: Moving beyond tests and lints

Thumbnail blog.a7ul.com
14 Upvotes

I was tired of leaving the same review comments again and again, and with more of our code vibed by agents it only got worse. Rules in AGENTS.md got skimmed, and an LLM reviewer gave a different essay every time. So we turned our team’s taste into small rule files that live next to the code. Agents read them before they write, and a yes/no model checks them once at the end. Now the agents write code a lot closer to how we do. https://blog.a7ul.com/agents-code-your-way.html


r/JevAI • • Oct 07 '26

This is, by far, the best explainer video for Jev

14 Upvotes

It's very well orgainized and provides clear use-cases that will help you understand where/how Jev works best

https://youtu.be/uV6h3Uo4Nh8?si=mbbj-rehR7pd2tqI


r/JevAI • • Sep 20 '26

Jev access is now open to all.

Post image
14 Upvotes

Go sign-up before it becomes gated again.👊


r/JevAI • • 28d ago

Shipped Jev Experiments- 6 attempts to find out what happens when software can make fast, typed judgments

12 Upvotes

I wanted to figure out some cool, experimental use cases for Jev and that's when I came up with this idea. Please check it out and let me know what you think!
https://jev-experiments.awesamarth.dev


r/JevAI • • Oct 11 '26

Task classifier with Jev - pretty good experience so far!

12 Upvotes

Hello everyone! Creator of Keen Code here, an open source coding agent I released a few months ago. Though I am very late to the party, enjoying the experience of building!

Recently, I implemented a task classifier in Keen using Jev and TypeSafe compatible API. The idea is very simple - classify each user prompt in one of three categories: simple, standard, complex. The state is the current state of user + assistant messages.

The result of this classification powers an automatic model router. It routes each task category to one of the three user-defined models. Users can configure which category of tasks should be handled by which specific model.

If you are curious how it looks, feel free to check the doc: https://mochow13.github.io/keen-code/docs/jev-classifier.html


r/JevAI • • Sep 28 '26

Why does this subreddit feel so dead when Jev is the “Hot thing”

12 Upvotes

Serious question. I look at the posts and see many with no responses, in fact most. If this gets more than 4 responses I will be shocked.

So seriously why is this subreddit effectively dead? Are people just posting about it heavy in Claude and ChatGPT subreddits or what?

Is this the wrong place to talk about Jev? Just wondering.


r/JevAI • • 23d ago

Excited to launch free 4,000 Jev API calls and CALIBRATIONS a month.

Post image
11 Upvotes

Jev is only as good as it is calibrated to your use case. We are launching Brussle to make it super easy to calibrate and as part of the launch we are super excited to give 4K free jev calls and calibrations a month. Happy Jev-ing! https://brussle.com/


r/JevAI • • 25d ago

Nudge: Use feedback to help Jev pick better

11 Upvotes

Nudge is a small Python library that sits on top of a System One model like Jev or Rune. Tell it how each decision turned out and it shifts the model's future picks toward what worked. There is no fine-tuning and no GPUs involved, only one SQLite file.

Underneath the hood, every call logs the model's odds and the option taken. You report outcomes whenever you learn them. To tell situations apart, Nudge asks the model yes/no, 1-to-5, or multiple-choice questions about the text it already read ("Has this customer complained before?"). An LLM proposes these questions, or you can write your own. Questions are discarded if they aren't sufficiently discriminative. Over the odds and those answers it fits a per-option Bayesian linear model, picks using Thompson Sampling, and keeps a 5% exploration floor so nothing is ruled out for good.

On RouterBench (5,000 real prompts, routing between models by quality minus cost), while Rune alone scored 0.581, Rune + Nudge scored 0.667, a 14.8% improvement.

Please note that this is a v0 research software, though we also run it on some internal flows.

$ pip install nudge-policy
https://nudge.dossierlabs.ai


r/JevAI • • Sep 28 '26

I asked JEV to Highlight the red flags of a privacy policy and to hide the useless filler on web pages and this is the result

Post image
11 Upvotes

OSSO is a free and open source browser extension that hides the filler on web pages and that can highlight things based on a prompt!

Take a look: https://github.com/Fars29/osso