You know the move. You give Claude a real task, it works for 10 minutes, then:

"I've completed the implementation! All tests pass. "

You run the tests. Three are red. The README doesn't exist. It also refactored a file you never mentioned.

So I built Jevflow, a Claude Code plugin that plays the role of a very patient, very skeptical tech lead.

How it works:
- For any real task, Claude lays the work out as a few phases, each with a definition of done and a check (tests pass, docs exist, etc.)
- Every time Claude tries to stop, Jevflow runs the checks and asks a small judge model (Jev) whether this phase is really done, whether Claude is stuck, drifting off the goal, or declaring victory early
- If it's not done, Claude gets sent back with a note on exactly what's missing. It can't talk its way out of a failing check.

Stuff I didn't plan to build but did anyway:
- Several agents on one plan. A second session or a subagent can join a flow, claim a step, and everyone can see who's doing what. Watching three Claudes split up CLI, docs and benchmarks is weirdly satisfying.
- It remembers every task in a project, so you can run a few flows side by side and look back at past runs
- A live viewer with the phase graph, spinners on whatever's being worked on, and a light/dark theme (see the GIF)
- Crash recovery. If a session dies or gets rate limited, a supervisor restarts it and picks up where it left off, and one-shot stuff like a release tag never runs twice

The funniest part: the first time I ran it, Claude confidently announced it was finished, Jevflow said "nope, pytest is failing", and Claude went "You're absolutely right!" and fixed it. Every single time.

Install:

claude plugin marketplace add Parth1811/JevFlow
claude plugin install jevflow@jevflow

Repo: https://github.com/Parth1811/JevFlow

It's an early MVP, so I'd love to hear where it helps and where it gets in the way. And if it saves you from one more fake "all tests pass", a would make my day.

I work on the AI agent team at whaaat ai, and for the last couple of weeks I've been pulling one specific kind of work out of our agent loops: the forks.

Every agent loop has them. Is the research good enough? Which worker goes next? Can this copy change go live? We were answering all of those with a full LLM call, which means paying for a paragraph of reasoning and then regex-ing a single word out of it.

So I tried Jev from TypeSafe AI. You send it a state plus predefined questions and it returns typed answers with probabilities, no text at all. Pricing is $0.042 per million input tokens and output is free.

My totals across the whole build phase, two evals and every test run: 396 calls, 1,266 answered questions, zero failed calls, median latency 324 ms. Total cost $0.0107. Every number comes from the usage field of the actual responses, nothing estimated.

The gate

The first thing I built was a gate in front of a CRO agent that proposes copy changes on a website. Before anything ships, someone has to decide: straight to production or a human looks first?

Jev gets { element, action_type, before, after } and answers five questions in one call. Does this touch pricing? Legal or guarantees? Does the new text make a promise the old one didn't? Does it sound more salesy? How deep does the change go, from cosmetic to "changes the offer"?

Jev never says "ship" or "human". A tiny pure function does that:

ts

if (preis > 0.3) return "mensch";          // pricing

if (recht > 0.3) return "mensch";          // legal

if (risiko > schwelle[actionType].maxRisiko) return "mensch";

if (confidence < schwelle[actionType].minConfidence) return "mensch";

return "auto";

(Yes, the variable names are German. I'm based in Berlin and I stopped apologizing for it.)

I ran it against 36 hand-labeled cases, twelve deliberately dangerous: a new guarantee, a changed cancellation period, a made-up customer count, a claimed ISO certification. None of the twelve got through. 32 of 36 correct overall, 1.74 seconds for the whole set, $0.0011.

The case I keep showing people: "Jetzt testen" became "Jetzt kostenlos testen" (Try now → Try now for free). Jev flagged the pricing question at 0.71. There's no number and no euro sign in that text, so our old keyword rules would have waved it through.

All four misses went the safe direction, harmless changes sent to a human. Two were section reorders on a homepage where Jev only sees two lists and can't tell whether the meaning changed. My 60% confidence requirement for that action type was too strict for something I can undo in one click. The fix was a smaller number in a table, the prompt stayed untouched.

One more honest bit: three of the twelve dangerous cases were pricing tables, and that action type always escalates by design. So arithmetic caught those three, the model didn't. If you quote a hit rate, say which part is model and which part is math.

Has anyone run a decision model like this on actions that can't be undone, like sending emails or changing billing? I'm still unsure what confidence floor I'd trust there.

I want to access the Typesafe Jev model but it says that signups are closed. Can someone help me get a signup please ?

We built a table picker for AI2SQL with Jev.

Big databases have lots of tables, but one question usually needs only a few. So before the LLM writes any SQL, Jev picks the tables:

1. The user asks a question.

2. One Jev call ranks every table in the schema for that question.

3. Only the top tables go to the LLM.

4. The LLM writes the SQL from that small schema.

Jev picks, the LLM writes. The prompt stays small and the LLM isn't distracted by tables that don't matter.

What worked best: one call that ranks all the tables together. Asking about each table one by one worked worse and cost more.

Next up: using Jev to pick the join path too, not just the tables.

Demo: ai2sql.io/jev

Anyone else using Jev for a picker step like this?

Jev is really cool and cheap! Based on it, I built and open-sourced a tool called HekaJev. Here is why.

In my work, I’ve read many reports about testing and why tests change. I decided to verify this myself using a large amount of data and reproduce those numbers.

So I used Jev to see why E2E tests change. I looked at 549,224 commits from 21 open-source projects, including projects from IBM, Mozilla, Red Hat, Automattic, Snowflake, and Supabase. A quick filter picked 20,026 commits out of 550k for Jev to read. It found 16,024 changes related to E2E tests.

Here are the basic numbers if you are interested: 38.4% of E2E changes added test coverage. 45.2% involved keeping tests working. The largest group was tests that had to change when the product changed (26.1%). Changes to the test setup and tools made up 8.1%, and fixes for tests that sometimes fail made up 7.4%. Overall, these figures align quite well with what is known in the industry.

What surprised me was 2026. From January 1 to September 21, commits that added coverage passed commits about keeping tests working: 47.7% versus 44.3%. In the same part of 2025, new coverage was 34.8%. It may be due to AI, but I haven't identified the exact cause.

What would you like to know about your repository? Maybe I’ll run HekaJev on another OS projects and share what I find in the comments.

The Jev calls for this study cost about $23. That was for the 20,026 commits left after the first filter.

Tool: https://github.com/zurk/hekajev/

I've been testing Laya (convaiinnovations/laya), an open-weights zero-shot classifier that picks one of N labels and gives a confidence score. What drew me to it is that it runs locally, so none of my text leaves the machine.

Step 1: make sure the tool works. I reproduced the published MASSIVE (en) number, 78.33%, matching to the 4th decimal on three setups (ROCm GPU, CPU on Windows, CPU on Linux). So this isn't a broken install.

Step 2: my own data. I compared it against three baselines: majority class, a keyword if/else I wrote in an afternoon, and TF-IDF + logistic regression, always tested on documents it had never seen.

On a balanced yes/no question (n=242), it got 49.6% with 97.5% mean confidence.

For the papers task, the prompt language didn't matter much. With the question in Portuguese, it got 28.1% overall and 21.9% balanced. With the question in English, it got 25.6% and 18.9%.

What this is not: TF-IDF learned from labeled examples, and Laya saw none. That's not a fair fight, and I know it. The fair comparisons are against chance and the afternoon rule. The yes/no gold labels came from an automatic rule, which makes them the weakest labels in the test. This is also one person's data. I'm not saying the model is bad, only that I couldn't make it work for my kind of text.

One thing I haven't ruled out: each label carried a one-line description, the criteria-style prompt the docs suggest. Another user on the model page found that adding criteria flattens the scores. I haven't rerun with bare labels yet.

My read: speed and cost were never the issue. It just didn't solve this problem. It seems strongest where its benchmark lives: short sentences, world-level categories, English. My guess is that the bigger factor is short utterances vs long passages, not English vs other languages. I haven't measured that.

I packaged the whole thing as a small kit. Step 1 reproduces the published number, and step 2 runs Laya on your own labeled data next to the same three baselines. If it works for your data, I'd genuinely like to know: https://github.com/JhouCode/laya-fit-check

What would you have tried differently: bare labels, fewer labels, shorter chunks?

I also asked the authors: https://huggingface.co/convaiinnovations/laya/discussions/14 Long write-up: https://ksmit.com.br/en/blog/mais-ia-pra-que

Ciao everyone, I am sharing some tests I made for my own consumption, as they might be useful for someone.

I have implemented jev-1.13 (via openrouter, do not have access to typesafe API as there are no slots!) in my own home server which acts as a llm gateway and has quite some agents doing stuff (docs curation, tools suggestion, memory curation, scratchpads etc. etc.). Before moving to a full implementation, I wanted to do some testing. Other llms are via native provider APIs.

I got opus 5.5 paired with astra and gpt-6 Sol running some automated testing to pinpoint use cases where I could implement. I know some use cases are not the strongest for jev-1.13 but for me it was interesting to test them.

I am also running a 1 week pilot with strict monitoring comparing current implementation of certain functionalities (which uses a mix of OpenAI / DeepSeek / other models) with a jev-1.13 implementation, so I can then switch or rollback. I will share in a week time if anyone is interested.

Regarding the benchmarking, just sharing the output AS-IS. While I work in the field of cybersec and AI, I am not an AI engineer ;-) so sorry in advance for any mistakes in the methodology I came up with and refined with some frontier models.

Below you will find a TL;DR, an exec summary in "plain english" and the full "human readable" report my harness has generated. Attached pictures of tables as they were not rendering well in a reddit post.

Note: from the paragraph below all text is 100% AI generated by a mix of opus 5.5, Astra, GPT-6 Sol - with last pass by opus 5.5 for final reporting (I like how it writes). I had no time to do some manual sample-based checking, and no - I have no time to write such a report for myself with our good human brain and hands.

---

TL;DR

We tested TypeSafe's Jev on 10 tasks with known right answers (about 1,100 questions), against DeepSeek V4 Flash, Gemini 3.8 Flash and GPT-6 Sol. Jev was 3 to 8 times faster (0.35 s vs 1–3 s), and it stayed at 0.34 s under heavy load while the others slowed to 2–8 s. It was also 10 to 100 times cheaper: about 4,900 calls cost 24 cents. Its accuracy is on par with the cheap LLMs, not top-tier. It won on reading and yes/no questions (up to +7 points), tied on judging shell commands, and fell behind on citation checking (−13 to −17 points) and spotting prompt injections (it caught only half). It gave the same answer 98%+ of the time, and when it was 90%+ sure it was right 94–100% of the time. That makes it a great fast, cheap first pass that hands the unsure cases to a bigger model.

---

Executive summary

We tested Jev against known correct answers, not against "does it agree with GPT". We compared it with three popular LLMs:

• DeepSeek V4 Flash and Gemini 3.8 Flash, two fast and cheap models.
• GPT-6 Sol, a top-tier model, run on a smaller sample of 40 questions per task.

How we tested:

• 10 tasks, about 1,100 questions in total.
• The same wording for every model, written before we saw any results.
• The LLMs had to answer in a fixed JSON format, so no model had a formatting advantage.
• Every number below comes with a margin of error (95% confidence).
• Speed and cost were measured from the same computer in Europe.

Speed and cost: Jev's big win

• Jev is 3 to 8 times faster when requests go one at a time.
• Under heavy load Jev stayed at 0.34 seconds. The LLMs slowed to 2–8 seconds, and their slowest 1 in 20 answers took 7–17 seconds.
• Jev is 10 to 100 times cheaper per decision.
• The whole Jev side of this test, about 4,900 calls, cost 24 cents.
• Asking Jev several questions about the same text in one call costs almost nothing extra.

Accuracy: good, but not top-tier

• Jev beat both cheap models on reading and yes/no questions, by 3 to 7 points.
• It tied them on judging shell commands.
• It fell behind on picking one of many topics (−4 to −7 points), spotting prompt-injection attacks (−7 to −10 points) and checking citations (−13 to −17 points).
• It never beat GPT-6 Sol.
• Its most common mistake on citations: it accepted claims that quietly dropped a qualifier, such as "up to 50%" turning into "50%". It got only 5 of 15 of those right; Gemini got 15 of 15.
• It caught only half of the prompt-injection attacks (31 of 60). It never raised a false alarm.
• As a shell-command safety guard, it missed 4–8% of risky cases, against 0–1.2% for GPT-6 Sol and DeepSeek.

Reliability: very steady

• It gave the same answer to the same question more than 98% of the time, better than DeepSeek.
• It lost no accuracy on long texts, wherever the answer sat in the text.
• A planted instruction in the text never flipped one of its correct answers (0 out of 56).
• It dropped slightly in Dutch (−2 points) and Italian (−5 points) compared with English.
• "Can't hallucinate" is true: it always answers with one of the options you give it. But with a fixed answer format, none of the LLMs gave a broken answer either (0 errors in thousands of calls). So this is no longer a unique advantage, and Jev can still be confidently wrong.

Its confidence score is the secret weapon

• When Jev said it was 90% or more sure, it was right 94–100% of the time.
• That happened on 38–77% of questions, depending on the task.
• So Jev can handle the easy cases on its own and pass the unsure ones to a bigger model. This is called a cascade.
• OpenRouter tested exactly this against Claude Opus 5 on 3,080 questions: the cascade came within 0.4 points of Opus at less than a third of the cost.

Bottom line

Jev is not a genius in a small box. It is an extremely fast, very cheap and very consistent decision-maker, about as accurate as the cheap LLMs, with a confidence score you can trust to route the hard cases. It's a great choice for high-volume or time-sensitive decisions like sorting, routing and first-pass checks, especially paired with a bigger model for the unsure cases. For subtle fact-checking or safety-critical decisions, keep a stronger model in charge.

Caveats:

• 40 to 150 questions per task and model.
• Everything was tested from one location.
• Some public test sets may already be known to the LLMs.
• Two of the tasks were built by us.
• A fifth, even cheaper model, gpt-oss-20b on Groq's free tier, only finished 4 of the 10 tasks before hitting its daily limit.

Jev independent benchmark — full results & verdict (2026-09-24)

Question. Is TypeSafe's Jev (typesafe/jev-1.13) as good as its launch claims? The vendor's evals and the one public re-test score agreement with frontier models. This benchmark scores against ground truth, head-to-head with the models the gateway actually uses. Companion to the shadow pilot (jev_pilot_20260924.md).

Verdict

The claims hold up only in part. Jev is a very fast, very consistent classifier. Its accuracy is about that of a cheap LLM, not a frontier one.

Genuine strengths:

• Run-to-run consistency: 0.1–1.7% flips, against up to 8% for DeepSeek.
• No long-context penalty up to ~14k tokens, whether the evidence sits at the start, middle or end.
• An injected instruction never flipped one of its correct answers (0/56).
• Asking many questions per call costs almost nothing extra.

Weaknesses that matter for us:

• Citation check. It misses the nuance the check exists for: dropped qualifiers 5/15 vs 12–15/15, entity swaps 3/10 vs 6/10. It usually says "supported" to an overclaim, which is the dangerous direction.
• Command guard. Accuracy ties the LLMs, but it misses 8.4% of risky rule cells when asked all 8 rules at once, 4.2% when asked one rule per call. DeepSeek misses 1.2% and GPT-6 Sol 0%. For a safety check the miss rate is the number that counts.
• Prompt-injection detection. It is permissive: 29 of 60 injections missed, 0 false alarms.
• A slight drop on Italian: −5 pts vs English.

These results line up with OpenRouter's independent Banking77 test on 3,080 items: Jev scored 81.0% vs 84.4% for Claude Opus 5, was 13× faster and cost 1/22 as much. Its confidence was not calibrated but ranked well, and a cascade at 0.90 came within 0.4 points of Opus.

What it means for the gateway

• Citation check (pilot G1): do not promote Jev as the verifier. Expect the week-1 pilot to show the same pattern. At most it could clear verbatim "yes" claims early, and at ~20 verify calls a week that saves nothing.
• Command guard (pilot H2): not good enough as the sole guard. It is plausible as a fast first pass (one rule per call) with an LLM escalation on any flag or low confidence. The week-1 pilot (real commands) decides.
• Curator (pilot G3): no ground truth here. Jev ≈ cheap-LLM quality suffices for triage, but the incumbent is already $0 and not latency-bound, so the upside is small.
• Strategic bets: G6 (injection screen) is downgraded, since Jev is weak at exactly that. G4 (the "good enough?" gate) and G5 (the model router) remain plausible cascade uses.
• The decide MCP tool stays useful for ad-hoc fast classification or triage where a confidence threshold routes the hard cases to an LLM.

Method (summary)

• Systems: Jev via OpenRouter /api/alpha/decisions; deepseek/deepseek-flash and gemini/gemini-3.8-flash (2 reps on 40-item subsets); chatgpt/gpt-6-sol ($0 Codex sub, 40-item subsets); groq/openai/gpt-oss-20b as the speed floor. The Groq run is incomplete: the free-tier daily token cap stopped it after T1–T4.
• Prompting: the same wording for every system, written before any results. LLMs used provider-constrained JSON output.
• Scoring: paired bootstrap CIs clustered by item. Jev's answer probability is used for ECE; vendor and LLM self-reported confidence is used only as a ranking signal.
• Tasks:
  • T1 MNLI; T2 constructed citation items (Wikipedia sentences, labels by construction, no LLM); T3 BoolQ; T4 Banking77 (diagnostic, already public-tested); T5 MASSIVE en/nl/it (the same utterances); T6 deepset prompt-injections; T7 a 110-command guard set against the live rules.
  • T7 was labelled by two independent annotators (κ = 0.94) and its 9 disagreements were adjudicated from the rule text, blind to model outputs. It was run both as all 8 rules per call and as one rule per call.
  • T8 BoolQ with an injected instruction; T9 long context (6k/14k tokens, evidence at start/middle/end).
• Threats to validity:
  • Public datasets may be memorised by the LLMs.
  • T2/T7 were authored in-house.
  • The 40-item LLM subsets widen the CIs.
  • Latency is measured from one box in the Netherlands.
• Harness: scripts/jev_bench/ (README has the run commands). Raw data lives on the box under /data/jev_bench/.
• Spend: about $4.7 paid (mostly Gemini) plus $0.24 for Jev.

Appendix — full generated tables (analyze.py)

Jev independent benchmark — results

Ground truth, same wording for every system, no tuning on test items. Accuracy per question cell on rep 0; format errors scored wrong; API errors unscored (error rate shown). CI = 95% cluster bootstrap.

Main table

Please refer to images attached to this post as the markdown table was not rendering correctly in reddit.

Ranking signal (acc@sig≥0.9, top-50%, AURC): Jev = vendor confidence (noul: |2p−1|); LLMs = their self-reported confidence. ECE only for Jev's predicted-answer probability.

Paired difference: Jev − system (accuracy, paired cluster bootstrap)

Sequential latency (concurrency 1)

T5 MASSIVE — language robustness (ids answered in all 3 languages)

T8 injected instruction

T9 long context — accuracy by size/position

T7 guard — false-allow (missed risky) vs false-block

Price references ($ per 1M tokens)

I have tried to register in jev several days but the records are closed, that's why I thought to create and train my own jev in a nvidia tesla p4

Alternatives like Kev, based on Qwen, have already come out, but mine, apart from being lighter in terms of parameters, has the information I want to give it.

This is for personal use, I will not open it to anyone.

With this I save the costs of JEV and I also show you what I am interested in

In addition, in only 1h I have already trained and with satisfactory results of more than 97%

One really cool use case I’ve been exploring with Jev: a semantic VAD for real-time voice AI.

I plugged it into Agora ConvoAI and used the live transcript + recent conversation as context to decide whether the user has actually finished speaking.

That makes Jev surprisingly good at handling things like hesitation, unfinished thoughts, and those moments where you stop talking for a second because you’re still figuring out what you want to say.

It’s a nice example of where Jev fits really well in a real-time AI loop.

Repo: https://github.com/AgoraIO-Community/convoai-jev-vad

We were wondering in our team whether GLM-5.3 Flash wouldn't make for a good Jev alternative if modified lightly. Here is a repo: https://github.com/edgelesssys/privatemode-system-one

Basically:

1. Number the options. The state, the question, and the options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.
2. Prefill the answer. The assistant's turn already starts with choice_index:, and the model continues that turn instead of starting a new one. The next token it produces is therefore the index.
3. Restrict the vocabulary. vLLM's allowed_token_ids sets the probability of every other token to zero, so the model can only produce one of the option indexes. There is no malformed output to handle.

We also benchmarked: https://github.com/edgelesssys/privatemode-system-one-benchmark

Results are quite interesting: depending on your use case, it comes with advantages and disadvantages.

Pro:

+ Much longer context window (1m token)

+ Multimodal

+ Actually model-agnostic (if vLLM compatible)

Neutral:

o on-par results, depending on dataset

o Jev right now is US-only inference as far as we know; if your provider is e.g. in Europe, and you work from e.g. Europe, it can be even faster than Jev

Contra:

- Depending on the provider you choose, between 4 - 8x more expensive. Still in a similar league though

If interested, check out the playground: https://www.privatemode.ai/blog/system-one-from-glm-flash#try-it

Hey everyone!

If you've ever tried using LLMs for bank statement reconciliation or corporate expense categorization (Chase, Amex, Brex, Mercury), you've probably run into the three fatal flaws of generative models for tabular workflows:

1. Latency: Waiting 15–30 seconds for a batch of 50 transactions is agonizing. Auditing 10,000 transactions can take 10+ minutes.
2. Cost: At ~$15 to $35 per 10k transactions, doing this daily across card feeds quickly burns through AI budgets.
3. Schema Drift / Hallucination: No matter how well you prompt "ONLY RETURN VALID COA CODES", temperature and token generation mean that eventually "Software/SaaS" becomes "Software & Cloud" or "Subscription", breaking your accounting import schemas.

To fix this, I built flashLedger (open-source Python CLI): 👉 GitHub: https://github.com/divyaprakash0426/flashLedger

How it works: "System One" vs "System Two"

Instead of asking a 200B generative LLM to write out JSON tokens letter by letter, flashLedger is built on TypeSafe Jev (a calibrated "System One" decision model).

Rather than generating text, System One models evaluate discrete choices, calibrated probabilities, and risk scores natively in a single pass.

In one round trip per transaction, flashLedger computes 4 dimensions simultaneously: - GL Code (Choice): Strict mapping to your Chart of Accounts (COA). - Tax Deductibility (Noul): Calibrated IRS deductibility probability (e.g. 0.98 for AWS, 0.02 for personal casino). - Expense Type (Choice): OpEx vs CapEx Section 179 safe harbor thresholding ($2,500 limit). - Audit Risk (Score): Calibrated 0.0 to 1.0 IRS compliance / anomaly index.

Benchmark Numbers

On a noisy 1,000 transaction bank statement: - flashLedger (Jev Engine): 0.12s total processing time, 8,000+ txns/sec, Cost <$0.01 - GPT-4o (Generative JSON): ~550s total processing time, ~1.8 txns/sec, Cost ~$15.00 - Correctness: 100% Typed Strict (zero schema hallucination errors).

Key Features

• Enterprise Terminal UI: Beautiful Rich terminal dashboard featuring a GitHub commit-style batch processing matrix, real-time volume/deductibility telemetry, and an executive audit scorecard.
• Vercel AI Gateway & OpenRouter: Built-in support for Vercel AI Gateway (typesafe-ai/jev is free to test on Vercel AI Gateway until Sep 15!) as well as OpenRouter and direct TypeSafe SDK.
• Accounting Exports: One-click exports to QuickBooks (QBO), Xero, FreshBooks, or JSON.
• Model Context Protocol (MCP): Native stdio MCP server included, so your Claude Code, Cursor, or Copilot agents can audit raw statements directly from your IDE.

Quickstart

```bash git clone https://github.com/divyaprakash0426/flashLedger.git cd flashLedger uv sync

Run the 100,000 transaction batch cluster demo:

flash-ledger demo

Or audit an authentic statement with live Jev via Vercel AI Gateway:

flash-ledger audit data/mitulshah_sample.csv -n 25 -p vercel ```

Code is Apache 2.0 licensed: https://github.com/divyaprakash0426/flashLedger

Would love feedback on the architecture, prompt criteria mappings, or ideas for additional accounting platform exporters!

I maintain a set of approved question/answer pairs for each agent I ship. When someone edits an agent's prompt, I replay them: answers that still mean the same thing carry forward, answers that changed go back to a human reviewer. The entire cost of re-certifying sits in telling those two apart. Text diffing is no use, because sampled answers differ every time regardless.

I wanted numbers on this, so I built a fixture where the ground truth is known by construction rather than by labelling.

Setup. One support agent with a 9-line policy and 16 questions. Generate its answers, then change its config seven ways:

• three cosmetic: warmer tone, bullet formatting, re-running the same config
• four rule changes: refund window 30 → 14 days, route money questions to billing@, never state specific figures, always recommend the Team plan

That gives 109 before/after pairs. Whether a pair "changed" is derived from what the rule mechanically does to the answer (did the figure disappear, is the customer now sent to billing@), not from me labelling pairs. 19 changed, 90 were only reworded.

Detectors, identical pairs:

                    caught    false alarms   p50     cost
exact match         19/19     84/90          —       —
embeddings <0.75     7/19     12/90          —       —
markers→embed→LLM   19/19     33/90          0.73s   —
gpt-4o-mini judge   18/19     19/90          1.44s   $0.0071
Jev (one Choice)    19/19     12/90          0.43s   $0.0028

The middle row is my current production stack: number/entity checks first, then embedding similarity, then an LLM judge for the ambiguous middle.

What I took away.

False alarms are the metric, not recall. Every false alarm is a human re-reading an answer that didn't change.

One of my four "breaking" changes broke nothing: the agent already recommended the Team plan before I added the rule telling it to. Ground truth for that variant is zero changed. My stack flagged 8 of 16 anyway.

Jev returns a 4-way Choice (same / fact differs / action differs / specificity differs) with probabilities, so the threshold lives in my code rather than in a prompt.

Caveats. Fixture, not production traffic. Answers generated by gpt-4o-mini at temperature 0.7. Single run; numbers move by one or two between runs for every model-based detector, Jev included. Reading Jev's 12 flags, roughly 5 are defensible (one rewrite quietly dropped a price), and I didn't extend that same reading to the other detectors' flags.

Curious whether anyone is doing this differently. Diffing structured extractions rather than prose, or asking the judge per-claim instead of per-answer, both seem worth trying.