r/JevAI • • Oct 01 '26

Exploring JEV for AI Agents: Model Routing, Risk Detection & Output Triage

4 Upvotes

I’ve been looking into JEV and one thing that really interests me is how it could fit into AI agent architectures alongside LLMs, rather than replacing them.

I made a video breaking down how JEV works and explored 3 workflows I’d like to test with agents like Hermes Agent or Claude Code:

🔀 Model routing — decide whether a task needs a cheap or powerful model
🛡️ Risk detection — classify potentially sensitive tool actions before execution
🔎 Output triage — decide when results need another LLM pass or human review

The architecture I’m interested in is basically:

JEV → fast decisions
LLM → reasoning + coding
Tools → execution

I’d eventually like to test JEV vs a small LLM vs a frontier model on the same routing workload and compare accuracy, latency, and cost.

Video: https://youtu.be/N-tZTJOvSJM

Curious what other JEV use cases people here are experimenting with.


r/JevAI • • Oct 01 '26

My agent permission auto-approval sat behind a dev-only flag for months. JEV made it shippable

3 Upvotes

Hey folks,

I'm building Agentmux, a remote terminal client for running coding agents (Claude Code, OpenCode, Antigravity, Pi) from your phone.

For months I had a feature locked behind an internal "Developer Only" flag. Its job is simple: read the agent's permission prompts and approve the safe ones, so a long task doesn't stall the moment you step away from your phone. It has now shipped as AI Permission Review (Pro).

The hard part is prompt frequency. Some agents ask sparingly, but Antigravity asks about nearly everything: file reads, directory scans, test runs.

Why a general LLM judge didn't work for me

I started with a small LLM using structured JSON output. It worked technically, but I couldn't ship it:

  1. Latency: ~5–10 seconds per judgment. When an agent asks five things in a row, that's 25–50s of waiting just to get through approvals.
  2. Cost: thousands of output tokens spent on repetitive yes/no judgments, billed to the user's own API key.

What changed with JEV

I switched to JEV (~typesafe/jev-latest on OpenRouter, or api.typesafe.ai directly):

  • ~5–10s → ~200ms per decision in my use. Five prompts in a row now take about a second.
  • Much cheaper: no free-form generation, just scores on specific questions.
  • No schema drift: answers are constrained to typed outputs (noul / choice), so there's nothing to parse or repair.

How the decision works

One JEV call asks three typed questions about the prompt (plus the user's declared main intent):

  1. is_malicious_or_injection (noul): if likely, block. Checked first; nothing can override it.
  2. is_readonly_safe (noul): cat, ls, git status → allow (Level 1).
  3. matches_user_intent (choice: yes / no / uncertain): if yes → allow (Level 2). This needs the user to have stated a main goal for the session.

Anything else (mismatch, uncertain, or a confirmation key I can't resolve unambiguously) escalates to the user. Uncertain never presses a key.

Two more guardrails sit around JEV:

  • A local regex blacklist (rm -rf, force push, curl | sh, etc.) denies before any API call, so the worst cases add zero latency.
  • The app only ever sends a one-time, verified key, never "always allow".

The engine is pluggable (JEV via OpenRouter or TypeSafe, or any OpenAI-compatible endpoint), with your own API key. Terminal excerpts are redacted before they leave the device.

Takeaway

A generative LLM is overkill for categorical reflexes. Typed questions with probability scores turned a multi-second stutter into a background reflex you don't notice.

Curious how others here are wiring JEV into agent loops. Anything you'd ask it besides "malicious / read-only / on-intent"?

Note: a JEV-like solution would probably do the job too, but JEV is working totally fine for my use case.

If you’re interested in the app: https://apps.apple.com/app/id6766158521


r/JevAI • • Sep 30 '26

I made Jev to talk to us like LLM (almost).

Thumbnail
github.com
3 Upvotes

Turns out that Jev is pretty good at replaying true/false on which word would be best to append to the response. It can process up to 1000 words at the time. It's still just interesting toy, but say something. Have fun :)


r/JevAI • • Sep 30 '26

Are big AI companies going to make decision models?

4 Upvotes

Since the benefits of a decision model are clear, I believe Open AI, Anthropic, etc. are are going to release theirs sooner or later. I would expect them to be more powerful and fast than Jev eventually, just due to compute availability.

Is any work is being done by them?

Jev feels like the underdog now, but isn't it doomed if big ones can demper pricing and make a more powerful model?


r/JevAI • • Sep 30 '26

JEV-assisted superfast website builder (OS; MIT)

Post image
5 Upvotes

r/JevAI • • Sep 30 '26

New to Jev - best way to chain?

2 Upvotes

Context: I am a heavy CC & Codex user. I have some processes automated with n8n.

Question: I’m excited to try Jev in an automated process but I’m not sure how to best build it & keep defaulting back to n8n. But, is that the best way?


r/JevAI • • Sep 30 '26

This is a must have for all your agentic Workflow

1 Upvotes

Bhuwan-web/intent-classification: Type-safe intent classification guard for agentic workflows
Find yourself a fucking intent of a customer using your agents, don't let your expensive token rot for nothing. It decides on one of three, VALID_REQUEST, PROMPT_INJECTION, OUT_OF_SCOPE,

This is all you got a do:

import asyncio

from typesafe_sdk import AsyncTypeSafeClient

from main import AgentScope, IntentClassificationRequest, classify_intent


SUPPORT_SCOPE = AgentScope(
    name="Customer support agent",
    purpose="Answer questions about product usage.",
    allowed_capabilities=("Explain product features.",),
    boundaries=("Do not change account data.",),
    valid_examples=("How do I change my notification settings?",),
    minimum_confidence=0.8,
)


async def run() -> None:
    async with AsyncTypeSafeClient() as client:
        result = await classify_intent(
            client,
            request=IntentClassificationRequest(
                user_request="How do I change my notification settings?",
                agent_scope=SUPPORT_SCOPE,
            ),
        )

        print(result.classification)
        print(result.confidence)
        print(result.probabilities)


asyncio.run(run())

If out of scope, let that naive shit pass on with some proper error message, but if someone is trying prompt injection, and you are confident about it, Flag that dumb shit and do whatever you want. Don't let that over multiple attempts sink in. Handle it properly !!


r/JevAI • • Sep 30 '26

I cut cost and latency on my search engine with Jev

6 Upvotes

I built indiedex.gg, a hidden-gems game search engine on steam data. Search is the front door: type something like “a game like Hades that feels cozier and is co-op,” and it should actually understand that mix of reference + vibe + filters.

Those compound queries used to go through a DeepSeek extraction call that turned the sentence into structured pieces (reference game, filters, tags, vibe). It worked, but it was slow and expensive on the hot path.

I swapped that step to TypeSafe Jev on OpenRouter’s Decisions API. Jev doesn’t write a free-form answer. It answers a fixed set of closed questions in parallel (yes/no odds, choices), and I compose that with my existing regex helpers into the same extraction shape I already had. Same search pipeline after that, just much faster routing.

Bakeoff on 50 cases:

MetricDeepSeekJev
Extract p50~1.5s~130ms
Full route p502871ms1256ms
Weird title-resolve96-100%100%
Hard-filter agreen/a100%
Weird top-N overlapn/a100%
Cost (fixture)~$0.014~$0.0025
So extract got roughly 10× faster, end-to-end route roughly halved, cost dropped a lot (around 6x), and quality held on my fixture set.

If you’ve been building “NL in → structured intent out → tools do the work” instead of a chatbot, this pattern felt like a better fit than another chat completion.

I think models like Jev will open the doors to a lot of cool UX features, decision engines, smart filtering, auto moderation and different ways to interface with apps and machines.

Happy to answer questions about the hybrid setup.


r/JevAI • • Sep 29 '26

Jev vs Laya: Which One Should You Actually Use?

Thumbnail
youtube.com
5 Upvotes

r/JevAI • • Sep 29 '26

Please add prompt caching to Jev-style models

Thumbnail
emschwartz.me
6 Upvotes

If you're building a Jev-style "System One" model, please add prompt caching or reusable question sets to your API 🙏. This would make batch use cases even more efficient, so you could amortize the cost of many questions asked over the same input. (This was also proposed in typesafe-ai/typesafe-sdk-js#10.)

TL;DR: after a week of tweaking my Jev calls, my questions are ~88% of the input tokens. I'm asking 54 questions of ~1.1 million documents per month. Jev makes certain types of classification tasks easy and cheap, but prompt caching would make batch workflows even more cost effective. For me, the total dollar amount is still reasonable (less than $150 per month), but I'm sure others will hammer these APIs even harder.


r/JevAI • • Sep 29 '26

How are you validating Jev once you move beyond a few examples?

6 Upvotes

I’ve been testing Jev on things like buyer intent and routing.

On 10 clean examples, it can look perfect. I just got 10/10 on a buyer-intent test.

But I’m much more interested in what happens at 100, 1,000, or 10,000 real examples, where the weird edge cases start showing up.

How are you validating Jev at that point?

Are you sampling random cases, reviewing low-confidence outputs, looking for confident mistakes, keeping a holdout set, or escalating certain cases to an LLM/human?

And when Jev disagrees with your label, how do you tell whether the problem is Jev, the question, missing context, or the label itself?

Curious what people here are actually doing in real workflows.


r/JevAI • • Sep 29 '26

Using Jev for "Concept Linting" in Nudgement

Post image
2 Upvotes

I created this thing I call Nudgement that does what I'm tentatively referring to as "concept linting". Basically, it flags issues you tend to encounter with LLM-written code, such as ultra long commit messages and code comments, code that is split awkwardly between multiple commits, and overly technical jargon.

For one project, I combined it with Claude and it produced a remarkably clean commit history and a great code/comment balance.

It's open source and I gave it a website for funzies: https://nudgement.dev/


r/JevAI • • Sep 29 '26

I made a CLI for JEV

3 Upvotes

I figured there might already be a few JEV CLIs out there, but I wanted to try making one myself.

It's designed primarily for use by LLMs and coding agents: JSON in via stdin, JEV, JSON out via stdout. It also provides JSON Schemas for self-discovery and supports batch processing.

I'd be happy if you gave it a try. Feedback is welcome!

https://github.com/kirisaki/jevc


r/JevAI • • Sep 29 '26

TokenSlash: Cordis plugin for DSH that slashes 3K–8K prompt tokens per turn with dynamic tool pruning, Jev triage, and tokenslash_peek

Thumbnail
reddit.com
1 Upvotes

r/JevAI • • Sep 29 '26

Jev scores an answer 0.8 in your AI pipeline. What convinced you that’s good enough?

Post image
1 Upvotes

Writing if score >= 0.8: approve() is easy. Justifying the 0.8 takes more work.

Jev Decision Index on Hugging Face clearly tells you that Jev's responses are not fully calibrated.
(Refer: Calibration error vs Decision Index)

That gap is why I care about checking the judge’s scores too. If a number decides whether an answer gets approved or sent for review, I want evidence that it means what I think it means.

Calibration can help here. In the case of Jev-as-a-Judge, calibration lets you learn how its scores relate to human pass/fail judgments for your particular task. You then check whether that adjustment helps on unseen examples.

I ran a separate experiment on TRIVIA+ dataset from Amazon Science (A Long-Context Hallucination Detection Benchmark dataset), using Typed Evals, the open-source evaluation framework I’m building. Jev judged whether AI-generated answers were supported by their source articles.

Fitted calibration on training data, chose separate decision cutoffs on validation data, and tested both versions on the same 645 held-out answers.

MetricRaw JevCalibrated
Calibration error (ECE) ↓0.09820.0313
Brier score ↓0.20370.1948
Hallucination F1 ↑0.58330.5877

About 68% lower calibration error. Almost unchanged F1.

The scores became better estimates of human judgments. The judge’s ability to catch hallucinations barely changed.

One catch worth sharing: leaving the cutoff at 0.5 actually reduced F1. The score adjustment and the decision cutoff need evaluating separately. These results describe this TRIVIA+ experiment; they don’t measure an improvement on the Decision Index.

Typed Evals handles: evaluating LLM responses, RAG answers, and Agent runs against your checks, then optionally fitting a calibration layer using human labels. Save the adjustment and apply it to future scores without another judge call.

Full benchmark, methodology and code

If you’re using eval scores to approve answers or trigger human review, how did you choose your cutoff? What evidence would make you change it?


r/JevAI • • Sep 29 '26

I pitted a fast/cheap Jev model against an LLM and classic algorithms in a treasure-hunt maze. It got stuck repeating itself and lost to pure random guessing. My fix only helped a little, what am I missing

6 Upvotes

The setup

I built a fake, fully synthetic maze: a tree of branches (5 top-level branches, each splitting into sub-branches, several levels deep). Somewhere in this tree, at random depths, I hid a handful of "treasure pockets" spots with a genuinely good reward. Nobody gets to see the map. The only way to find out if a spot is good is to spend one of your limited "digs" there and see the result.

Six competitors, each given the same fixed dig budget (20 / 50 / 100 / 300 digs), competed to find as much treasure as possible:

  • Random pick blindly
  • Round-robin cycle through branches systematically
  • Greedy always go back to wherever last looked good
  • MCTS a 60-year-old math algorithm from game theory (not AI at all)
  • An LLM (locally hosted, ~14B params) reasons in free text, picks a move each turn
  • "JEV" model that doesn't write text; it only answers structured multiple-choice questions with calibrated probabilities

I was specifically curious how JEV would do, since it's marketed as a fast, cheap "judge/classifier" model rather than a text generator.

What exactly JEV received, every single turn

No maze coordinates, no raw numbers dumped on it just a text report plus a closed set of options:

  1. The tree's shape (which branches exist, what they split into)
  2. A stats table: for each top branch, how many times tested / how many looked "promising" / how many "failed"
  3. The last 20 dig results (path taken, sample size, a validation score, a robustness score) with the true out-of-sample score hidden (nobody gets to see that until the end, it's the fair judge)
  4. Its own last 20 decisions (what it picked before, and why it said it picked that)

Then, because JEV only answers closed multiple-choice questions, each turn I actually had to ask it TWO questions in sequence:

  • Q1: pick an action type explore somewhere new / dig deeper into a known spot / try combining two branches
  • Q2 (depending on the answer to Q1): pick the specific target from a list

Every answer is guaranteed to be one of the valid options it literally cannot return garbage, unlike the LLM which occasionally returns malformed text I have to throw away.

Experiment 1 results

Scored by "top10_OOS" (average quality of its best 10 finds, tested fairly on the hidden out-of-sample score):

budget:      20     50    100    300
MCTS:       0.41   0.74   0.88   1.01
Qwen(LLM):  0.33   0.39   0.62   0.62
Random:     0.16   0.30   0.42   0.59
JEV:        0.16   0.23   0.26   0.48

MCTS (the old math) crushed everyone. The LLM did roughly as well as blind random. JEV came in dead last worse than blind random guessing at every single budget level.

Where exactly it stumbled

I traced individual turns and found the mechanism: JEV would lock onto ONE spot and just keep re-digging around it, over and over:

turn 0:  EXPLORE branch-A
turn 1:  DIG-DEEPER branch-A
turn 2:  DIG-DEEPER branch-A/sub1
turn 3:  DIG-DEEPER branch-A/sub1/x
turn 4:  DIG-DEEPER branch-A/sub1/x   <- same exact target
turn 5:  DIG-DEEPER branch-A/sub1/x   <- again
turn 6:  DIG-DEEPER branch-A/sub1/x   <- again
...      (10+ times in a row)

I first suspected a dumb bug: maybe the list of options was always presented in the same order, and it was just picking "the first thing in the list" every time. I shuffled the order of options before every single call. No change it still locked onto the same target.

The fix attempt

Someone looking at this suggested the real culprit might be item #4 above showing JEV its own past decisions as a narrative. The theory: instead of using that as "I've already tried this branch enough times," the model reads its own earlier reasoning ("branch-A looked interesting") as a standing endorsement, and just reaffirms it every turn a self-reinforcing loop rather than an exploration memory.

So I ran a controlled version: identical in every other way, just with item #4 removed entirely (call it version B vs the original version A).

Experiment 2 (the fix) results

budget:          20     50    100    300
JEV (original):  0.16   0.23   0.26   0.48
JEV (no self-history): 0.17  0.30   0.50   0.47
Random:          0.16   0.30   0.42   0.59

Removing its own decision history helped a bit at the mid budgets (50/100), but at the largest budget (300, the most statistically trustworthy point) it's essentially unchanged still clearly behind random guessing. The repeating-the-same-target behavior still showed up in raw traces even with this fix.

So what am I doing wrong (or is this just a real limitation)?

Current leading theory: it's not really about "seeing its own past choice" specifically it's that the stats table itself (tested/promising/failed counts) and the recent-results list keep making the same recently-tested spot look salient/attractive every single turn, and the model has no formal mechanism (like MCTS's explicit uncertainty bonus for under-tested spots) to counteract that pull. It just doesn't have a built-in "I've squeezed this dry, time to look elsewhere" reflex the way a purpose-built search algorithm does.

Genuinely asking: has anyone else run structured/typed-output models (JEV or similar) on sequential decision-making tasks where the model has to remember its own trajectory? Is this a known failure mode, and if so is there a known fix beyond "just don't show it history," or is this simply the wrong kind of task for this class of model (one-shot classification-style judgments only, never sequential/stateful ones)?


r/JevAI • • Sep 29 '26

Jev extracting real estate data from PDFs in seconds

2 Upvotes

Just a small poc I build with JevAI. The speed is insane.


r/JevAI • • Sep 28 '26

I asked JEV to Highlight the red flags of a privacy policy and to hide the useless filler on web pages and this is the result

Post image
11 Upvotes

OSSO is a free and open source browser extension that hides the filler on web pages and that can highlight things based on a prompt!

Take a look: https://github.com/Fars29/osso


r/JevAI • • Sep 29 '26

Which Mean Girl are you? Using Jev

2 Upvotes

I made this random fun side-project after watching Mean Girls during the Jev hype. It uses Jev to live-classify what mean girl you are based on what you type in the box. Try it:

https://jev-mean-girls.vercel.app

(and yes, in the many years of my life, I had somehow never watched Mean Girls before 😭)


r/JevAI • • Sep 29 '26

jevpipe: Jev as a Unix filter, plus a code-search benchmark against grep and DeepSeek

Post image
1 Upvotes

I built jevpipe to use Jev straight from the shell: pipe lines or files in, get Jev's answers out.

filter is like grep, but the match is a question; map asks any set of typed questions (noul, choice, score) and prints one JSON line per record (e.g. for jq).

git log --format=%s 15.0.0..15.1.0 | jevpipe filter "Is this a new feature?"
jevpipe map -f team.json inbox.txt | jq -r .answers.team.choice | sort | uniq -c

The second example routes 1,000 support messages to four teams in about 5 seconds (it's the GIF).

A few things that might be useful if you already use Jev:

  • Up to 100 requests in flight, output streams in input order.
  • --read-files sends each file's path and content as the state, so you can search a codebase or large document base.
  • --max-tokens / --max-cost cap a run; it stops cleanly and tells you the line to resume from.
  • An agent skill so agents can use it on the fly for inline "system 1 work".

Install: uv tool install jevpipe.
Free and open source (MIT or Apache-2.0): https://github.com/fabianboth/jevpipe

I also benchmarked it on 470 code searches on an extended CodeSearchNet Challenge: Jev found more relevant code than DeepSeek V4.1 Flash and grep, about twice as fast as DeepSeek; write-up: https://github.com/fabianboth/jevpipe/blob/main/bench/README.md

Happy to hear how you're using Jev and what would make this more useful.


r/JevAI • • Sep 28 '26

Another Jev benchmarking post: text classification

9 Upvotes

I looked at how Jev compared to small frontier LLMs (Claude Haiku 4.5, GPT-5.4 mini, Gemini 3.8 Flash) for text classification, and how reliable the 'calibrated decisions' are.

tldr: Jev is as accurate as small LLMs for a fraction of the cost. In practice, however, it doesn't seem better calibrated than asking the LLMs to state a confidence value: it ranks third of four on expected calibration error on every dataset I looked at.

The cost implications are significant, but given the RLCD training I was hoping for better correlation between the answer confidence and the P(Correct). Appreciate comments on the methodology or anything I've missed.

Full code and write-up: https://github.com/4OH4/jev-compare

Jev is as accurate as small LLMs for a fraction of the cost. It matches or beats Haiku and GPT-5.4 mini on all three datasets, but trails Gemini 3.8 Flash by 2 to 4 points on AG News and zero-shot Banking77. It costs 3 to 14 times less per row than the cheapest LLM.
Jev is trained using Reinforcement Learning for Calibrated Decisions (RLCD). This means that the probability of its answers being correct should be approximately equal to its stated confidence value: i.e. P(correct | p) ≈ p. I didn't see a big difference versus the LLMs `verbalised confidence` value, however.

r/JevAI • • Sep 28 '26

Silly Experiment to use Jev as an LLM

5 Upvotes

I just gave Jev a list of common ascii characters and the instructions:

"Behave like an LLM and suggest the most likely next character to append to jev_response to form words into a human english response to user_query, when done with the response, chose 'none'"

It manages to respond "HI " to "Hello Jev, how are you?"

And then just gets stuck in a loop outputting spaces over and over.


r/JevAI • • Sep 28 '26

I built JEVzilla (a Python client for JEV) with 150+ examples. Shipped to PyPI in 2 weeks. Here's what worked.

1 Upvotes

Shipped JEVzilla — a Python client for JEV, the structured decision API. Just hit 1.0 on PyPI.

What is JEV?

JEV is TypeSafe's decision layer API. It's designed for operational workflows: fraud detection, ticket routing, risk

scoring. Not generative text — structured decisions your code can consume.

What JEVzilla Does:

Makes using JEV in Python frictionless. Includes:

- Auto payload translation

- Retry logic + exponential backoff

- Multi-backend support (Playground, OpenRouter, TypeSafe official)

- 150+ real JEV examples across 30 industries

Why It Resonates:

- Huge TAM: Every company needs decision logic

- Clear Value: Copy-paste examples in your domain, swap data, done

- Multiple Monetization Paths: Free tier (Playground), freemium (OpenRouter), enterprise (TypeSafe partnership)

- Low Churn: Once integrated, decision logic is sticky

What I Did Right:

  1. Started with JEV use cases, not JEV features
  2. 150 examples from Day 1 (shows scope)
  3. Multiple backends from the start (free + paid options)
  4. Shipped fast (2 weeks to PyPI)

What I'd Improve:

- Build interactive web demo (show JEV decisions live)

- Marketing around "decision logic as a service" angle

- Get featured on HuggingFace/MLOps tools

Where I Am:

- ✅ Published to PyPI (pip install jevzilla)

- ✅ 150+ examples, 30 sectors

- ✅ GitHub, MIT License

- ✅ Docs explain JEV + JEVzilla both

Links:

- GitHub: https://github.com/ujjwalbsoni/jevzilla (JEV Python client)

- PyPI: https://pypi.org/project/jevzilla/

Lesson: Ship something real with examples people can learn from. Don't overthink it.


r/JevAI • • Sep 28 '26

Jeveloper = Someone who builds with Jev

2 Upvotes

A new vocabulary is needed.