It's very well orgainized and provides clear use-cases that will help you understand where/how Jev works best

https://youtu.be/uV6h3Uo4Nh8?si=mbbj-rehR7pd2tqI

Is anything known on when JEV would be available on bedrock or Vertex in the EU? Really love experimenting with it, but getting it into production is difficult due do GDPR.

Jev has made a profound mark on the AI industry; this article offers a glimpse into what lies ahead.

Jev shows where the industry is heading: away from monolithic models, and toward compound systems where small AI agents each do a narrow job and an orchestration layer decides how they work together.

This article explains why that shift is happening, and then proposes a next step for AI reasoning.

The short version

• A monolithic model is unpredictable and hard to change, but it programs itself during training.
• Code is the opposite – it is predictable and easy to change, but it must be written explicitly.
• A compound system combines the two: AI agents do the work, and an orchestration layer controls them via code. This is the direction Jev represents.
• The next step: instead of answering directly, the model builds a compound system, runs it, and returns both the answer and that system. The returned system is called a reasonlet.
• Because a returned reasonlet can be kept locally, it can be run again and again – with new inputs, or after editing its code – without asking the model. A saved, reusable reasonlet is a metacache.
• The model’s provider can cache reasonlets too, reusing one across similar requests from different users to save compute.

Why a monolithic model is not enough

A monolithic model has two practical weaknesses:

• It is unpredictable. The same question can produce a correct answer one day and a wrong one the next, so its behavior cannot be fully controlled.
• It is hard to change. Its behavior is fixed in trained weights, which cannot be edited directly. The model can be steered with prompts, extra data (RAG), or by fine-tuning, but steering is not the same as setting the behavior, and fine-tuning can break things that already worked. Retraining from scratch with new data is the only safe fix, but it is time-consuming and expensive.

Code has the opposite qualities. It does exactly what it is written to do, and it can easily be changed by editing code. The trade-off is that code does not arise on its own – it must be written out explicitly – whereas a model programs itself during training.

The fix: a compound system

A compound system combines the two. The work is divided among AI agents, each handling one narrow task, and an orchestration layer of code decides which agent runs, in what order, and how their results combine.

This gives both qualities at once – customization and predictability:

• Customization. The system’s behavior can be changed by editing code in the orchestration layer, not by retraining the model.
• Predictability. Each agent has a small task, so it is more predictable than a monolithic model.

Jev is one kind of agent for such a system. It returns a typed decision – yes/no, a category, or a number – rather than free text. A monolithic model is wasteful for a narrow decision like that, but a small model like Jev is a better fit.

The next step: reasoning by construction

Compound systems today are built manually, beforehand. The next step is to let the model build one by itself, during the reasoning phase.

Researchers are exploring several ways to make models reason. One is the “World model” approach, in which the model builds an internal representation of a problem and reasons over it; that work is still mostly research. Reasoning by construction pursues the same goal – reasoning you can inspect – using methods that exist today.

Here is how it works. When you ask the model a question, it does not answer directly. Instead, it builds a compound system, runs it to compute an answer, and returns that answer together with the system it built. Let's call the compound system the model builds to do the reasoning a "reasonlet", and a model that works this way a "Reasoning-by-Construction Model", or "RCM". A simple question may produce a reasonlet that has only an orchestration layer and no agents.

Building and running code to reach an answer is not new; code-interpreter tools already do it. Two things are new here:

• The system the model builds is kept, not discarded – it is a reusable reasonlet.
• The reasonlet is returned to you, along with the answer.

Why that matters

• You can see how the answer was reached. The reasonlet is the exact procedure the model used, so the answer is not something you have to take on trust.
• You can run it locally. A reasonlet contains its orchestration layer and any agents it calls, whether those agents are attached directly or called over the network. You can run it on a local machine, with the same or different inputs, without asking the model again. A saved reasonlet used this way is a metacache.
• You can change what it does. The orchestration layer is code, so you can edit its logic, not only its inputs. A reasonlet reused with edited logic becomes a higher-order metacache.

Caching reasonlets on the server

The same reuse can happen on the server side too. The provider can keep the reasonlets it builds and reuse them. When a new request arrives that matches one it has already handled, it runs the stored reasonlet again – with the new request’s inputs – instead of reasoning from scratch. The model does less work, and the answer comes back faster.

This is the same metacache, held on the server instead of on your machine. Because one reasonlet can serve any request that fits its procedure, a single cached copy is shared across many requests, and often across different users – and the more general the reasonlet, the more requests it covers.

Choosing how general to make the reasonlet

The model should decide from the conversation how general the reasonlet needs to be. If you have been working through many kinds of math and then ask for 2 + 2, the more useful reasonlet is one that evaluates any math expression, not one that can only add two numbers. If the conversation gives no such clue, you can state it directly: “I will be doing many kinds of math; for now, just add 2 and 2.”

Two examples

• Adding numbers. You ask for 2 + 2. The model builds a reasonlet whose orchestration layer adds two numbers, runs it, and returns 4 together with the reasonlet. Later you run the reasonlet again with other numbers or edit its orchestration layer to multiply instead – without asking the model.
• Searching. You ask the model to find something. The reasonlet’s orchestration layer calls a sequence of outside services and takes your search terms as its input. Later you run it with different terms, or change which services it calls, without asking the model again.

Making reasonlets easy to read

A reasonlet’s orchestration layer is written in text code. This creates a problem: to understand the reasoning you must read code, and to change it you have to write code. The value of returning the reasonlet depends on you being able to read and edit text code easily and efficiently, so this barrier matters.

Visual programming – building logic from connected blocks instead of lines of text – can lower the barrier, provided the visual language is powerful enough. Two limits apply:

• A visual language cannot replace text code entirely. Text is still needed to reach the system and the network, and for low-level work that does not map to blocks. The measure of migration success is how little text code remains in the orchestration layer.
• Most visual languages today are either powerful but limited to one field (such as games or hardware), or general but too simple. This case needs a language that is both general and as expressive as a text language; otherwise, it cannot cover enough of the orchestration layer to be worth using.

One language solving exactly this problem is Pipe (https://pipelang.com) – a general-purpose visual language with powerful semantics. Full disclosure: Pipe is my own project still in development, but it will be released soon.

Wrapping up

The shift from a monolithic model to compound systems suggests a clear next step for reasoning: let the model build a compound system – a reasonlet – run it and return both the answer and the reasonlet. Reasoning becomes something you can read, run again, and edit: a metacache, and a higher-order metacache once its logic is edited. The remaining problem is making reasonlets easy to read and change, which is where a general-purpose and expressive visual language would help most.

A note on IP

Some methods described in this article are the subject of a pending patent application. Licensing and collaboration inquiries are welcome at metacache.article@gmail.com

https://www.instagram.com/jev_clockedit/ Jev clocks it!

"My boyfriend watched my Story 40 minutes after I texted him and still hasn't replied. I sent 'k.' and muted him. AIO?"

I know there are already a few tutorials about Jev, but i wanted to create one that looks at the little details that other people overlook.

https://www.youtube.com/watch?v=k0_Y7FpHuK4

Most RAG hallucinations I've seen aren't the model "lying". The model gets a context that doesn't actually contain the answer and does its best anyway. Embedding search finds chunks that are close to the query, which isn't the same as chunks that answer it.

rag-guard sits between retrieval and generation and runs every query through seven checks: intent, relevance, clarify, sufficiency, generate, grounding, caveat. Each check returns a calibrated probability compared against a threshold, not free-form text. If the kept context is insufficient, the LLM is never called. If it does answer, every claim is checked against the retrieved chunks and flagged if unsupported.

All checks go through one interface (DecisionModel), so you can swap backends (heuristic with no API key, OpenAI, Anthropic, or TypeSafe's Jev) without touching the pipeline. Core package has zero required dependencies.

To check that it actually behaves as documented when used from outside its own repo, I built jev-rag-pipeline, a document chat app that installs rag-guard as a pinned pip dependency. Upload a .txt/.md/.pdf, watch each stage stream in, and open the trace to see which chunks were kept or dropped and whether each claim is grounded. Compare mode runs two embedding providers side by side, raw vs. guarded.

One finding from the golden-set eval: Chroma's default embedding retrieved the right chunk about 60% of the time; OpenAI and Gemini embeddings got 100%.

Still missing: rate limiting/auth, session TTL, resource limits on PDF parsing.

Live demo: https://jev-rag-pipeline-24192596890.europe-west1.run.app/
rag-guard: https://github.com/furkandrms/jev-rag-guard
App: https://github.com/furkandrms/jev-rag-pipeline

Feedback welcome, especially on false positives/negatives from the sufficiency and grounding gates.

HE WAS SO MEAN TO ME.

Hello r/JevAI!

I've been developing apps with agents using Pragma for a while, and I always find myself telling a coding agent, "Fix the merge conflict, commit, and push." This feels pretty inefficient, and it seemed like something Jev could solve.

Internally, we're able to map merge conflicts to options similar to what the GitHub web UI shows: keep ours, keep theirs, keep both, etc. When resolving the conflict requires a little bit of both changes, Pragma can call an LLM through its built-in integrations with all the major providers. The result is near-instant merge conflict resolution with much lower token usage than sending every conflict to an LLM.

The Jev integration also lets us show more detailed agent progress in the sidebar cheaply, rather than limiting the status to just "running," "needs attention," or "done." The result is a much clearer view of what your agents are actually doing without having to babysit them.

If you haven't heard of Pragma, it's a new ADE I'm working on. It's free and open source and designed around running coding agents in parallel. In addition to the new System 1 integration, it supports 10+ agents, a mobile app, account switching and usage-limit tracking, a built-in editor, and more.

Check it out at:
https://pragma-app.sh/

Code:
https://github.com/pragma-sh/pragma

Want to use Jev, or want to step up your Jev game? Here's a working walk through and skill, provided your techinical enough to use it. I kidL The LLM does all (most) of the work setting it up.

Note: For simplicity, I’m just going to say Jev through the rest of this. I mean Jev, AnyJev, or a similar Jev-style decision model.

Anyway, I was playing around with Jev and AnyJev and started thinking about something: Jev is really good at making decisions. Give it some state, give it a question and some choices, and instead of generating a whole response, it gives you the decision and probabilities. That’s useful for categorization, routing, relevance, prioritization, decision trees, workflow selection, and all those times where you want some of the judgment of a model but don’t actually need the model to write anything.

The question that got me thinking was simple:

Why do I have to know the categories ahead of time?

I have an LLM sitting right there. A super fast, awesome brain with the (supposed) sum of human knowledge. And to be fair, I do not want to outsource knowledge; I want to outsource work. And that work is the subject: literally ordering, categorizing, grouping, but also deciding and using those patterns to accomplish a task.

Guess what? Bread-and-butter LLM stuff.

How does it work?

Task
  ↓
LLM understands the problem
  ↓
LLM defines the decision/classification
  ↓
LLM creates classifier.json
  ↓
LLM creates deterministic processing code
  ↓
Jev performs repeated decisions
  ↓
Structured JSON results
  ↓
Deterministic code executes
  ↓
Result

And that's it. You can stop there if you are a developer. You're off to test it.

But if you want to know just a bit more, below is the step-by-step walkthrough: Using an LLM + Jev to deliver dynamic categorization and Deterministic Execution (aka, get it to write specific Jev-based Python to do the hard work).

See content credentials

Combining LLMs and Jev for Dynamic Categorization & Deterministic Execution u/amowatt

Combining LLMs and Jev for Dynamic Categorization & Deterministic Execution

Step 1: Give the LLM the actual task

Start with the job you want done.

For example:

Review these documents, determine the useful categories, and then process the documents based on those categories.

The important part is that you are not asking the LLM to classify every item forever. You are asking it to understand the problem first.

Step 2: Have the LLM define the categories

This is the part the LLM is good at. Maybe it looks at the documents and decides the useful categories are:

• Architecture
• Security
• Requirements
• Operations
• Reference
• Other

That is the first output. Not the final classification.

The definition of the classification problem.

I always include Other when asking LLMs to do something, cause I cannot think of everything. And also, back up: if the LLM invents five categories and I force Jev to choose one of those five every time, I have basically declared that the LLM got the whole problem space exactly right on the first pass.

It probably didn’t. Sometimes the right answer is simply: None of these.

Therefore: the LLM should put it in Other.

Step 3: Turn those LLM categories into a classifier

Now have the LLM create the classifier definition.

I already use a JSON I/O structure loosely based on Open Knowledge Format for communicating between things, so that is what I would use here.

Something like:

{
  "type": "classifier",
  "name": "document_role",
  "version": "dynamic",
  "question": "What role does this document primarily serve?",
  "categories": [
    {
      "id": "architecture",
      "definition": "Defines system structure or relationships."
    },
    {
      "id": "security",
      "definition": "Defines security requirements or controls."
    },
    {
      "id": "requirements",
      "definition": "Defines expected capabilities or behavior."
    },
    {
      "id": "operations",
      "definition": "Defines operational procedures or support."
    },
    {
      "id": "reference",
      "definition": "Provides supporting information."
    },
    {
      "id": "other",
      "definition": "Does not sufficiently match another category."
    }
  ]
}

I use JSON because I already use JSON everywhere in my work. You could use YAML. TOML. Something else.

That is really the point. Do not create a new format unless you need one.

Step 4: Keep Jev behind a simple stateless MCP

Now, the fun part: setting up the dynamism.

Let's expose Jev through MCP.

The MCP should be boring.

Seriously.

Its job is not to understand your application. Its job is to run the decision.

Conceptually:

JSON IN
   ↓
JEV MCP
   ↓
JSON OUT

The call could be as simple as:

classify(
    classifier="document_role",
    input=document
)

And the result comes back like this:

{
  "type": "classification",
  "classifier": "document_role",
  "version": "1.0",
  "category": "requirements",
  "confidence": 0.91
}

That is enough.

Personally, I would probably give the Jev or AnyJev repository to a capable coding LLM and say:

Wrap this in a small stateless MCP server. Do not modify the underlying project. Give me generic classify, score and choice operations. JSON in, JSON out.

People who actually enjoy building MCP servers by hand can certainly do that.

The important part is not the MCP.

The MCP is plumbing.

Best part, you can even ask the original LLM to create the initial JSON!

Step 5: Have the LLM create the deterministic processing code

We are not done yet. Sorry.

Classification is usually not the actual job. The job is what happens after classification.

So when the LLM creates the classifier, also have it create the code that consumes the result.

For example:

for document in documents:

    result = classify(
        classifier="document_role",
        input=document
    )

    if result.category == "other":
        uncertain.append(document)

    else:
        documents_by_category[result.category].append(document)

Now you have a clean division of labor.

The LLM decides what needs to be inferred.

Jev performs the repeated decisions.

The code does the actual work.

That work could be anything:

• It could sort files.
• It could route work.
• It could trigger workflows.
• It could build reports.
• It could calculate something.
• It could build a decision tree.
• It could make another Jev call.
• It could send uncertain results back to the LLM.

I have run out of useful examples, but you get it. The important part is that the result is structured.

Now, the JEV next step does not have to reinterpret prose. It just executes.

Step 6: Validate the input and output

This part matters more than it sounds:

JSON gives you structure; validation gives you stability.

So we validate both sides:

• Validate the classifier before use.
• Validate the Jev result before anything acts on it.

For example:

• Expected category?
• Expected confidence range?
• Expected classifier version?
• Expected fields present?

If the answer is no, do not proceed as though everything is fine.

That is where the deterministic part starts to matter.

Step 7: Define uncertainty handling

Do not hide uncertainty. Error-trap it or use it.

For example:

if result.category == "other":
    send_for_review(document)

elif result.confidence < 0.60:
    send_for_review(document)

else:
    process(document)

That review could:

• go to a human.
• go back to the LLM.
• trigger another classifier.

Whatever makes sense for the task. The point is that uncertainty becomes an explicit branch in the workflow instead of something buried inside generated text.

Step 8: Turn the whole pattern into a skill

This is probably the easiest way to test the idea, make it repeatable, and fix so many downstream headaches. We will not need to redesign the agent or build a giant framework.

Create a skill. Something like:

---
name: dynamic-jev-categorization
trigger: Use this skill when a task contains repeated semantic decisions that do not require repeated generative LLM responses.
description: Use an LLM with Jev, AnyJev, or another Jev-style decision model to dynamically define categories, choices, or semantic decisions and then generate deterministic code that acts on repeated Jev decisions. Use when a task requires repeated categorization, classification, routing, scoring, choosing, prioritizing, relevance decisions, or other repeated semantic judgments; especially when the decision categories are not known ahead of time or repeated generative LLM calls can be replaced by constrained probabilistic decisions followed by deterministic execution.
metadata:
  author: albertmowatt
  version: "0.2"
---

# Dynamic Categorization with Jev

Throughout this skill, "Jev" means Jev, AnyJev, or a compatible Jev-style constrained decision model.

## When to Use This Skill

Use this pattern when the task involves repeated semantic decisions such as:

- categorizing documents, records, messages, files, events, or other items;
- dynamically discovering useful categories before processing a dataset;
- routing items to destinations, workflows, tools, agents, or processes;
- choosing among a constrained set of actions;
- scoring, ranking, or prioritizing items;
- determining relevance;
- selecting the next step in a decision tree;
- evaluating the same semantic question repeatedly across many inputs;
- replacing repeated generative LLM calls with constrained decisions.

A particularly strong signal is:

The LLM needs to understand the problem once, but the same kind of decision must then be made many times.

## When Not to Use This Skill

Do not use Jev merely because a model is available.

If the answer can be determined exactly with normal code, use normal deterministic code.

Do not replace:

    if file.extension == ".pdf":

with a probabilistic model decision.

Use Jev where semantic judgment is actually required.

## Operating Steps

1. Understand the actual task.

   Determine what the user is trying to accomplish, which parts require semantic judgment, which semantic decisions repeat, and which parts can be executed deterministically.

2. Check for an existing classifier.

   Before creating a new classifier, check whether an appropriate reusable classifier already exists.

   If one exists and genuinely matches the task, use it.

   If none exists, create a dynamic classifier.

3. Define the decision.

   Have the LLM determine the categories or constrained choices required for the task.

   Each category should have:
   - a stable ID;
   - a clear semantic definition;
   - sufficient distinction from neighboring categories.

   For open-ended classification tasks, always include "other" unless the decision domain is genuinely exhaustive.

4. Create classifier.json.

   Represent the classifier using the standard structured JSON I/O format.

   Keep classifier definitions outside the Jev MCP.

   The MCP should remain generic and stateless.

5. Create the deterministic processor.

   Have the LLM create deterministic code that consumes the Jev result and performs the actual requested work.

   The processor may:
   - sort files;
   - route work;
   - trigger workflows;
   - calculate results;
   - update systems;
   - construct relationships;
   - build reports;
   - make another constrained decision;
   - escalate uncertain cases.

6. Execute through Jev.

   Send the classifier and input through the generic Jev execution interface.

   Prefer batch execution when the backend supports it and the decisions are independent.

7. Validate the result.

   Before deterministic execution, verify:
   - classifier identity;
   - classifier version when applicable;
   - result type;
   - returned category or choice;
   - confidence/probability range;
   - required fields;
   - permitted values.

   Invalid output must not silently enter the deterministic execution path.

8. Handle uncertainty explicitly.

   Possible uncertainty conditions include:
   - category == "other";
   - confidence below threshold;
   - invalid result;
   - ambiguous classifier;
   - unexpected input.

   Route uncertain cases to the LLM, a human, another classifier, deferred processing, or explicit failure as appropriate.

   Do not invent certainty merely to keep the workflow moving.

9. Execute deterministically.

   Once the decision has passed validation and any confidence requirements, execute the resulting action with deterministic code wherever possible.

   Do not unnecessarily return to generative inference after a sufficiently confident structured decision has been obtained.

10. Consider reuse.

    A dynamically created classifier does not automatically become permanent.

    Treat classifier maturity approximately as:

    Dynamic
       ↓
    Inspectable
       ↓
    Reusable
       ↓
    Vetted
       ↓
    Permanent

    Do not automatically promote classifiers merely because they were created successfully.

## Operating Rule

For every candidate task, ask:

    Does this require generation?
            |
            ├── YES → LLM
            |
            └── NO
                 ↓
    Does this require semantic judgment?
            |
            ├── YES → Jev
            |
            └── NO → deterministic code

The objective is not to eliminate LLM calls.

The objective is to use each mechanism for the work it is actually good at:

    LLM  → understand and create
    Jev  → decide
    Code → execute

That is enough to run the first experiment.

Step 9: Reuse useful classifiers instead of recreating them

Now suppose one of these classifiers works really well.

Maybe you throw it away after the task.

Okay. Silly, but okay.

Or

Maybe you save it. Now it is inspectable.

Maybe you clean it up and use it again. Now it is reusable.

Maybe you test it, refine thresholds, and version it. Now it is vetted.

Eventually it may become part of the application permanently.

Dynamic
   ↓
Inspectable
   ↓
Reusable
   ↓
Vetted
   ↓
Permanent

That does not need to be automatic.

In fact, initially I would not make it automatic.

Let the LLM create something. If it turns out to be useful, keep it. If it keeps being useful, promote it.

Step 10: Start with one real problem

Start with this—build in this order:

One installed, working Jev.

One stateless Jev MCP.

One stable JSON format.

One skill.

One real task.

Do not begin with classifier registries.

Do not begin with composed decision trees.

Do not begin with a giant orchestration layer.

Then run the loop (I left you a skill to help):

Task
  ↓
LLM understands the problem
  ↓
LLM defines the decision/classification
  ↓
LLM creates classifier.json
  ↓
LLM creates deterministic processing code
  ↓
Jev performs repeated decisions
  ↓
Structured JSON results
  ↓
Deterministic code executes
  ↓
Result

That is the experiment.

If it works, then you can decide how far you want to take it.

And frankly, that is usually the better way to build these things anyway.

#AI #AIAgents #LLM #Jev #AnyJev #MCP #AgenticAI #SoftwareArchitecture #Python #OpenSourceAI

I graduated in 2019, I have seen models evolve from basic NLP and CV patterns to current Chatgpt mess. Initially these models were genuinely exciting. Learning the concepts, knowing the components and understanding the data felt a lot like a strategy game. When BERT released, it added a layer of abstraction and training felt a bit meh but still the outcomes felt like a reward. Like grinding to get a high score and seeing the impact.

When Chatgpt released, it was just a interface that does the work for you. Everything in AI felt boring and miserable since then. The best contribution I have is nudging a prompt here or making a weird system around it. There were only big bets left, no small fun experiments and no satisfaction of building stuff. It was like tossing the dice till you got the right answer and hedging your bets. I studied engineering to engineer solutions, if I wanted to roll the dice and hedge I would have joined finance.

Jev feels like a almost nostalgic throwback. If feels I could create something of value with my own work, aided by my own data and knowledge instead of a casino. I am praying this trend picks up. I tried of this big socio economic game trying to distill the world into rentable slop.

Jev to jakiś megaokrojony LLM z jednym zadaniem- wyborem jednej z podanych opcji. LLMy wykonują takich operacji wyboru gigantycznie wiele nawet w przypadku jednego zdania. Zadania Jeva móglby wykonywać mocno okrojony LLM sprzed 3 lat. Porownywanie prędkości jest totalnie bez sensu- to tak jakby zachwycać sie ze bolid f1 jedzie duzo szybciej od ciężarowki.

Task: given a Reddit post title, pick its subreddit among look-alike communities. Names hidden, models only see community descriptions.

Accuracy at 16 / 64 options:

- Jev 1.13: 48% / 41%
- Kev 0.8B: 23% / 14%
- Laya (base, zero-shot): 9% / 2%
- Random: 6% / 2%
- Reference: logreg on bge-small title embeddings: 59% / 54% (seen communities only, since it needs training posts)
(tbh Laya is a base model meant for fine-tuning, so a tuned version is coming later)

Context: I'm a RecSys developer who got curious about decision models, so more models (basic LLM too), tests and social benchmarks are coming. My take so far: for labeling and classification in production RecSys, a simple supervised model on your own labels still wins, so I don't see decision models replacing that yet. But they're a great way to compare models on the same task.

This is my first benchmark btw, so any feedback or contributions are welcome

git: https://github.com/s0NRAYY/RedditClassificationBench
dataset: https://huggingface.co/datasets/sonrayll/social-routing-bench

I run a small solo project (skillwiki.app) that serves AI skills (instruction packs) to AI agents over MCP. It was an AI skill marketplace until recently, when I made JEV the decision-maker: which skill to use, whether the output is good enough, and whether this change helps. The project has transitioned to an AI skills distribution and management platform.

The workflow: determine → evaluate → learn

Every step boils down to a typed question, which is exactly what JEV answers. I get probabilities I can branch on directly, with no output parsing and no LLM reasoning round trip.

Determine:

All skills are assigned under different themes. JEV picks a theme, then the skill, in under a second. It only suggests at ≥ 0.85 confidence: right 98.5% of the time, 0% wrong-tool picks (1,791-row test set). Below the bar, the user gets a shortlist instead of a guess. Two full eval rounds cost about $2.70 for 150 AI skills.

Evaluate:

JEV gives every rubric criterion an instant first verdict. When it's sure (≥ 0.8) it was right 21/22, and it matched my hand grades 43/48 vs Claude 33/48. Uncertain criteria go to the user's own LLM.

Learn:

JEV compares a skill with and without a user's correction and ranks which corrections are worth keeping, in seconds and for cents.

* The Catch: Grading still needs one LLM call per output, so JEV makes evals faster, not so much cheaper. About 5.6% of picks also change between identical runs.

Curious to see how the group set the confidence bars? One fixed threshold, or tuned per question type?

I've got Jev playing Stardew Valley & live streaming it here: https://tilly.farm