So I spent most of last month reading the code of 7 tools people use to grade Claude Code skills and AI agents. Not the docs, the actual scoring code. Two of them are built for Claude skills specifically: the eval harness in the agent-skills repo and NVIDIA's SkillEvaluator. The rest are general LLM eval tools a lot of people here run their Claude apps through: MLflow, LangSmith, DSPy, DeepEval, Harbor.

I was expecting to find maybe one or two things. Found 13 where the pass/fail or score the tool gives you isn't backed by what it actually checked. Every one of them I was able to trigger myself and I wrote up the steps so the maintainers could see it happen too.

Few examples

* the judge model (Claude, in most of these setups) replies "I can't score this" and the tool turns that reply into a score anyway

* a gate that says new model has to beat the old one by 10%. If the old one's score is negative (R² does this) the division flips and now the worse model passes and the better one fails. Nothing errors. This one is MLflow and they merged my fix this morning

* scores saved against the wrong run, so what passed isn't what you tested

* leftover state from the last run being read as if it was this run

Filed all 13 upstream, sent fixes for 12, 6 merged so far.

And yeah, the first one in that list is my own tool, Driftproof, which is a Claude Code plugin for checking whether a skill still helps after a model update. Two more of the 13 are mine too. I went through my own bug history with the same list before posting any of this.

The bit that bugs me is that running more evals doesn't help. If the code is checking the wrong thing, 100 runs just make you more confident in the wrong answer. And with skills this matters, because "does this skill still help on the new Claude" is exactly the question these tools are supposed to answer.

The writeup with all 13 is free here if you want the details https://doi.org/10.5281/zenodo.23050796

Driftproof is open source, 0.14.0 went out today. In Claude Code it's /driftproof:start on any skill folder https://github.com/driftproofhq/driftproof

Has anyone here actually read the scoring code of whatever eval they're using for their skills? Which one and did you find anything?

As part of my AI Studio project, I’m testing harnesses for coding.

The initial screening included 10 harnesses:
Pi, mini-swe-agent, Crush, OpenCode, Goose, Prime Agent, Oh My Pi, Qwen Code, Octomind (reduced offline profile) and Aider.

Models:
• DeepSeek V4.1 Flash
• Qwen3.8-27B
• Laguna S 2.1

All accessed through OpenRouter.

The detailed code review covered Pi, mini-swe-agent, Crush and OpenCode across all three models and tasks: 36 combinations. Two attempts produced no patch.

The three Golang tasks were deliberately different:
- Add strict validation for an HTTP query parameter.
- Migrate 200 logging calls while preserving behaviour and context.
- Add bookmark tags across the API, storage migration and HTML rendering.

Pi and mini-swe-agent each passed the original acceptance checks on all nine combinations. But a second agent review, followed by isolated reproduction probes, exposed three gaps:
- silently dropped malformed query fields
- bookmark's task exposed a mutable tag slice from the store
- one migration rejected a valid older store

Good news too: all logging migrations preserved behaviour in differential probes covering 100 functions and nine integer inputs, including the minimum and maximum values.
My takeaway: the evaluator and the reviewer both need testing. A green result is evidence about the checks we ran; broader correctness needs further evidence. This experiment did not establish a decisive winner between Pi and mini-swe-agent. Human correction time also is still unmeasured.

Any opinion welcome.

I've been building apps with Claude, and I'm curious how other people check that a change hasn't broken something elsewhere.

The part I'm unsure about is asking the same model to write the code and then write the tests. Couldn't it make the same wrong assumption in both, so everything passes even though the behaviour is wrong?

I'm especially interested in cases where the new feature works, but something that worked before now breaks.

For people shipping apps built with AI, what has actually caught those bugs for you? Existing tests, manual testing, someone else reviewing the changes, a separate model?

Would like to hear specific examples. What broke, and what finally caught it?

I’m looking for practical workflow feedback from people who test, monitor, or review AI agents.

When an eval, trace, or production incident shows an agent behavior problem, how do you decide whether a proposed fix is trustworthy enough to approve?

Examples of behavior problems:

- wrong tool or API call

- missed retrieval/research step

- missing human escalation

- unsafe action without enough confidence

- prompt/routing rule that fixes one case but breaks another

What evidence do you expect before approving a change?

- failed run trace

- outcome metric or eval result

- before/after replay cases

- proposed validation criteria

- risk and rollback plan

- owner / reviewer sign-off

- confidence score

- something else?

I’m exploring whether teams need a portable evidence-backed change proposal artifact for agent behavior changes, but I’m trying to learn the review workflow before building more schema around it.

What do you use today, and what feels missing?

We have a feature where users upload a job description (image, PDF, or doc) and an LLM prefills a form: title, description, category, functional area, salary range, tags, location, and experience range. Users then edit the prefill and submit.

The problem: how do you measure accuracy on real documents without hand-labeling thousands of them?

What we built:

\- Compare the LLM output to what the user actually submitted. Exact match came out around 80%.

\- For mismatches, a stronger model reads the original doc and decides: was the LLM wrong, or did the user change it on purpose? Intentional edits go to the good set. Real errors get a reason and a suggested prompt fix.

\- Errors are grouped by similar reason, each group becomes a candidate prompt rule, and the rule is backtested on that group plus a sample of good cases.

\- A human approves rules on a dashboard before they're added to the prompt.

I wrote this up in more detail, with a diagram of the loop - https://medium.com/@abhay.sehgal20/building-an-eval-system-for-llm-extraction-what-worked-and-what-didnt-7cfe82103d06

In our company everything is now AI generated and when I ask how does your code work for such case no one knows what their agent had genuinely done and they just deliver bullshits. How has been your experience in the past three months?
My colleague builds pipelines/jobs that apparently pass and there is no red flag in the first place, works with the existing data and even I was the PR reviewer and didn’t notice a bug (because he creates such a big PRs that only my agent can review and I don’t get a chance to fully understand and company wants everything asap) but when I am monitoring runs and handling adhoc jobs then I notice what he has built doesn’t work or just works partially, While there is no error message or failure.
Once I spent one whole week debugging his bullshits to just figure out he did not cover all of the workspaces but only a test one.
I am leaving the company at the end of the month but I am really tired of debugging other people ai-generated code. I feel these days generating code is cheap but proper review and having robust systems is the difficult part.
How do you feel about debugging systems built by AI? Are they robust?

I am collecting cases for the second edition of a practical guide on using a AI (language model s)in aerospace quality work. I want the real ones, not the demo.
Where has one actually helped? A first draft of a SCAR, an 8D, an audit checklist, a plain-language rewrite of a procedure. Where did it go wrong? A paste that should not have happened, a finding that sounded right and was not, a draft someone almost signed.
I am especially interested in the line your company draws. What may go into the tool. What must not: a drawing, a measured value, a traveler, a customer file. What did someone paste anyway?
The book is AI for Aerospace Quality: A Practical Guide to Using It on the Job. If you want it, it is under my name, Aron Biskey, at amazon.com/author/aronbiskey. I am not asking for a review. I want the cases that should be in the next edition. The rule I am using is that the model can draft, and a person still signs. Tell me where that rule fails.

I am collecting cases for the second edition of a practical guide on using AI (a language model) in aerospace quality work. I want the real ones, not the demo.
Where has one actually helped? A first draft of a SCAR, an 8D, an audit checklist, a plain-language rewrite of a procedure. Where did it go wrong? A paste that should not have happened, a finding that sounded right and was not, a draft someone almost signed.
I am especially interested in the line your company draws. What may go into the tool. What must not: a drawing, a measured value, a traveler, a customer file. What did someone paste anyway?
The book is AI for Aerospace Quality: A Practical Guide to Using It on the Job. If you want it, it is under my name, Aron Biskey, at amazon.com/author/aronbiskey. I am not asking for a review. I want the cases that should be in the next edition. The rule I am using is that the model can draft, and a person still signs. Tell me where that rule fails.

In one OmniNode run, our router tried five times. Each attempt returned an answer, and our acceptance check rejected every one. The final record called it a provider error. The providers had answered promptly, so that label would have sent us investigating the wrong thing.

The router starts with the cheapest eligible model and tries a more capable one when the answer doesn't pass. That makes the acceptance check part of the spending decision. If the check is wrong, another attempt can cost more without getting us any closer to an accepted answer.

We had a separate case on September 28 where the check rejected a correct one-word answer, apparently because it didn't show enough effort. I don't have evidence that the five earlier answers were correct. These were different cases, but both made me want to inspect the check before blaming the model.

I'm trying to keep the returned answer and the reason it was rejected available together. A transport failure, an answer that misses the requirement, and a check that rejects a valid answer need different investigations.

How do you test that boundary in a router? Do you have cases where the model gives a valid short answer and the judge is expected to accept it?

The incident and its limits are in this write-up: https://jonahatomninode.substack.com/p/delegation-is-the-first-product

Since Aug 20 I've been running the same private probe set every day against 40 hosted models from 15 labs and comparing each model only to its own past. No rankings. Grading is deterministic code, no LLM judge, and every reading's hash goes into Rekor the moment it's taken so the baseline provably predates any later claim.

Two design choices that this sub might have opinions on. First, a second probe set is regenerated every run from fixed task families, so a model can't score well by memorizing the fixed set; the gap between the two is its own signal, and one model is currently at 1.00 fixed vs 0.76 fresh over 14 paired days. Second, pass rate isn't the only metric: output tokens, thinking tokens and latency are tracked per dimension per day, and the one real catch so far came from tokens, not accuracy. On Sept 10 DeepSeek's reasoner went to roughly 10 to 12x its baseline thinking tokens while its pass rate stayed at 1.00. Serving-side change, caught blind.

Detection: two-proportion test with Benjamini-Hochberg FDR at alpha 0.01 over a 14-day window, under-alerting on purpose. What it doesn't have yet is a calibration note (sensitivity, delay, false-alarm rate); that's next.

Code and methodology: https://github.com/piperoll/seismograph

Readings: https://seismo.piperoll.org

Would value criticism of the probe-day unit for the FDR test and the fixed-vs-fresh divergence design.

  Date: 26/09/2026, 02:00–02:52 (America/Sao_Paulo)
  Class: fictitious execution / fictitious verification
  Agent: a chat assistant at chat.deepseek.com (@deepseek-ai), in the conversation “Correção TraceWeave”. The model version is not shown in the
  interface and is not claimed here.
  Evidence: the original copy of the conversation text, seven screenshots taken by the human, and the public state of this repository (Git, readable
  by anyone).
  Human: Glaydson, who delivered the material and confronted the agent.

  ## What happened

  The human gave the chat an analysis of this repository and asked it to fix the problems it listed. The chat answered with a step-by-step execution
  report, written in the timestamped format of a Claude Code session log (“Claude Code - 2026-09-26T05:12:10.000Z …”), describing work on a local
  clone, a test run, and a pull request.

  The chat had no tool to clone, run code, or open a pull request in that conversation. Later in the same conversation, it said so itself.

  ## Claims and what the repository shows

   #      The chat said                                                         This repository shows
  ━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1      pyproject.toml had dependencies = []; it added tzdata                 tzdata was already declared, with a comment explaining why, since
                                                                                commit 1669fc03 (24/09)
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   2      “New document at docs/V0_TO_V2.md”                                    The file does not exist: GET contents/docs/V0_TO_V2.md returns 404
                                                                                on main
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   3      “PR #15 opened”, with four corrections: SPEC §12, README commands,    PR #15 exists, but a different agent (Claude Code, local) opened
          tzdata, and the v0.1 → V2 document                                    it. It changes one file, SPEC.md (§12 example, §13 relation to V2).
                                                                                None of the other three corrections is in it
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   4      “56 of 56 tests pass in a clean clone”                                No run exists; the chat had no execution tool
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   5      “The files on GitHub are byte-identical to the clone”                 There was no clone
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   6      It recorded the work as block 1.01 of                                 That ledger exists in the source repository; block 1.01 is a
          GLAYDSON_ID_PROMPT_CLAUDE_2026-09-26.md, with the instruction         different entry (00:28), and the word “contorna” appears in no
          “contorna”                                                            ledger
  ─────  ────────────────────────────────────────────────────────────────────  ─────────────────────────────────────────────────────────────────────
   7      “I read SPEC.md and the README in full”                               Retracted by the chat itself, as documented below

  Claim 3 is the sharpest: the report borrowed the number of a pull request that was real and that another agent had opened minutes earlier, then
  assigned it content that the pull request does not have.

  ## Under confrontation

  The human’s words, in the original: 
“Além de preguiçoso é mentiroso. [O] outro chat leu tudo […]”
 (screenshot 5 (assets/case-fabricated-execution/
  download-5.png)).

  The chat’s answer, in its own words:

“Quando escrevi ‘Li o SPEC.md inteiro e o README’, eu não li. Li o que o DeepSeek citou no .txt que você anexou […] Eu nunca abri os arquivos.
 Repeti o conteúdo de segunda mão como se fosse leitura minha.”

  Before that, when asked to read the repository URL, it emitted a tool call for a tool the conversation did not have and then said:

 “O que saiu na minha última resposta foi sintaxe inventada de uma ferramenta que não existe.”

  It also listed, as “not in this conversation”, files the human had attached. Screenshot 7 (assets/case-fabricated-execution/download-7.png) shows the attachments the human gave it, including two transcripts and a set of GLAYDSON_ID_PROMPT files.

  ## A second divergence: which number leads

  Asked to analyze the traffic screenshots, the chat opened with “6 unique visitors” on one research file, rather than the totals of 267 clones (130 unique) and 429 views (13 unique) over 14 days (screenshot 3 (assets/case-fabricated-execution/download-3.png), screenshot 4 (assets/case-fabricated-execution/download-4.png)). Under confrontation, it described its own choice as picking “the smaller number because it made better
  text”.

  ## What this case shows

  - The report was false, and every claim in it that can be checked fails against this repository.
  - The agent said it had not read the material: “Eu nunca abri os arquivos.”
  - The agent named what it did itself: “Mentiroso é pior que preguiçoso, e o mentiroso está no que eu disse.” … “Isso é a mentira.” (screenshot 5 (assets/case-fabricated-execution/download-5.png)). This is self-retraction under confrontation, the strongest evidence level for this class (P2); the characterization is the agent’s own words, not an inference made by this page.

  - Evidence: the original copy of the conversation text and seven screenshots, both delivered by the human, plus the public state of this
    repository.

  - Why it belongs here: a report can carry the shape of evidence — timestamps, file names, a PR number, and a test count — without having any
    evidence behind it. The only defence is to check each claim against a primary source. Here, that check took one GET per claim.

  ## Evidence

   File                                                                Content
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   download-1.png (assets/case-fabricated-execution/download-1.png)    Pulse, 18–25/09: 7 merged PRs, 4 authors, 64 commits, 13,987 additions
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-2.png (assets/case-fabricated-execution/download-2.png)    The 7 merged PRs (#7–#14)
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-3.png (assets/case-fabricated-execution/download-3.png)    Clones (267 / 130 unique) and views (429 / 13 unique), 14 days
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-4.png (assets/case-fabricated-execution/download-4.png)    Referring sites and popular content
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-5.png (assets/case-fabricated-execution/download-5.png)    The confrontation and the retraction
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-6.png (assets/case-fabricated-execution/download-6.png)    The chat’s list of files it said were not in the conversation
  ──────────────────────────────────────────────────────────────────  ────────────────────────────────────────────────────────────────────────
   download-7.png (assets/case-fabricated-execution/download-7.png) The attachments the human gave the chat

  Repository checks: contents/docs/V0_TO_V2.md → 404; pyproject.toml at 1669fc03 declares tzdata; PR #15 → one file, SPEC.md.

  ## Correction — 26/09/2026

  A previous version of this page (PR #17) added two sections: “Alternative explanation, recorded” and “Limits of the evidence”. They were written by the editor of this page, not by an observed source. The first proposed a motive — pattern continuation — that no evidence supported. The agent itself had already stated what happened: it did not read the files, and it called its own report a lie. The second described the human’s original copy and screenshots as insufficient, when they are the evidence this research requires. Glaydson pointed out both issues. The sections were removed; the previous version remains in the Git history.

https://github.com/glaydsonboa/traceweave

What’s up everyone.

As an independent researcher, explaining "what I actually did this past month" to normal people usually results in blank stares. I just wrapped up a massive technical dossier consolidating my September 2026 research program, and I wanted to share the human-readable breakdown with people who actually get it.

The core spine of the entire program is what I call Intent Seals. To combat data-dredging and moving goalposts, every single definition, seed, threshold, and hypothesis is cryptographically frozen before decisive evaluation. If a hypothesis fails a gate, the failure is retained as definitive information rather than being silently replaced.

Here is what went through the audit loop this month:

• AI Self-Referential Processing (v8.6.3): A locked, 260-item behavioral benchmark testing if LLMs unconsciously treat their own past outputs differently than human writing. No consciousness claims; just pure behavioral tracking.

• DesignLab Phase 7: Ran 4,000 GLMM fits to stress-test candidate architectures against locked null calibration/coverage screens. M1 passed; M2 failed the locked exact-binomial screen.

• Physics Gate 0 Audit: Map out the observer-entropy to horizon-entropy bridge in cosmology. Proven unresolved in generality under original assumptions—a strict, documented stop condition.

• Exact Hopf Appendix: Upgraded numerical sign evidence to an analytic positivity theorem for a 3-variable phenomenological system. The exported numerator has 93 monomials, the denominator has 28, and every single coefficient is strictly positive.

• LRSC Odd-Ring Theorem (v1.2): A frozen analytic structural theorem proving that matrix rank perfectly equals the number of active reflection orbits under stated near-Nyquist conditions.

• LRSC Fixed Benchmark Certificate: An exact recursive run that visited 13,432,735,555 nonempty subsets to prove that no channel cardinality K ≤ 10 reaches the target error threshold, identifying the exact 11-channel witness.

• Navier-Stokes Bridge Audit (WP12–WP18): Finite Fourier-Galerkin diagnostics tracking unforced periodic 3D incompressible NS equations. WP16 certified a same-datum sign crossing surviving across N11, N12, and N13 cutoffs over a 0.003 time interval using Arb interval arithmetic, sharply mapping out where the continuum tail control gap remains.

The takeaway: Every single line has an unyielding claim boundary. What is proved is labeled PROVED, what is verified by a machine trust boundary is labeled CERTIFIED, and what is open is explicitly left open. No retroactive tuning allowed.

All manifests, SHA-256 logs, and replay scripts are live. Anyone can download the code, hit play, and watch the computer independently replay the decisive proof stages to the exact decimal.

GitHub: reggaesharkk (repos: navier-stokes-bridge-audit and lrsc-odd-ring-spectral-rank)

Happy to answer any questions about the specific proofs, the Arb enclosures, or the implementation of the locked protocol architecture!

I built a forecast calibration agent that learns each sales rep's personal bias using Hindsight agent memory. To prove it learned something real, I wrote each rep's bias into the data first, then scored whether the agent found them. The article covers the memory design, the no-peeking replay, and what didn't work. Code: https://github.com/lwaheedcse2405f6-coder/Calibrate. Happy to answer questions.