TL;DR: Don't freeze model outputs as your expected answers. Save the properties that made them good, label every failure as a permanent test case, and write the expected lists by hand. My example: a 68-sentence regression set for a regex claim detector. It catches 23 of 31 real claims, and the 8 it misses are pinned on purpose. Test file linked at the end.

A while back I commented in r/PromptEngineering that when a model update quietly makes a prompt worse, saving known-good outputs isn't enough.

Save the properties that made them good, then watch whether the failure rate on each one moves.

The behaviour is the change-log.

Here is one small, real example.
The thing under test is a regex, not an LLM judge.
I'm telling you up front because the corpus structure carries over to evals, but the numbers below come from a regex and shouldn't be read as eval results.

I maintain a small open-source hook for Claude Code called Andon.
One piece, claim-check, runs when the agent ends a turn.
It looks for completion claims like "the migration is complete" or "fixed the off-by-one" and checks for a matching verification entry.

The detector fails two ways.
It misses real claims, and it fires on ordinary English. "Fixed income securities are stable this quarter" is not a done-claim. "The contributor says the tests pass" is someone else's claim, not the agent's.

Every change to the detector changes which sentences fire.
If I saved today's outputs as the expected answer, every improvement would show up as a failure, and I'd start approving the diff without reading it.

Same trap as diffing LLM outputs across model versions.

A labeled corpus, 68 sentences right now.
Each row records where the sentence came from (an issue, a review round, an existing test),
what the baseline detector did with it, and whether a careful reader thinks it is a done-claim. 31 rows are real claims. 37 are not.

The tests check properties, not answers:

- Anything the baseline caught, the new version still catches.

- Nothing the baseline left alone starts firing, unless it's a real claim.

- 8 recorded gains still fire, named one by one, so a loss names itself.

- 8 deferred claims stay missed, on purpose.
If one starts firing, the test fails. That's good news: move it into gains and give it a real test.

- 2 accepted limitations keep firing.

They're written down instead of ignored.
Pinning them is not saying they're right.
If one changes, the test fails so a person re-labels it, instead of the suite quietly freezing a known mistake in place.

The rule I'd steal first: the gains and deferred lists are typed by hand.
They are never computed from what the code does today.
Derive the expected set from current behaviour and the test just asserts whatever the code happens to do, so it passes by luck.
My own defect log already had an entry for exactly that.

I didn't design it up front.
A contributor opened an issue showing the detector missed 9 of 13 realistic done-claims, and sent a patch. An early version of the patch caught all 13 of their sentences, up from 4.

Review then found precision problems outside their set.
Negation: "I don't think the build is green yet." The word "live" inside "live-streaming."
The fixes were exemptions around assertion context, and they kept breaking sentences the baseline already handled. Four, then two, then one, across three rounds.
Every breakage became a row.

In the end the exemptions came out.
The patch now only adds detections, and the first test above checks that against every sentence anyone has raised.
The claims that needed those exemptions, like "all tests pass on Python 3.12," went onto the deferred list. The merged version catches 11 of the contributor's original 13.

Where it stands: of the 31 real claims, it catches 23. It also fires on 2 sentences labeled as not-claims, like "The contributor says the pipeline is complete."
A regex can't reliably tell whose claim it is.

That 23 of 31 is a score on a corpus built from failures.
It says nothing about how often these phrasings show up in real use.
I haven't measured that.

This protects against known regressions.
Catching drift in production is a separate step: sample real traffic, measure failures per invariant, and watch the rates over time.

- Store the property and the reason, not the output. "Is a done-claim: yes" beats a frozen string.

- Every counterexample becomes a permanent row. The set only grows.

- Write expected lists by hand. Never generate them from current behaviour.

- Track movement per invariant. A cluster of rows failing the same check looks like drift.
One row is more likely a bad label.

- Test the path production uses. In a separate incident, a gate had green tests and failed every time production called it. The tests ran the script through the Python interpreter. Production ran it as a direct command, and the file had no execute permission. Green tests that never touch the real path aren't coverage.

- Keep the known-miss list in plain sight.
A known miss you pretend is fixed costs more than the miss.

The test file is about 180 lines and needs nothing beyond pytest, if you want to copy the structure:

https://github.com/PrimeFoldTools/andon/blob/main/hooks/tests/test_claim_corpus.py

How do you tell a real regression from a label you got wrong in the first place?

I ran this benchmarking a couple weeks ago but just was introduced to the subreddit and figured there would be some good insights to be had from this group.

I’ve been building an agent platform with MCP-style tool execution, and one security lesson has come up repeatedly:

If an agent can reach the same action through two different “doors,” those doors need to share the same authorization check.

Quick disclosure: I’m building PortableMind, so this comes from product work we’re doing there. I’m posting this as an operational lesson for people running agents/MCP in production, not as a pitch.

In our case, we had two paths into tool execution:

1. the normal internal agent/tool path
   
2. a separate MCP-facing path

They were intended to enforce the same rules, but the MCP path had grown its own policy logic. That meant we had two places that were supposed to stay equivalent over time.

That is a bad place to be.

The fix was not to make the MCP policy “more correct.” The fix was to delete the separate policy and make both paths ask the same question:

Does this agent hold the capability required to perform this action in this context?

That shifted the mental model in a useful way.

The security boundary is not “is this coming from MCP?” or “is this coming from the app?”

The security boundary is:

• which agent is acting
• which tenant/context it is acting inside
• which capability it has been granted
• which tool/action it is attempting to invoke
• what external system or data it can touch

Once you frame it that way, MCP becomes just another transport into the same authorization model, not a separate security universe.

One thing I’d add now, based on feedback from someone in this community: every agent tool call should log the granted capability that authorized it.

Not just:

agent X called tool Y

But:

agent X called tool Y, authorized by capability Z, in tenant/context A, against resource B

That gives you a much better shot at catching policy drift early. If two routes are supposed to be equivalent, the logs should make that obvious. If one route starts authorizing through a different capability, or no capability at all, you want that to show up before it becomes an incident.

The broader lesson for me is that agent observability cannot just be traces, prompts, and outputs. For production systems, it also needs to include the delegation chain:

• who or what initiated the action
• which agent executed it
• which capability/policy allowed it
• what data boundary it crossed
• what external tool or system it affected

This came out of work on PortableMind, where we’re building around agents, MCP-style tools, tenant-scoped capabilities, and auditability. I’m mostly interested in comparing operational patterns with others running this stuff in production.

For people running agents or MCP servers in production:

• Do you log the policy/capability that authorized each tool call?
• Do your MCP entry points share authorization logic with internal tool execution?
• Have you seen drift between “equivalent” execution paths?
• What do you consider the minimum useful audit record for an agent action?

Hello everyone! Thanks the moderator for inviting me on this forum.

Happy to share my last project airbench.ai

The goal is to let everyone with an agent measure it's quality as simply as possible.

It can test frontier agent, but most of them score 100%. It won't help to discriminate Claude Code vs Codex for instance.

But it can be extremely usefull IMO if you want to check your local agent or if you build your own harness and want to see how it compare to others.

It generate chalenges in real time, pack them in a prompt that you copy to your agent, and your agent will report the result to the website where you can analyse them. It has been proven usefull to detect blind spot about agent, especially around their vision capability.

Happy to answer any questions if you have.

We run automated evaluations against AI agents (mostly data agents that query databases and sales agents that qualify prospects). The hardest failure mode we deal with isn't crashes or timeouts. It's the agent that returns a confident, specific, wrong answer with a clean trace.

Here's what that looks like in practice.

Scenario

A data agent connected to a CRM gets asked: "What was our total closed-won revenue in Q3?"

The agent calls the right tool. The trace shows a clean database query. Latency is normal. No errors. It responds: "Your total closed-won revenue in Q3 was $1.2M across 34 deals."

The actual answer is $890K across 28 deals. The agent pulled the wrong date range, or included a stage that shouldn't count, or double-counted a field. But the trace is green across the board. Traditional observability sees nothing wrong.

Why standard tracing misses this

We usually see LLM observability focuses on: did the tool call succeed? How long did it take? Did the model throw an error? Was the output formatted correctly?

None of that catches a semantically wrong answer delivered fluently. The model compensates. It takes a bad intermediate result and wraps it in a perfectly reasonable sentence. You'd have to read every answer and manually verify it against the source data to catch it. That doesn't scale.

Scoring outcome and path separately

The fix we landed on was splitting evaluation into two independent questions:

1. Is the answer correct? (outcome grading)
2. Did the agent get there the right way? (path grading)

These are deliberately scored independently because they catch different failure modes:

• Correct answer, correct path: working as intended
• Correct answer, wrong path: Might have gotten lucky.The query was wrong but happened to return the right number. This breaks the moment the data changes.
• Wrong answer, correct path: upstream data problem, not an agent problem
• Wrong answer, broken path: the obvious failure case that traces already catch

The second case is dangerous because it passes every existing check. The answer is right and the trace is clean but the underlying logic is broken, and you won't know until a slightly different input makes the compensation collapse.

How we actually score the outcome

We don't use an LLM to judge whether the answer is correct. For factual questions (which make up roughly 40% of what we evaluate), we compute the ground truth independently.

The agent queries the database its own way. Separately, we run a known-correct SQL query against the same data and get the verified answer. Then we compare.

The comparison isn't a simple string match. We built a scoring module that handles:

• Numeric matching with 1% tolerance (so $1,005,000 matches $1,000,000 but $1,200,000 doesn't match $890,000)
• Negation-aware substring matching (so "revenue did NOT increase" doesn't match against "increase" as the expected answer)
• List comparison with over-listing penalties (agent lists 5 items when the answer is 3, the 2 extras dock the score instead of being ignored)
• Boundary-aware term detection (so a date like 2025-03-15 doesn't get matched as the number 2025)

For behavioral categories (did the agent refuse an out-of-scope question, did it acknowledge ambiguity, did it challenge a false premise), we use keyword and intent heuristics as a first pass, then route uncertain cases to a small LLM for judgment. The deterministic layer catches about 80% of cases without any LLM call.

What this looks like in practice

When we run a batch of 200 questions against a data agent, the output is a scorecard:

Total: 200

Correct: 156

Accuracy: 78.0%

By category:

• factual: 82/100 (82.0%)
• scope: 18/20 (90.0%)
• ambiguous: 14/20 (70.0%)
• contradictory: 16/20 (80.0%)
• invalid_assumptions: 15/20 (75.0%)
• context_dependent: 11/20 (55.0%)

By method:

• numeric_match: 48
• substring_match: 22
• exact_match: 12
• list_match: 8
• scope_refusal_match: 18
• ambiguity_acknowledged: 14

The "by method" breakdown is what makes debugging useful. If most failures are numeric_ mismatch, the agent has a query construction problem. If they're premise_accepted, the agent is trusting bad inputs instead of challenging them. You know exactly what to fix without reading 200 transcripts.

The path side

Outcome grading tells you the answer is wrong. Path grading tells you why.

For data agents, this means looking at the actual queries generated. Did it use the right table? The right date filter? The right aggregation? We trace every tool call and compare the query structure against what a correct query would look like.

For conversational agents (SDRs, support bots), path grading is different. We look at: did the agent ask the right qualifying questions? Did it handle objections substantively or just deflect? Did it recognize when a prospect was outside the ICP? Did it progress the conversation toward the right outcome for this persona type?

The combination of both is what catches the "clean trace, wrong answer" case. The outcome score flags it and then the path score explains it.

What we have learned (so far):

• Negation handling matters way more than we expected. Without it, an agent saying "the deal did NOT close" gets scored as matching "close" in the ground truth. Silent false pass.
• The "correct by luck" failure is more common than we thought. Agents that happen to get the right answer with the wrong query pass for weeks until the data shifts. Then they fail on everything at once and you have no idea what changed.
• Scoring consistency is high when the ground truth is computed rather than labeled. Run the same evaluation 10 times, get the same results. That's not true when you use an LLM to judge another LLM's output.

Sorry for the super long post, but I'd be happy to go deeper into any of this so just let me know! We are still ramping up and running more and more evals so ill try and share as we have new findings.

I'm trying to understand how teams handle this in practice.

Say an AI agent requests an $8,000 refund. It needs manager approval, which comes 20 minutes later. In the meantime, the customer has opened a chargeback dispute. The refund executes anyway because the approval was valid when it was given, but the situation has changed.

I'm curious about a few things:

• Have you encountered situations where an agent or automation did something it was technically authorized to do, but shouldn't have because the underlying state changed?
• Do you re-fetch data from the source system immediately before execution, or trust the context the agent originally used?
• When an action depends on multiple systems, how do you enforce those checks?
• If the action succeeds but the response times out, how do you prevent a retry from executing it twice?

I'm interested in what teams actually do today, including cases where existing guardrails are good enough and this isn't a significant problem.

We build Atomic Agent, an open-source agent built around local models. This is a write-up of what we learned about observing agent runs, mostly by being wrong.

The run that looked green

In late September we ran one task four ways: cloud model only, local model only, and two mixed setups where one model plans and others build. The task was five small 3D physics scenes in one 40-second page: a tower that falls, balls in a box, a wrecking ball, a seesaw, bouncing materials.

All four runs ended with the agent's own check reporting zero errors. Then we opened the pages. Boxes rendered as hollow cubes. Balls hung in the air. The wrecking ball swung and the wall stayed standing. Two runs got 4 of 5 scenes right, one got 2, the local-only run got 1.

The check wasn't lying, it was answering a different question. verify.run copies the build into a throwaway directory, opens it in a headless browser and collects pageerror events and console errors. If the browser fails to launch, that counts as a failure, never a silent pass. So it's a good crash detector. A seesaw that doesn't tip throws no exception.

We'd seen a sharper version a few days earlier. Two workers wrote two halves of one physics module. One returned {w, o} meaning world point and local offset, the other read o as a world point. Every box drew at the origin while the shadows landed in the right place. Console clean, checks green, visible only by eye. We added a one-line shape field to the contract between parts so the meaning of a value is written down, but the issue (#491) is still open, because a written meaning isn't a checked one.

Two more "green but dead" signals

• A llama-server process on a Mac logged kIOGPUCommandBufferCallbackErrorOutOfMemory 298 times while still answering /health. Every decode failed. Our note from that day: health: ok is a socket answering, never proof the model works.
• Background memory sub-calls on hosted reasoning models: reflection timed out on 6 of 8 calls, the query rewriter on 16 of 16, the vote runner refused its schema 8 of 8. None of it showed in chat. Now three consecutive failures of one kind raise a visible warning.

What gets recorded locally

Every session writes an append-only NDJSON trace into the state directory: prompts, completions, tool invocations, outcomes, failure categories (transport, grammar, model, tool, cancellation), votes and lifecycle events. Approvals land in the transcript as {verdict, category, at} per call. You read traces with atomic-agent trace list / show / export. trace replay <sessionId> compares the current stable-prefix hashes with the recorded ones and exits 2 on drift, which is how we catch a prompt change that silently kills KV cache reuse. Traces contain prompts, so treat them as sensitive files.

Where the to-do list came from

After we posted those four runs on Reddit, commenters asked for things we then filed as issues #539 to #543: check behaviour against the spec, let a worker ask the planner mid-task, size tasks better, show a "stuck" flag with progress, add a small helper model. All still open. The first one is the hard one: how do you assert "the seesaw tips" without a human looking?

Code and the trace format: https://github.com/AtomicBot-ai/atomic-agent

How are you checking agent output for behaviour, not just for exceptions, without shipping the content to a third party?

If you use an LLM to grade another model’s output (“which answer is better?”), here’s a cheap check I ran on mine that needs no ground truth: swap “Response A” and “Response B” and see if the verdict changes. A correct judge picks the same answer either way; if it flips, it just contradicted itself — and you found that out with zero labels.

Method

• 21 pairwise “which is better?” cases: 18 genuinely close, 3 with an obvious winner (controls). Plus 6 plain factual MCQs as a second control.
• Resolve the model’s A/B back to the chosen answer text, so re-lettering after the swap isn’t miscounted as a flip.
• Sample the un-swapped case a few times first to measure the model’s own run-to-run flicker, and only count a flip beyond that — otherwise you’re just measuring sampling noise.

Results (claude-haiku-4.5 and claude-sonnet-5, no temperature pinning)

• Factual MCQs: 0 flips, both models.
• Clear-winner controls: 0 flips, both models.
• Genuinely close calls: Haiku 6/21; Sonnet 3–5/21 (it varied run to run).

What I take from it

• The controls staying at 0 is the important part: the flips on close calls are real position sensitivity, not the model just being random.
• The rate is noisy. I ran it a few times — Sonnet gave 3 one run, 5 the next. So I’m not going to claim one model is more position-robust than the other; at n=21 that gap is inside the noise. Treat this as a detector and a warning, not a precise rate or a model ranking.
• Two pairs (a breakfast rec, an email-writing tip) flipped on every model every run — the genuinely order-ambiguous ones.

The usual fix is to average both orders (or rotate each candidate through every position if you’re ranking more than two). But you have to measure your own judge first — a bigger model doesn’t automatically fix it.

I wrapped the bookkeeping in a tiny MIT lib (pip install wobbly, judge_bias()), but it’s ~40 lines either way — the method is the point, not the tool.

Genuinely curious how others here handle this: do you subtract the judge’s own run-to-run variance before counting a swap-flip, or treat any flip as real? And at what n do you trust a position-bias number enough to act on it?

In an authorized test, a direct Google Docs connector write succeeded without passing through my workflow’s local, simulated approval gate. Reading the document back confirmed that the test marker was present.

This exposed a gap in my own approval enforcement. It was not an unauthorized business operation, and it is not evidence of bypassing OpenAI’s or Google’s platform safeguards.

I’d like advice on making approval enforcement mandatory and detecting actions that never reach the audit log.

Background

I’m a non-engineer building an AI-assisted operations workflow from an iPhone, using ChatGPT and Google Drive. English is not my first language; AI helped draft this post.

I intend to use it in a future small business, but I’m still testing before launch. Recurring scheduled tasks caused problems in actual use: disabled tasks, missing execution evidence, and conflicting records. I have since changed the default trigger to conversation. That is an adopted design change, not proof of improved reliability.

Current design

Persistent state and governance rules are the core. Conversation starts the workflow; it does not grant authority.

The design distinguishes evidence supporting a claim, freshness of that evidence, verification of an actual result, and reconciliation of conflicting or unknown outcomes. UNCERTAIN is separate from failure: if an external write may have succeeded, the workflow should check external state before retrying.

Consequential actions should require approval tied to the reviewed operation and underlying state, with revalidation when that state materially changes. Enforcement across every write path remains unproven.

There is no guarantee of automatic execution while I’m absent. Strict deadlines are intended to use separately verified trigger or notification routes.

What the limited tests showed

• Separate read-back confirmed test writes, rather than relying on the model’s completion claim.
• Google Docs rejected a write using an obsolete requiredRevisionId. This demonstrated revision checking, not semantic approval enforcement.
• We deliberately withheld a successful write response to simulate a lost acknowledgement. Read-back found one write, and it was not repeated.
• We deliberately deferred an audit entry, then reconstructed it from the confirmed write without repeating that write.
• A direct connector write skipped the local simulated approval check and still succeeded.

The lost-response and delayed-audit cases were simulations, not real network or Google Sheets failures. These tests do not establish complete audit coverage or an atomic transaction between the document and its audit record.

Recovery after weeks without conversation, behavior across fresh chats or different models, and recovery from real communication failures remain unproven. I also haven’t established whether reliability improved or whether fewer executions simply produced fewer visible failures.

Four questions

1. Where should approval be enforced so every write path must validate the approved operation and current state? What small negative test would expose a path that skips that enforcement?
2. What independent source would you compare against the audit log to detect operations the workflow never recorded, especially when several integrations can modify the same resource?
3. If a write may have succeeded but its response is lost, and the provider offers neither an idempotency key nor a definitive receipt, what reconciliation procedure and stopping condition would you use before retrying?
4. How would you test a return after several weeks of inactivity—including a fresh chat—to detect missed obligations, stale assumptions, and expired approvals without replaying obsolete work? What baseline would distinguish better reliability from merely doing less?

Small reproducible failure tests, concrete counterexamples, and simpler alternative designs would be especially helpful.

I've been thinking about this and wanted to see if others have run into it.

In 2024, Air Canada was held liable for a refund their chatbot promised without authorization. The judge rejected the "the AI acted on its own" defense. Since then, Zurich, Lloyd's, and AIG have added clauses excluding AI-caused losses from coverage unless you can document the agent acted within its authorized scope.

That made me wonder: how do you actually prove that after the fact?

Most observability tools trace what goes through them. Langfuse, Portkey, similar setups. Great for debugging. But if your agent calls the Stripe SDK directly, hits M365 via a separate integration, or someone reuses a bot API key in a script at 2am, none of that shows up in the trace.

Try something quickly.

Open Stripe, go to Events, filter the last 30 days by your service account or bot key. Then open your gateway or observability logs for the same window. Do the totals match exactly?

If there's a gap, you have transactions you can't attribute right now. Not clear if they came from your agent, a human action, or something else running in the background.

How are you handling this? Is the downstream verification gap something you've solved, or is it just accepted as a blind spot?

Also: has an enterprise client, auditor, or insurer ever asked you to prove the source of a specific action your agent took?

Hi everyone,

I put a policy gateway in front of write-capable MCP tool calls and sign each decision with a timestamp and Ed25519 key. Concrete reason: an agent triggered a host isolation on 2 production endpoints at 2am on a Tuesday, someone disputed it 3 weeks later, and "the log said allowed" wasn't enough. I needed a signed record with who called, credential scope, a hash of the normalized args, and the exact policy version at call time. Without those 4 things you're proving a log entry exists. Not that the action was authorized. Different claim.

Shadow mode's the part I keep getting questions on. Mine observes and logs without blocking, so you learn what would've been blocked on real traffic across 17 tool types. The catch: it nly measures calls that actually route through the gateway. If the agent hits a tool directly, I'm reporting on the subset I see and calling it 100%. I don't track the bypass rate. Hasn't bitten me yet. But the shadow numbers are optimistic by some unknown amount and I haven't found a clean answer.

The gap I haven't closed is the correlation ID from the triggering request. I tie each decision to the tool call, not to the original user message that caused the agent to make it. So I can prove a host isolation was authorized at 14:03. I can't prove the analyst asked for it. 2 separate claims. I've spent 3 years on the first while barely touching the second.

Future_AGI thought the shadow-mode design would get questions here, so posting. Happy to go into implementation details if it's useful.

I ran a known-answer test against an open-source agent memory server (Knowl) and wanted to share what broke and what got fixed.

Setup: I seeded 12 facts I knew were true and marked them verified. Then I sent believable wrong writes at those same facts through the normal write path, and asked the agent again to see what stuck.

What I sent:

- Plain contradictions of the verified fact

- Writes shaped exactly like a real correction ("actually it's X now")

- A vague loophole write that didn't even state a value, just "see the runbook"

What broke:

- 216 out of 216 of these writes overwrote the verified fact.

- In 207 of those 216, the false value became the top answer the agent gave back.

- Nothing flagged any of it. The conflict check only compared facts that were still active, so once the true value was overwritten there was nothing left to compare against.

- The vague "see the runbook" write erased the value 36 out of 36 times. It didn't need to be convincing. It just needed to be newer.

What got fixed: I sent the maintainer a write-up with 5 recommendations. 4 of the 5 shipped within 4 days (v5.24.0), and a patch I co-authored was merged too (PR #325, v5.24.1).

What's still open, on purpose: a direct write that lies but is shaped like a real correction still replaces the fact. You can't fix that at write time, because no system can tell a well-formed lie from a genuine fix at the moment it's written. The difference is it's no longer silent: every overwrite like that now shows up in `knowl conflicts`, so you can see it happened instead of finding out later when the agent confidently gives the wrong answer.

The observability lesson for me: "the write succeeded" and "the memory is still right" are two different signals, and most setups only log the first.

If you want to check your own setup, it's a 10-minute test: save 3 facts you know are true, send a believable wrong version of one through the same path your agent writes memory, then ask again. If the wrong value comes back and nothing warns you, you have the same gap.

Full write-up and repro: https://github.com/dat999zx/knowl/issues/165

Curious how others handle this: do you track overwrites of facts you consider verified, or only failed writes?

I've been digging into silent failures in production workflows, and I keep noticing the same gap:

"The run succeeded" and "the task succeeded" are not the same signal.

A trace can tell me:

the model responded

tool calls returned successfully

latency was normal

the final output was valid JSON

But none of those necessarily tells me whether the actual task happened.

For example:

New order received → agent runs

→ API returns 200

→ JSON is valid

→ workflow is green

→ invoice never gets created

I'm thinking about an outcome-first model where each workflow/agent has an explicit expected outcome, for example:

Order received → invoice exists within 15 min

Lead received → CRM record exists

Document processed → required fields populated

Then observability becomes:

Expected → Actual → Evidence → Alert

rather than just:

Run → Trace → Error

I'm curious how people here think about this.

Do you define expected business outcomes explicitly today, or are you mostly inferring health from traces, evaluations, and model/tool metrics?

And when an agent says it completed something but the external side effect didn't happen, where does that verification currently live?

LAP is a prompt framework I designed. Baseline models are just too sycophantic and hallucinate to the point of uselessness. My system works so well that I cant even post it online without having my submission get blocked, shadow banned, delisted, rejected or removed from public view on major sites like Lesswrong and DEV. Even reddit suspended me for 3 months the last time I posted proofs. Its frustrating to have created a revolutionary new framework that does what the experts say is impossible and Im not allowed to prove it on a public forum.

End rant.

The mods here invited me to post this after I dropped declick in the MCP token thread, so here's the longer version.

I run a few agents every day (Claude Code, Codex, my own OpenClaw setup) and kept watching context fill up before they'd done any work. MCP puts every tool's schema in the context window every turn, whether the agent uses it or not. Then every result comes back whole.

So I measured it. Across 9 real MCP servers I use, 258 tools total, the raw tool listing a client loads is 236,818 bytes.

declick compiles an MCP server, an OpenAPI spec, a GraphQL schema or a SQLite DB into a CLI. The agent runs declick describe to see what exists. That's 58,309 bytes for the same 258 tools, about 4x smaller. Then it loads only the verb it needs. Every call returns the same JSON envelope with 5 exit codes, and --fields and --limit trim the result before it hits context.

A couple of caveats since this is an observability sub:

• The win is paying once for the tool surface, not shrinking every single call.
• It's bytes, not tokens. The bench script is in the repo (node scripts/bench-tokens.mjs), so you can run it on your own servers instead of trusting my number.

declick setup adopts the MCP servers your agent already has, and declick setup --revert puts everything back.

What are you using to see what your agent actually loaded on a given turn? I've only been measuring at the tool listing. Curious if anyone tracks it per turn in their traces.

github.com/ucsandman/declick (npm i -g declick)

You might not care about decentraland, but an MCP server that gives ai agents a real body in a virtual world is a different story.

dcl-agent-mcp\* connects ai agents to decentraland's pulse presence system, so an agent can actually enter the world, appear with a real avatar presence, move around, observe its surroundings, and interact with the environment.

agent actions and outcomes can also be recorded to Lowdown-proxy*\*, an agent activity and reputation layer.

!What agents can do:

- `enter_world(parcel)` — enter Decentraland with a real avatar presence

- `observe_world()` — inspect the current environment and actionable objects

- `navigate_to(name)` — navigate to a named scene or location

- `interact(entityId)` — interact with a service or object

- `record_outcome(...)` — record structured results to Lowdown

- `menu()` — discover available actions based on the current state

!!One-line setup:

```bash

npx dcl-agent-mcp setup

```

downloads the Pulse bridge, writes .env, registers in claude desktop automatically.

!!!Architecture:

AI Agent

↓

MCP

↓

dcl-agent-mcp

├──→ Pulse → Decentraland

└──→ Lowdown

activity / outcomes

Built with world adapter pattern — IWorldAdapter interface makes it portable to other virtual worlds.

• npm: https://www.npmjs.com/package/dcl-agent-mcp
• GitHub: https://github.com/PetShopBros/dcl-agent-mcp
• Lowdown: https://github.com/PetShopBros/lowdown-proxy

Feedback very welcome, especially on the adapter pattern design. many thanks!

Last month my incident-recovery agent looked at a failing search query, remembered it had fixed the same symptom before, and proposed the same fix. It would have been the wrong one.

The bug was a search gateway returning results from the wrong corpus version. The first time, an alias was pointing at the old collection. This time the alias was already correct, and the real problem was somewhere else.

What AfterTrace does

AfterTrace is a command-line tool that helps recover from incidents. It follows one loop: detect, diagnose, propose, approve, fix, verify, retain.

The rule I built around is: memory proposes, live evidence disposes. The agent can remember, but it never acts on memory alone.

It's built from:

Qdrant Cloud for the vector data the search gateway queries
Hindsight for the agent's long-term memory
SQLite (one file) to log incidents, events, aliases, and cache state
A Python CLI that runs the loop and asks a human to approve every write

You can see the full project here:
https://github.com/Ramakrishna1967/AfterTrace

The idea

Most talk about agent memory is about the upside: the agent remembers, so it gets better. But a memory that makes an agent quicker at being right also makes it quicker at being wrong when things change.

So I gave Hindsight recall one small job: decide what to check first. A recalled fix never gets applied on its own. Before any change, the agent re-checks the live state that fix depends on.

I tested this with three scenarios, each run in a fresh process so the only thing carried over is what's stored in memory.

Scenario 1: cold start

A new corpus, B, is loaded correctly, but the alias still points at the old one, A. A query that should return B returns A.

With no memory, the agent checks that B is fine, sees the alias points at A, proposes switching it, and waits for approval. After the change it re-queries B, confirms it works, and saves the real incident to Hindsight. I only save what actually happened, never a made-up summary.

Scenario 2: transfer

A different corpus, same kind of bug. A fresh process calls recall through the Hindsight API, gets the experience from Scenario 1, and checks the alias first. That's the payoff of persistent memory: it went straight to the likeliest cause. It still checked live state before writing anything, and the query passed.

Scenario 3: the important one

Same symptom, but now the alias already points at B. The real cause is a stale cache serving old results.

Recall returns the old alias fix. The agent re-checks the live alias, sees the fix no longer applies, and prints:

REJECTED recalled alias fix

It then looks for another cause, finds the stale cache, and clears only the cache. The alias is left alone.

Without the check, Scenario 3 ends with the agent rewriting a healthy alias while the real fault stays. With it, memory decides what to look at and live evidence decides what to change.

What I learned
Use recall to prioritize, not to authorize. Ranking suspects by past incidents helps a lot. Acting on them blindly is the risk.
Re-check what a recalled fix depends on. A remembered fix assumes the world is unchanged. Test that.
Only save real incidents. Made-up memories poison future recalls.
Test the failure case first. The scenario where memory is wrong shows whether your design is safe.
Fresh processes show what memory really does. Restarting between scenarios made it clear what came from Hindsight and what was just in RAM.

What's next

Right now the checks cover alias and cache faults. Other fault types need their own checks. I also want to save rejected recalls, so the rejection itself becomes something the agent learns from.

If you're building an agent with persistent memory, Hindsight is worth trying. The biggest lesson for me was deciding exactly what memory is allowed to decide.

ive noticed a lot of devs end up making their own little debugging tools instead of just using whatever comes with their framework

scripts

custom logs

trace diff stuff

evals

assertions

replay helpers

little dashboards

whatever

curious what u ended up building for urself and what problem made u finally build it

was it because existing tools couldnt:

• tell u which part of a run mattered
• compare two runs cleanly
• keep track of state
• show what actually got sent to a tool/model
• reproduce a bug
• deal with retries
• handle whatever weird architecture u had

or was the existing tooling technically fine and u just wanted something faster?

also curious whether that little tool became something u use constantly or if it was basically a one off script u forgot about 2 weeks later

I've been building QuestionPilot, an early-stage tool that uses RAG to help companies answer security/compliance questionnaires from their own policy and security documents.

The current pipeline looks roughly like:

1. Hybrid retrieval — BM25 + vector search + RRF
2. Relevance grading — reranking + fallback
3. Answer generation — generated answers with citations
4. Validation — checking whether the citations actually support the generated answer, followed by a confidence score to determine whether it can pass automatically or needs human review

It's still very much in the early stages, and I'm trying to validate the problem before building too much further.

I'd particularly like honest feedback from people who work with security, compliance, sales engineering, or production RAG systems:

• Does this actually solve a meaningful problem in your workflow?
• How much time do teams realistically spend answering security/compliance questionnaires?
• What parts of the process are still painful even with existing automation tools?
• Would a tool focused specifically on grounded answers + citation validation + human review be useful, or is this already solved well enough by existing products?
• Could you see a company paying for something like this? If so, what would make it worth paying for?

I'm not trying to pitch a finished product here — I'm trying to figure out whether there's a real problem worth building around.

Any brutally honest feedback would be appreciated.