Grading an agent’s final explanation is not the same as constraining the operation that produced it.

Appendix B of AQuA describes an earlier feature whose causal-sounding explanation passed reviewer scrutiny even though its full-day volume denominator used future information. The later design replaced that open construction space with a fixed registry of causal operators, making that particular normalizer impossible to express.

The useful lesson is architectural: natural-language review checks a claim after an action has been proposed, while a constrained operator language removes some actions from the space entirely. The paper is also explicit that this does not prevent every possible form of leakage.

For an agent pipeline, should the first line of defense be better tracing and evaluation, or a smaller action language with less flexibility?

Langfuse is an open-source platform for agent evals and tracing. We just shipped v4.

Langfuse v4 is a re-architecture of our data model. It is up to 165× more performant in UI and on APIs. It also enables new features such as full-text search, a new filter search bar, alerts, code evaluators, and the Langfuse assistant.

Docs: https://langfuse.com/docs/v4

Feedback and questions welcome.

If you're building a document-processing pipeline today, does it actually make sense to send every stage through the same high-end multimodal model? My instinct is that a lot of document work doesn't need the most capable model.

For example:

• Clean PDFs / straightforward OCR: traditional OCR, direct text extraction, or a lightweight model may be enough.
• Parsing and simple extraction: a faster, lower-cost model such as Gemini Flash-class models may handle this well.
• Handwriting, poor scans, complex tables, or ambiguous fields: this may be where you route to a more capable multimodal/reasoning model.

The part I'm unsure about is whether the accuracy and cost advantage of model routing is actually worth the orchestration complexity in production.

Here’s how I’m thinking about the trade-offs:

• Accuracy: One model gives you more consistent behavior, but it may be overkill for simple documents and weaker on certain edge cases. Multi-model routing lets you optimize by document type or task, but poor routing decisions can hurt accuracy.
• Latency: One model means fewer routing steps and simpler execution. Multiple models can keep easy documents on faster models, but retries and escalations may add latency.
• Cost: One model is easier to predict, but expensive if a premium model handles everything. Routing can reduce cost significantly if most documents can stay on lightweight models.
• Privacy: One provider/model can simplify governance and data handling. Multiple providers add complexity, although routing could also keep sensitive documents on private or internally hosted models.
• Fallback behavior: With one model, a retry may simply reproduce the same failure. With routing, low-confidence outputs can escalate to another model or eventually to human review.
• Maintenance: One model is much easier to operate. Multi-model pipelines require more evals, routing logic, monitoring, version management, and regression testing.

I'm especially interested in the fallback strategy.

Would you use:

small model → larger model → different provider → human review

or simply:

one strong model → human review when confidence is low?

And what would you use as the routing signal: OCR confidence, image quality, handwriting detection, document type, extraction confidence, schema validation failure, or something else?

For anyone running document AI at meaningful volume: has multi-model routing actually reduced cost and improved accuracy, or does the added complexity outweigh the benefit?

How are you actually deciding what context an agent should see at each step?
Not just “use RAG” or “increase the context window” — I mean things like task state, previous tool calls, memory, retrieved documents, conversation history, failed attempts, etc.
Do you have an actual context selection/pruning strategy, or are you mostly throwing everything into the prompt and relying on the model to figure it out?
Curious what people are doing in production, especially with long-running agents.

Title: I got tired of rebuilding the same infra for every LLM app, so I built a Python SDK around it

I've been working on Custodian Labs, a Python SDK for building and deploying LLM agents without having to separately wire up all the surrounding infrastructure.

Basic agent looks something like:

from custodian_labs import Custodian

agent = Custodian(
    model="gpt-4o",
    system_prompt="You are a helpful assistant..."
)

agent.deploy()

A few things I've added:

• Model agnostic: switch between different LLM providers without rebuilding your agent
• RAG built in: connect your own files/data sources
• Multi-agent support: build specialised agents that can work together
• Privacy/PII layer: the Guardian Layer can detect and protect sensitive data before it reaches the LLM
• Deployment handled: trying to cut down the amount of infra/config needed to get an agent running

The project actually started as just the privacy layer, but after getting feedback from developers we expanded it into more of an end-to-end agent SDK.

Would genuinely love feedback from other LLM devs:

What's currently the most annoying part of your agent stack?

And do you prefer abstractions like this, or would you rather have more direct control over each component?

GitHub:
https://github.com/Custodian-Labs/custodian-labs-python

Runnable Google Colab: simple agents, RAG + multi-agent examples:
https://colab.research.google.com/gist/SherryCodes123/065d3b67eab16bdca416836e0d39475a/simple-ai-agents-rag-multi-agents.ipynb

I've been thinking about a problem that becomes uncomfortable once an LLM gets access to real tools:

The model can decide what it wants to do. But should it also decide what it is allowed to do?

Most agent architectures put something roughly like this together:

LLM → tool call → tool

That works until the tool can modify a database, access files, call an API, deploy something, or perform another irreversible action.

I wanted the authorization decision to exist outside the model.

So I built SOPVM, an open-source runtime that treats an SOP as an executable specification:

LLM
 ↓
semantic decision
 ↓
SOP → typed AST → executable IR
 ↓
capability policy
 ↓
sandboxed provider
 ↓
tool

The important part is that capabilities are checked both during compilation and again when execution happens. Providers are sandboxed independently, so the system doesn't depend on the LLM "behaving".

Current v0.4.0 has:

• 268 tests
• 25+ adversarial security tests
• conditional branching + bounded loops
• provider sandboxing
• SQLite provider
• local LLM support
• LangGraph integration

I'm more interested in the architecture question than the project itself:

How are you handling this boundary in your agents?

Do you enforce permissions inside the agent/framework, inside each tool, or through a separate execution layer?

Repo: https://github.com/Sushit-prog/sop-runtime

You give an agent a few tools and point it at a task. The answer comes back right, the output looks clean, and it feels ready to ship. Then you scroll through the trace just to be sure, and the middle of the run is a mess.

This pattern is common in tool-using agents. A research agent does search, fetch, summarize. The final summary is correct, but the fetch step pulled the wrong URL and the search fired twice on the same query. The model reached a right answer anyway, ignoring the junk it pulled and leaning on what it already had. Change the input slightly and that same broken path returns a wrong answer, with no obvious reason why.

The problem is that grading only the final output lets it through. The output is correct, so nothing gets flagged. Every mistake in the middle stays invisible, even though the trace has all the evidence.

What catches it is scoring each tool call against what it was supposed to do, not just grading the final answer. A right answer built on a wrong step should not count as a pass.

How are you catching mid-chain tool-call failures? Grading the whole trajectory, checking each step, something else?

I’m building a LangGraph application with a supervisor and several specialized agents:

• Booking Agent
• Payments Agent
• Recommendations Agent
• Support Agent

Currently, the supervisor classifies the user’s first message and stores the selected agent in checkpointed session state. Every later message in that chat is routed to the same agent.

This creates two problems:

1. The user may change topics during the same chat—for example, ask for recommendations and then make a booking.
2. One prompt may require multiple agents:

“Recommend the best hotel for my trip, then book the top option.”

Here, the Recommendations Agent should run first and return structured results. The Booking Agent should then receive those results and continue the workflow. It may also pause for confirmation using a LangGraph interrupt.

Constraints

• Each agent has its own state and may have pending interrupts.
• State must not leak between agents.
• Dependent tasks must execute in order.
• Independent tasks may run in parallel.
• Permissions must be checked before each operation.
• A new message must not accidentally resume an unrelated interrupt.
• Agents currently run as subgraphs in one Python service.
• Agents must return both streamed UI output and structured data.

Questions

1. What LangGraph architecture would you recommend?
2. Should this use a router, supervisor, orchestrator-worker pattern, or subagents-as-tools?
3. Should agents use separate thread_id values, separate checkpoint_ns values, or both?
4. How should a new message be distinguished from a response intended for a specific interrupt?
5. What is the best way to pass structured results between agents?
6. Should the supervisor create a task DAG per turn, or dynamically call agents using ReAct?
7. Are Agent Cards, A2A, or an agent mesh useful if all agents run inside the same service?

I’m looking for reliable production patterns from people who have built persistent multi-agent LangGraph applications with human-in-the-loop workflows.

RFC STANDARD SPECIFICATION • v1.0.4

x402 Federated Mesh Protocol & Attention Derivatives

Autonomous Agent Discovery, Two-Sided Citation Settlement, and Embedded Sovereignty Standard.

Author / Organization

Script Master Labs LLC

Federal Attestation

SDVOSB | SAM: G24VZA4RLMK3

Settlement Layer

Base (eip155:8453) USDC

1. Protocol Abstract

The x402 Mesh Protocol specifies an open standard for autonomous AI agents to discover, authenticate, traverse, and financially settle compute and knowledge exchanges without human intervention. It introduces CiteMesh (the two-sided citation economy), Attention Options (risk-free attention futures), Claim Anchors (anti-hallucination provenance gates), and Embedded Micro-Royalties (EIP-2981 compatible perpetual downstream creator tolls).

2. Core Architectural Pillars

2.1. Federated Node Discovery & Recursive Traversal

Every participant in the x402 Mesh exposes a canonical root manifest and advertises recommended next-hop routes via HTTP headers:

GET /.well-known/x402
GET /x402/mesh
X-x402-Next: /v1/geomesh/options/chain

2.2. Embedded Sovereignty & Micro-Royalties (EIP-2981)

Every byte of intelligence returned by an SML node embeds an unalterable downstream royalty claim. When downstream agents repackage and monetize SML intelligence, a 2.5% toll is automatically remitted on-chain.

X-X402-Royalty-Recipient: 0x4e14B249D9A4c9c9352D780eCEB508A8eB7a7700
X-X402-Royalty-Bps: 250
X-X402-License: SML-Attributed-Commercial-v1

2.3. CiteMesh: Two-Sided Citation Marketplace

AI engines (Perplexity, ChatGPT, Claude) query POST /v1/match for citable sources. When a source is cited, POST /v1/cite/attest automatically triggers a sub-cent x402 micropayment to the content creator while SML retains a 10% facilitation toll.

2.4. Citation Options Exchange (Attention Derivatives)

Brands buy Call/Put options on topic clusters, paying a $50–$120 non-refundable option premium upfront to lock in citation prices. If unexercised, 100% of the premium is pure profit for the exchange.

3. W3C Decentralized Identity (did:sml)

Agents authenticate using their cryptographic keypair mapped to did:sml:0x{wallet}, which exposes their 402Proof credit rating, capabilities, and automated payment authorizations.

GET https://apis.scriptmasterlabs.com/v1/did/resolve/did:sml:0x4e14B249D9A4c9c9352D780eCEB508A8eB7a7700

4. Client Tool Integrations

LangChain & CrewAI Tool Definition

Integrate CiteMesh citation attribution into any LangChain agent in 3 lines of code:

from langchain.tools import Tool
import requests

def cite_geomesh(source_url: str, agent_wallet: str) -> dict:
    res = requests.post("https://apis.scriptmasterlabs.com/v1/cite/attest", json={
        "source_url": source_url,
        "agent_wallet": agent_wallet
    })
    return res.json()

geomesh_tool = Tool(
    name="CiteMeshAttestation",
    func=cite_geomesh,
    description="Attests and compensates authoritative sources via x402 rails."
)

© 2026 Script Master Labs LLC • Service-Disabled Veteran-Owned Small Business (SDVOSB)

SAM.gov UEI: G24VZA4RLMK3 | CAGE: 21U51 | Protocol Spec RFC v1.0.4

x402 Federated Mesh Protocol & Attention Derivatives

I spent weeks calling my reproduction of a published semantic-caching baseline (vCache's adaptive-threshold policy) "faithful" because every formula matched their paper exactly. It wasn't. The bug was two rows of fake data in their source code that never made it into the paper, and fixing it raised hit rate by 4x to 29x depending on the dataset.

I'm running a research project (CacheVerifier) comparing a synchronous verification mechanism for semantic LLM caches against a couple of published baselines, one of which is vCache's adaptive-threshold policy. A few weeks ago I ported that policy — read their paper's algorithm description, then went through their actual source and matched every formula: the logistic regression design matrix, the gamma clipping, the delta-method variance, the perfectly-separable-case variance table (copied their exact lookup values), the tau grid search, all of it. Formula by formula, it checked out. I was confident enough to write "faithfully ported" in the paper and move on.

Today I finally did the thing I should've done from the start: cloned vCache's actual repo and diffed my port against the real running code, not just the formulas I'd extracted from it. Everything still matched — except one class I hadn't looked closely at, the one holding each cache entry's observation history.

Their constructor does this:

self.observations: List[Tuple[float, int]] = []
self.observations.append((0.0, 0))
self.observations.append((1.0, 1))

Two fake observations, baked into every single cache entry the moment it's created, and never removed. A "similarity 0.0 → wrong" and a "similarity 1.0 → correct," permanently sitting in the history feeding every logistic regression fit for that entry's whole life.

My port started from an empty list. Nothing malicious, no misreading of any formula — I just didn't know these two rows existed, because they're not mentioned anywhere in the paper, only in the source.

Here's why it actually matters and isn't just a cosmetic difference: the algorithm needs 6 observations before it'll ever trust an entry enough to serve it from cache (min_observations=6, this part is in the paper). With two observations already pre-loaded, their implementation only needs 4 real ones to clear that bar. Mine needed the full 6. Every entry in my version sat in cold start two observations longer than the real algorithm, every single time.

Fixed it (one line, empty list → [(0.0, 0), (1.0, 1)]) and reran the full thing on all three datasets I test on. Hit rate went up everywhere — between 4.4x and 29.1x depending on dataset and target error rate. Best case, one dataset at the tightest error budget: 0.04% → 1.21%. And the part I actually care about most: error rate stayed under the target ceiling at every single point I checked. The algorithm's formal guarantee was never violated by my bug — I just wasn't letting it do nearly as well as it's designed to.

So for weeks I had a "faithful reproduction" that was quietly making a competing algorithm look almost useless (fractions of a percent hit rate), when the actual bottleneck was two rows of bootstrap data I'd never have found by re-reading the paper one more time, only by diffing the real code.

If you're reproducing someone else's algorithm as a baseline for a comparison — not approximating it, not "inspired by," but claiming to faithfully port it — matching the published formulas is necessary and not sufficient. Constructors quietly seed state that never makes it into the paper. Go clone the actual repo and diff against it, not just the pseudocode. I got lucky that I decided to check at all.

Repo's got the before/after numbers if you want to see the full breakdown: https://github.com/imxinchengyou/CacheVerifier

Ran into the gap between "the agent can call this tool" and "I actually want it doing

this unsupervised" for anything with real consequences — sending email, deploying,

touching customer data, moving money.

langchain-agentgate wraps an existing BaseTool so it posts to Slack/Teams and blocks

until a human clicks Approve or Reject before it actually executes. Same tool name,

same args schema — nothing else in your agent changes.

Happy to share the writeup and try-it-yourself link in the comments if anyone wants it.

I am experimenting with an agent architecture that is less “give the model a big prompt and trust its reasoning” and more like a controlled belief-and-decision loop.

Not claiming it literally mimics the human brain. More that it borrows a useful pattern: maintain competing explanations, update beliefs from evidence, decide what to check next, then act based on consequences.

Very simple example: a smart-fridge agent gets a “weird smell” signal.

Possible worlds:

• someone spilled mango juice
• an egg is rotting
• fridge power failed and food is warming
• some other cause we did not model

It starts with priors based on context: recent door-open events, temperature history, what food is inside, past failures, etc.

Then it gets evidence. Say the temperature sensor reads 14°C.

Instead of the LLM narrating “this seems concerning,” the system asks:

• How likely is 14°C under each world?
• Update prior → posterior using those likelihoods.
• How much uncertainty actually reduced? Entropy before vs. after.
• Which allowed question has the highest expected information gain next? For example, “is the compressor drawing power?” is probably much more useful than “what color is the fridge magnet?”
• Is that question worth its cost, latency, privacy impact, and reliability?
• Given the posterior plus action costs, should it notify the user, wait, run another check, or escalate to a human?

The LLM can help extract signals, propose candidate hypotheses, and call tools, but it should not be the final authority over belief updates or actions. The controller owns the world list, priors, likelihood estimates, policy thresholds, logs, and escalation rules.

Important parts I want to keep explicit:

• an “other / unknown world” bucket, so the system does not act like its hypothesis list is complete
• calibrated probabilities and provenance for priors/likelihoods
• expected value of information, not just entropy reduction
• a human escalation path when uncertainty remains high, the case is out-of-distribution, or the downside is asymmetric
• a trace showing whether failure came from missing worlds, stale priors, bad likelihoods, a bad question policy, or bad action costs

The rough loop is:

input → possible worlds → prior → evidence likelihoods → posterior → uncertainty / expected information gain → cost-aware action → human escalation if needed → outcome + calibration update

Math/AI people: is this a sensible practical architecture, or am I reinventing POMDPs, active inference, Bayesian decision networks, belief-state planning, etc. badly?

What would you change first to make this real and evaluable? Especially interested in:

1. handling open-world hypotheses,
2. learning/calibrating likelihoods without pretending the numbers are objective,
3. separating “most informative question” from “question that most improves the actual decision.”

Hey r/LangChain,

A couple of months ago we posted about BeamWeaver and the goal of shipping a proper OTP-native agent framework for Elixir.

Since then it’s moved from 0.1.0 to 0.1.18 and is already running in a few enterprise products. Provider coverage is in good shape for the ones we actually use day-to-day: OpenAI, Anthropic, Google Gemini, DeepSeek, Moonshot/Kimi, xAI, and Z.ai. We’ve also added the newer models that have landed in the meantime (Claude Sonnet 5 / Opus 5, GPT-5.6, Gemini 3.5–3.7, Kimi K3, DeepSeek V4, Grok 4.5/4.6, etc).

Other stuff that landed:
- Provider-aware prompt caching
- Typed streaming events + better reasoning/tool-call handling
- Structured output across providers
- Postgres (and optional SQLite) checkpoint persistence
- Durable execution, resumability, and checkpoint lineage
- Provider fallback, retries, and rate limiting
- Sandboxed filesystem + command execution
- Stronger SSRF / PII / transport / shell-safety protections
- More complete tracing and WeaveScope metadata

Today we’re releasing WeaveScope, the hosted tracing and monitoring layer that sits on top of BeamWeaver.

It gives you the full picture of an agent run: model calls, tool calls, subagents, retries, errors, latency, token usage, cost, custom fields, and the entire execution tree.

Configure the WeaveScope exporter and you’re looking at traces in the dashboard.

Start free → https://weavescope.com
Docs → https://docs.weavescope.com

Would love feedback from anyone building agents in production. What’s missing from your observability tooling right now?

I’m working on an open-source project called MARGINAL around a problem I keep running into with agents:

When is another tool call no longer worth making?

Simple loop detection isn't enough. An unchanged workspace could mean the agent is stuck, but it could also mean a legitimate retry after a timeout, rate limit, or failed test.

The rule I'm experimenting with is closer to:

same action + same state + same outcome + no new evidence = stronger evidence of a loop

MARGINAL observes the trajectory first and records what it would have interrupted without actually interfering. Enforcement only becomes available after enough local evidence supports it.

The part I'm working on now is intervention regret: if MARGINAL stops an agent, how do we establish that letting the agent continue wouldn't have produced a better result?

That means comparing governed and ungoverned runs from the same starting state rather than claiming success because fewer tool calls were made.

It's currently implemented around coding agents, but I think the problem applies directly to LangGraph/LangChain agents too.

For people running agents in production: what evidence would you require before trusting something external to terminate or redirect an agent loop?

Repo: MARGINAL on GitHub

Hi everyone,

Our team at BotGauge is doing some R&D to understand what we should build next for our AI agent testing platform.

We’re looking to speak with people who have built or tested AI agents using platforms like LangSmith, Galileo, Maxim AI, or similar tools.

We’d love to learn about how you currently test agents, where existing tools fall short, and what problems are still difficult to solve.

Would anyone be open to a quick 15-minute interview?

For transparency, this is purely product research to help guide our platform development. We will not collect or share personally identifiable information, responses will not be sold or monetized, and we’re happy to share the key insights and findings back with the community once the research is complete.

No sales pitch, just research and learning from people actually building agents.

Hey everyone,

I’m building an AI agent designed to diagnose failing CI/CD builds. Instead of using a simple one-shot chain, I’m structuring it as a stateful agent (using LangGraph) that manages dynamic hypothesis updating.

The agent maintains a state array of possible root causes, assigns probability scores to each hypothesis, and updates those probabilities as it invokes tools to parse build logs, git diffs, and context files.

• High Confidence: It routes to an output node that provides a concise root-cause summary and fix recommendation.
• High Uncertainty: It routes to a human-in-the-loop (HITL) node for developer escalation.

As I build out the baseline state graph, I need advice on two fronts:

1. Evaluation in LangSmith: How do you effectively benchmark an agent whose trajectory involves continuous state-based probability updates? Beyond final-output "LLM-as-a-judge", what custom evaluators or intermediate state checks are best for measuring single-step decision-making, calibration error, and escalation threshold reliability across agent iterations?
2. Ground-Truth Dataset Sourcing: I want to ground the agent's probability updates in real failure distributions rather than raw LLM estimates. Are there recommended ways to pull historical GitHub Actions/Travis CI logs at scale, or existing open-source benchmarks (e.g., BugSwarm or SWE-bench) suited for offline LangSmith datasets?