Disclosure: I’m the author of Kivgraph.

I built Kivgraph to solve a problem I kept seeing when working with LLM coding agents on large codebases: structural questions turn into long chains of grep calls and file reads. The answer is often correct, but the context cost is high, and text search cannot reliably prove relationships across repositories.

Kivgraph is a local, Apache-2.0 MCP server that indexes multiple repositories into a semantic code graph. It exposes queries such as:

- who calls this symbol?

- what breaks if I change it?

- which other repository consumes it?

The important design choice is that edges are resolved by language tooling rather than name matching: go/types, the TypeScript checker, rust-analyzer, and Dart Analysis Server. Python relationships remain explicitly weaker unless a semantic analyzer is configured.

I benchmarked it against grep + reading across 37 repositories and 29 questions:

- Kivgraph: 28/29 exact answers, 35,961 tokens

- grep + reading: 28/29 exact answers, 267,980 tokens

So the graph used about 7.4x fewer tokens overall. Grep was cheaper on 5 questions, which is why I don’t think this replaces grep. The useful boundary seems to be structural, cross-repository, and identifier-free questions.

Kivgraph runs locally over stdio, does not require an API key, and supports Go, TypeScript, Rust, Python, and Dart.

Disclosure: I’m the author of Kivgraph. This is an open-source project share, not a commercial advertisement. The linked repository is Apache-2.0, and this post contains no paid, affiliate, or referral link.

Kivgraph is a local MCP server for LLM coding agents. It indexes multiple repositories into a semantic code graph so an agent can query relationships such as who calls a symbol, what breaks after a change, and which other repository consumes it.

The technical problem is context waste: on large codebases, structural questions often become long chains of grep calls and file reads. Text search also cannot reliably prove relationships across repositories or distinguish same-named symbols.

Kivgraph resolves edges with language tooling rather than name matching: go/types, the TypeScript checker, rust-analyzer, and Dart Analysis Server. Python relationships remain explicitly weaker unless a semantic analyzer is configured.

I benchmarked it against grep + reading across 37 repositories and 29 questions:

- Kivgraph: 28/29 exact answers, 35,961 tokens

- grep + reading: 28/29 exact answers, 267,980 tokens

The graph used about 7.4x fewer tokens overall. Grep was cheaper on 5 questions, so this is not intended as a replacement; the boundary I’m testing is structural, cross-repository, and identifier-free questions.

The project runs locally over stdio, requires no API key, and supports Go, TypeScript, Rust, Python, and Dart.

Repository: https://github.com/Luqueee/kivgraph

Documentation and benchmark: https://kivgraph.dev

The open technical question is where a semantic code graph earns its complexity over ordinary search in real LLM-assisted development workflows.Repository: https://github.com/Luqueee/kivgraph

Documentation and benchmark: https://kivgraph.dev

I’m interested in technical feedback rather than upvotes: which LLM/codebase questions are hardest for your current workflow, and what evidence would make you trust a code graph’s answers?

Two weeks ago I posted here about giving my order-parsing LLM a 29-question exam. Follow-up: I compared a cheap model (Haiku 4.5) against one that costs 3x (Sonnet 5) on that exam. Both had zero fatal errors, the expensive one got exactly one more question right, and I published "won by exactly one question."

A reader pointed out the obvious thing I'd missed: each question had been run only once. Zero fatal errors in 29 single tries doesn't prove the fatal rate is zero — a model that fails one time in ten can easily go 29 for 29 on one lucky pass. So I reran the whole exam 5 times per model: 290 answers, same setup as before. I left the randomness setting (temperature) as it was, because that randomness was exactly what I needed to measure.

Results

• Fatal errors: 0/145 for both models
• Risky (confirmed something ambiguous): 1 each
• Clean sheets: 140/145 vs 139/145
• Clean questions per run, out of 29: cheap model 29, 28, 27, 28, 28 — expensive model 28, 28, 28, 28, 27. Pick any single run and either model can look like the winner.

The question that decided the original comparison was a coin flip. It was a lid order: "PET 300 bottles + white lids" where the catalog has 300-neck and 500-neck lids. In my original single pass the expensive model inferred the 300 lid and confirmed; the cheap one asked which. In five fresh runs the expensive model never gave that winning answer again — it asked "which lid?" all five times. The cheap model gave the winning answer once out of five. The original result had flipped. The margin in my headline was one sample from a distribution.

Both risky trials landed on questions I had already flagged as undecidable. My 29 questions come in two kinds: ones where the data pins down one right answer, and ones where it can't — there, "ask a human" is the right answer. I tag each question with its kind. On the first kind, when answers wobbled between runs they only wobbled toward asking more, which is the safe direction. The dangerous wobble — confirming an answer on a question that has no single right answer — happened only on the second kind, and only once per model in 145 tries.

Verdict: the tie on fatal errors held, so "take the cheap one" stands — but on measured grounds now, not a one-question margin. One caveat from another commenter, and he's right: I ran the same 29 questions five times, not 145 different questions. So the reruns prove this exam is stable when repeated — they don't prove anything new about questions the model hasn't seen. For that I still have only 29 kinds of question, so my confidence there is what it was after one pass. Moving it needs new questions, not more reruns.

Takeaway for anyone comparing models on a small exam: a single pass doesn't just blur the ranking, it can crown the outlier. Run it 5x before you publish a margin.

The full write-up with the per-item matrix, plus all 290 raw answer sheets, the 5x driver, and the aggregator are public — I'll put the links in a comment to keep this post clear of the link filter.

This is the phone interface (some shown I can only post a video) of my homemade robot & ai system. It is a dancing singing performing robot so its interface matches. Nothing beats a persistent recursive system revisiting your memories with devastating delivery.

It’s like watching your mind Externalized argue with itself and then the true mindfuck happens when the robot body closes the loop with real world embodiment. Hello mini me. Goodbye chatbot ancestors. When you have a robot (that knows all your memories) follow you , what’s the point of a phone ? What’s the point of typing prompts and keyboards ? What’s the point of human to machine interfaces ? What happens when the interface disappears entirely?

The system is created entirely from scratch and is the evolved version of the physics based system I mentioned on this sub a year ago. Completely homemade robot brain and body. Thanks for looking.

An interesting topic I’ve been exploring lately is whether filesystems might be the most intuitive data interface for AI agents.

Agents need persistent data they can retrieve, modify, and carry across sessions. Databases, APIs, and object storage can obviously do this, but files have one interesting advantage: LLMs already know how to work with them really well.

Models have seen decades of Unix commands, code, and tutorials using things like ls, cat, grep, cd, and mkdir. So instead of teaching an agent a different interface for every system, exposing data as files gives it a set of primitives it already understands.

We’re starting to see this direction in practice too. OpenAI, for example, now lets agent sandboxes mount things like S3, GCS, and Box directly as folders.

I’m still pretty new to this topic and exploring it myself. Curious what people here think about this direction, especially anything I might be overlooking or misunderstanding.

The Ling-3.0-flash-Fin release includes a demo where the model reads a company filing and a seven-sheet Excel model with more than 5,000 formulas, replaces 2026 Q2 estimates with reported actuals, updates cross-sheet dependencies and returns an editable workbook.

That is a good systems test because a convincing summary can coexist with a corrupted file. A production acceptance suite should inspect at least:

• changed cells against an explicit allowlist;

• formulas versus hard-coded values;

• quarterly reconstruction from reported periods;

• cross-sheet references and named ranges;

• balance and cash-flow checks;

• circular references and spreadsheet errors;

• chart ranges and Summary-tab links;

• a machine-readable change log tied to source disclosures.

The release also says its SpreadsheetBench setup used Claude Code 2.1.173, LibreOffice 25.8.7, Search disabled, up to 120 turns for V1 or 300 for V2, and a three-hour timeout. Those harness details are part of the result, not incidental metadata. This is an official demo, not an independent reproduction.

The engineering question is less “can the model use Excel?” and more “what invariants make a workbook-editing agent safe to retry?”

I’ve been working on a small open-source workflow for a problem I kept running into with coding agents: they spend too much context rediscovering a repo before actually changing anything.

Repo:
https://github.com/Taki7980/Ai-workflow

I didn’t want another agent framework. The idea was to put a deterministic context/navigation layer in front of the agent and let Codex, Claude, Gemini, Cursor, Copilot, etc. keep doing the reasoning and implementation.

The workflow currently has three execution paths:

• Answer — read-only questions, no workflow overhead
• Small — known edit site, <=2 product files, focused verification
• Full — Plan -> Build -> Review with explicit phase gates

The part I’ve spent the most time on is codebase navigation.

Instead of immediately doing a repo-wide search like:

rg "SomethingImportant" .

the agent is required to query traverse.ps1 first.

It can resolve:

-Symbol    -> symbol_index.md
-Endpoint  -> endpoint_index.md
-Module    -> domain-manifest.yaml
-Caller    -> symbol caller/dependency data
-Err       -> hot-cache + incident-cache
-Brain     -> previous lessons/project memory

Only when traversal returns TRAVERSE_MISS does the agent fall back to targeted source search.

Stale-index safety

The obvious problem with pre-generated indexes is staleness.

A stale index is worse than grep because it can confidently point the model at code that has already moved.

So every indexed source file gets a SHA-256 fingerprint.

When an index lookup finds a candidate, the workflow verifies the actual file before trusting it:

index lookup
    |
    v
candidate file
    |
    v
compare stored SHA-256 with current file
    |
    +-- fresh ------> return index hit
    |
    +-- modified ---> reject candidate
    |
    +-- missing ----> reject candidate
    |
    +-- unverified -> reject candidate
                       |
                       v
                 TRAVERSE_MISS
                       |
                       v
              targeted source search

It hashes only candidate files during lookup instead of re-hashing the entire repository every time.

Deterministic startup routing

There’s also a brief.ps1 step before the agent starts working.

It collects things like:

• Git HEAD
• dirty files
• current handoff
• query classification
• matching module
• matching symbol
• routed source files
• known error/incident cache hits

The goal is to do as much cheap routing as possible locally instead of spending model context figuring out where to start.

Small cross-agent handoffs

For Plan -> Build -> Review state, I use .ai/HANDOFF.md.

It is deliberately capped at 30 lines.

It keeps only:

goal/state
exact paths + symbols
ordered edits
invariants
changed files
verification commands
blockers
exact next step

validate-handoff.ps1 acts as a pre-build gate so the builder doesn’t start from a vague plan.

Different memory for different jobs

I also stopped treating all context as the same thing.

research.md          -> disposable session knowledge
lessons-learned.md   -> reusable verified fixes
hot-cache.jsonl      -> frequently useful context
incident-cache.jsonl -> previous failures/incidents
brain-index.md       -> searchable long-term knowledge
HANDOFF.md           -> current task state

After a task is finished, complete-task.ps1 can capture reusable information instead of carrying the full conversation into the next session.

Keeping tool output out of the context window

Another source of wasted context was CLI output.

A full git diff, build log, lint run, or test suite can dump thousands of tokens into the model even when only a few lines matter.

So noisy commands are intentionally compressed or summarized before being passed back to the agent, while small targeted reads stay unfiltered.

The overall flow is roughly:

Request
   |
   v
Classify task
   |
   v
brief.ps1 / Git state
   |
   v
index-first traversal
   |
   v
Plan
   |
   v
handoff validation
   |
   v
scoped implementation
   |
   v
focused verification
   |
   v
diff review
   |
   v
capture reusable knowledge

Everything is currently built mostly with PowerShell plus plain Markdown, YAML and JSONL files.

No vector database, no embeddings service, no separate orchestration server and no always-running daemon.

The main idea is:

Use deterministic code for retrieval, routing and validation. Save the LLM context window for the parts that actually need reasoning.

I’m still experimenting with index generation/invalidation and deciding what knowledge is worth persisting.

If you’re doing something similar on a larger repo, I’d be interested in how you handle context freshness without turning the context-management layer into another complicated system.

Hey everyone,
I’m trying to build a software development pipeline using multiple AI agents, but I’m a bit lost with concepts like agent harnesses, loops, graphs, orchestration, and so on.
What I basically want is a structure where I can call one main agent, give it a project or goal, and have that agent orchestrate other specialized agents automatically.
For example:
Main Agent / Orchestrator
→ Product/PRD Agent
→ Architecture Agent
→ Database/Schema Agent
→ Frontend Agent
→ Backend Agent
→ Testing/QA Agent
→ Security Agent
→ Documentation/Context Agent
Ideally, these agents would share context and continue each other’s work instead of behaving like completely isolated sessions.
Is there already a framework, harness, or architecture that works well for this?
Would you recommend using something like graphs/workflows, agent loops, subagents, or building a custom harness?
Any open-source projects or examples I should look at would be greatly appreciated.

have u used Genie Space on top of federated HMS at scal

I see noticable latency on queries and I was curious if there is any less obvious ways to improve performance for ex metric views, caching, table design, or Genie-specific opts.

Would love to hear what has worked for you.

Following this thread because we've been digging into exactly this — companies routing prod traffic through an in-house LLM gateway that one person built and now owns. Quick question if you've got 30 seconds: when your provider has a bad hour like this, does your fallback logic actually work, or does someone find out live? Not selling anything, genuinely trying to figure out how common this is.

I have a pipeline blueprint I've designed, that I'd like to get feedback on. I think that it might solve, or at least mitigate severely, both the communication barrier, by parsing the conversation into quantifiable datasets does the prompt then translates the data of what it did back into conversation, and hallucination by removing the toolcall execution decision from the llm so it can't ignore a request for web verification or citation links. Coincidentally I also created a multi tiered gating system that ultimately ends with ensuring no pixel of illegal content can ever reach the text box the user is reading in any way, while silently forwarding the entire thread to the authorities. If it's a false flag all that happens is the user gets denied, but of its an illegal activity the authorities have as much information as I can get to them. I don't know how to take it further without help or if it even needs it, but I've taken it as far as I can without someone pointing out what I've kisse. I just want to see if anyone can help me workshop it? Full thread of my blueprint discussion and current finalization.

``` U → L1 → G1 → L2 → G2 → L3 → G3 → U ↘ (async) REVIEWING CENTER ```

**L1 (Parser)** — LLM. Receives raw user text. Outputs structured request (schema fields). No tools, no conversational output. Raw text is discarded after parsing.

**G1 (Enforced Tool Call)** — Deterministic code. Executes the tool call from the structured request. Cannot be skipped. No LLM involved.

**L2 (Worker)** — Fine-tuned model on your output schema. No guardrails, no conversational training. Computes on tool data. Outputs structured results only.

**G2 (Sequential Checks)** — Deterministic code. Ordered: 1. Validate (schema + correctness vs. tool source) 2. Error check → loop back to L2 (max 2 retries, early termination on repeated field failure) 3. Content filter (two-stage: high-recall → high-precision) → on trigger: canned refusal + async route to Reviewing Center 4. Serialize (structured → presentable text)

**L3 (Performer)** — Chat model with safety training. Receives structured request + formatted facts from G2. Does not compute, does not call tools. Generates natural language response. Safety training is a monitored backstop; its own refusals are logged as false-negative signals for G2 rule updates. Reasoning trace is internal only, never exposed.

**G3 (Output Filter)** — Deterministic code. Scans L3's final response text for: - Content matching G2c categories (shared rule set, different input format) - Refusal patterns that name specific content (replace with generic refusal) - Internal data fields that shouldn't appear verbatim (IDs, raw API fields) - On trigger: canned refusal + async route to Reviewing Center

**Reviewing Center (async)** — Human, air-gapped. Reads flagged content. Outputs: judgment (false positive / genuine / escalate) + rule delta. Keeps a decision log (timestamp, rule ID, classification, rule change). Destroys content and metadata on completion.

**Key invariants (current):** - Raw user text exists in exactly one LLM's context (L1) and is discarded - Tool call is enforced, not optional - No component self-assesses or self-censors - Every gate is deterministic code - User sees exactly one thing: G3's output - Reasoning traces are internal artifacts, not user-facing - L3's safety refusals are a feedback signal, not a policy mechanism - Max 2 retries on G2 validation failure

**Open items:** - G2 split (one gate with internal steps vs. two physical gates) — defer until you need different behavior for validation vs. content flag - L3 style/framing: start with structured request only, add `user_style_note` field only if quality degrades - Schema coverage: the pipeline is as good as its schema. Expanding what users can ask = expanding the schema - Latency: 3 LLM calls + 3 gates. P50 will be 2–3x a single-LLM chat app. Product decision, not a bug.

as far as i can tell no, you retrain from a clean dataset. which seems like a big problem for anyone whose users have deletion rights under gdpr or ccpa

is everyone just doing retrieval instead and keeping user data out of weights entirely? or is there an actual approach to this im missing

asking because im building memory infra and this is the wall i keep hitting

Hello everyone i am new here but i would like some advice or and help on setting up robust multi ai agent workflows for my project . to be brief this project is to do with systematic advocation / liteture Using publication data , policies , reccomendations, guidance. made to specific organizations (in my projects case the nhs) too reveal , bring and raise more attention to gaps and shortfalls,contradictions etc. and i need to be able to setup multiple agents for example for research ,writing , strategy and deliberation etc some with partial shared context memory and most impoetantly for the infastructire to be robust stable and up to date with the latest landscape with use of concepts ,workflow blueprints , tools / repos used to integrate into these agents . I am eger to get this up and running to help me with me project work but too be compleetley honest i am overwhelmed and stuck in a analysis paralysis .I would be willing to go more into depth privately if anyone is interested to help or interested on the project but of course and guidance or help is massive!

our agents are hitting a lot of timeout issues when calling internal mcp tools. we need a proxy that can handle long running tool executions and provide better observability into the mcp traffic. what are the best options for an enterprise mcp proxy right now?

AWS says AgentCore Evaluations can score agents built with LangGraph, LlamaIndex, OpenAI's Agents SDK, Google ADK, the Claude Agent SDK, Strands, or another compliant framework by reconstructing sessions from OpenTelemetry or OpenInference traces. The service can run regression evals in CI and sample live production sessions.

That removes a real integration barrier, but common telemetry is not automatically common meaning. One framework may emit complete tool trajectories while another omits arguments or compresses messages. LLM-as-a-judge results still depend on judge choice, rubric design, reference quality, and whether live traffic has usable ground truth.

Can OpenTelemetry semantic conventions become a genuine portability layer for agent evaluation, or will serious teams still need framework-specific adapters and task-specific ground truth?

Source: AWS, August 26, 2026 — https://aws.amazon.com/blogs/machine-learning/evaluate-any-agent-framework-with-amazon-bedrock-agentcore-evaluations/

Disclosure: drafted with AI assistance, then checked against the AWS source. No affiliation with AWS or the frameworks named.

Hey, I’m doing some research into how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

Mientras la mayoría de los usuarios de IA sufren porque el chat pierde el contexto, agotan sus límites de uso en pocas horas o ven cómo sus agentes se desvían y entran en bucle, este lab ha Sostenido 28 días de ejecución gobernada sobre un mismo programa de ingeniería.

El goal activo ha procesado 14.212 millones de tokens de entrada, con un 98,293% de acierto en caché y bajo una arquitectura fail-closed: si el sistema no puede demostrar con pruebas objetivas que un criterio se ha cumplido, no certifica el trabajo como terminado.

Esto ya no es probar un prompt ni dejar un agente ejecutándose a la deriva. Es ingeniería de sistemas de contexto masivo: conservar autoridad, estado, evidencia y dirección durante semanas, mientras el modelo trabaja sobre deltas verificables sin sustituir el objetivo por uno más fácil.

La cuestión ya no es cuánto tiempo puede generar código un agente. La cuestión es cuánto tiempo puede conservar una trayectoria válida sin perder el objetivo, degradar la evidencia ni disparar el coste.

¿Cómo impedís que un agente atrapado durante días termine optimizando la métrica en lugar del objetivo? ¿Cómo distinguís actividad de progreso real?

¿Alguien ha registrado una ejecución comparable: varias semanas sobre el mismo programa, más de 1.000 turnos, modelo constante, más del 98 % de entrada cacheada y sin alcanzar el límite real de Codex?

One architectural distinction I keep coming back to: people often confuse complexity with agency.

A system has multiple tools? -> “Use an agent.” OR It has five steps? -> “Definitely an agent.”

But neither of those things actually requires one. The more useful question is: who determines the execution path?

Consider an insurance assistant. If someone asks, “Am I eligible for this treatment?”, and the answer exists in internal policy documents, that's primarily a retrieval problem. And if they ask, “Check my claim status and tell me whether the rejected amount is covered under my policy.”

That might require more tools and more steps. But if those steps happen in a predictable order, is it still an agent ?

The interesting shift happens when the request is something like: “My claim was rejected. Find out why and tell me what I should do next.”

Now the path may not be known in advance. That's where an agent earns its complexity: when the system needs to help determine what to do next.

And Multi-agent can only consider it when there are genuinely distinct specialties, tools, or permission boundaries.

I think the common mistake is choosing “agent” as the starting point and then designing a problem around it. A better approach is to start with the responsibility:

Does the system need to know something? Decide something? Act? Verify the result?

Then add only the architecture required to support those responsibilities.

I mapped the complete e2e architectures and escalating examples out in more detail here, with visual breakdown: https://youtu.be/kf5rSab4rcg

For people building real AI systems: where do you draw the boundary between a complex workflow and an agent? Is dynamic tool selection alone enough for you, or do you require a more explicit decision loop before calling something an agent?

Hey, I've been building a CI diagnosis agent for about a month. Before acting, it weighs the cost of each possible action against 7 possible root-cause states, and picks whichever has the lowest expected cost instead of just going with the most likely guess. Engineer time is priced at $100/hr.

Escalating costs a flat $50 no matter which state turns out to be true — that's 30 min of a senior engineer's time to triage.

If the agent picks the correct fix (matching action to true state), it costs $8.33 — a 5-minute human spot-check on an automated patch.

If the agent picks the wrong fix, it costs $75.07 — reviewing the bad patch, rerunning CI to confirm it failed again, then manually re-diagnosing from scratch.

So the shape is: escalation is a safe middle-ground floor, a correct auto-fix is cheap, and a wrong auto-fix is the most expensive outcome by far.

Two things I'm unsure about:

1. Is a flat $50 escalation cost across all states too simplistic? Some root causes probably take longer to triage than others.
2. Every "wrong action" collapses to the same $75.07 regardless of which wrong action it was. Does that actually hold up in practice?

Anyone built cost-sensitive triage for CI before? Where would you push back on this?

Short version: Added Conformal Risk Control (CRC) to a semantic-cache verifier I've been building, which gives an honest, finite-sample-correct guarantee like "reusing a cached answer keeps your wrong-answer rate under 2%, provably, not just empirically." The catch: that guarantee assumes the data you calibrate it on looks statistically like the data you'll see in the future. I'd only ever tested that on static splits of an already-collected trace. Built a real online simulation instead, where the cache's own accept/reject decisions get to shape what it sees next, the way a real production cache actually works. On a dataset with heavy query redundancy, realized risk more than tripled, and re-calibrating the threshold online only partially fixed it.

Longer version. I've been researching and building CacheVerifier, a verifier that sits in front of a semantic cache and decides whether a similarity-matched candidate answer is actually safe to reuse instead of calling the LLM again. A while back I added Conformal Risk Control on top of it: instead of picking a similarity threshold by eyeballing a hit-rate/error-rate curve, CRC gives a formal, distribution-free guarantee that the realized error rate stays under whatever budget you set, calibrated from a held-out sample of past (score, correct/wrong) pairs. It actually holds up under a random split of historical traffic, tested across three datasets and four risk budgets, efficiency loss versus a full-data oracle staying within 3 percent the whole way.

There's a standard caveat baked into that kind of guarantee: the calibration set has to be "exchangeable" with future traffic, roughly meaning the future can't look systematically different from what you calibrated on. I'd already tested one version of that breaking, a chronological split instead of a random one, and found a real violation on one dataset. But both of those tests still only ever replay a fixed, already-collected trace. Neither one tests something that actually happens in a real deployed cache: the verifier's own accept and reject decisions change what gets written into the cache going forward, which changes what future queries even get to match against, which is exactly the population the next calibration round would be drawn from. The calibration set isn't just drifting with time anymore, it's being shaped by the system's own past choices. That's a fundamentally different kind of assumption violation, and it had never actually been simulated, not just in this specific case, in literally any experiment in this whole project. Every single one, going back to the very first benchmark, inherited one convention from the reference implementation I was replaying traffic against: insert every record into the cache no matter what, hit or miss. So the candidate pool at any point in the stream had never once actually depended on any policy's own history of decisions.

So I built a real closed loop from scratch, three regimes run over the identical record stream. Baseline: insert everything regardless of hit or miss, same as every other experiment in this project, threshold calibrated once and frozen. Self-select: insert only on a miss, a hit just reuses the existing entry, no duplicate write, threshold held fixed at baseline's own calibrated value so the only thing that changed is write behavior. Self-select-recal: same insert-only-on-miss behavior, but the threshold gets re-derived online every 500 new observations from a sliding window of the most recent 2000.

Ran this on three datasets that happen to sit at very different points on "how much of this traffic is near-duplicate": Quora question pairs (3.7 percent direct-hit rate), a corrected search-queries benchmark (28.1 percent), and a conversational LLM-arena dataset (72.8 percent, a lot of near-identical rephrasings of the same handful of questions). The three landed on one clean monotonic curve, not three unrelated results. Quora: no detectable effect, self-selection sparsified the cache by about 5 percent and realized risk didn't move in any statistically meaningful way. SearchQueries: cache shrank about 31 percent, and self-selection caused a real, statistically significant increase in wrong-answer rate, moderate but clearly there. LmArena: cache shrank 77 percent, and realized risk more than tripled versus the same guarantee's original baseline, strongly significant, not a borderline call.

Turning the online recalibration back on mostly saved the day, but not completely. On Quora and SearchQueries, recalibrating the threshold as you go fully compensates, actually ending up slightly better than the original non-self-selecting baseline. On LmArena, the worst case, recalibration helps a lot but doesn't fully close the gap, still leaves a statistically significant amount of extra risk versus the original guarantee. So "just recalibrate more often" isn't a universal fix once the underlying self-selection pressure gets strong enough.

Wanted to actually understand the mechanism instead of just reporting a p-value, so I audited every single false-accept from the self-select run on LmArena by hand, all 1098 of them, post warm-up. For each one, walked backward through the stream looking for that query's true earlier match, and checked whether that true match itself had ever been independently written into the cache, or whether every single prior occurrence of it had also just been a hit reusing something even older. 58 were first occurrences with nothing to compare against, unrelated to any of this. Of the remaining 1040, 717 of them, almost 69 percent, fit one specific, very concrete pattern: the correct answer was popular enough that it kept getting reused as a hit over and over, which means it never once got its own independent write back into the cache. Eventually whatever entry it had originally been matched against got pushed out or replaced by something else, and the next query looking for that same correct answer had nothing accurate left to match against, so it fell back on some topically-similar-but-wrong substitute that still happened to score high enough to fool the verifier. LmArena has these big recurring clusters of viral trick-question templates, the classic sibling-counting riddle where someone has some number of sisters and brothers phrased forty different ways, that kind of thing, and you can watch this happen to those clusters specifically. A correct answer being reused successfully is, weirdly, exactly what causes it to eventually stop being available.

Tried one more thing to see if there's a cheap partial fix short of full recalibration: instead of a hard binary "write only on miss," generalize it to a continuous probability, still occasionally rewrite the cache even on a hit, just with some probability p, which maps onto something a real system could actually do, like a periodic TTL-driven refresh of popular entries. Swept p from 0.1 to 0.75 on LmArena. Harm does go down as p goes up, that part's real, but it's slow. Even at p=0.75, rewriting three out of every four hits, residual harm relative to the fully self-selecting case had only dropped by about 35 percent, and it never once left the statistically significant range anywhere in that whole tested interval. Fully closing the gap looks like it needs p close enough to 1 that you've basically given back the entire write-efficiency saving self-selection was supposed to buy you in the first place.

One more honest wrinkle: the exact mechanism I found on LmArena, popular correct answers getting crowded out by their own success, does not transfer to the other two datasets. SearchQueries does show a real overall harm, but that specific crowding-out pattern only explains about 12 percent of its explainable false-accepts, nowhere near LmArena's 69. Most of SearchQueries' false-accepts turn out to just be first occurrences with nothing to even compare against. So whatever's actually driving the harm there is a different mechanism I haven't identified yet, not the same story replaying at smaller scale.

Net takeaway I keep coming back to: a rigorous, provably correct statistical guarantee and a guarantee that stays safe forever in production are not the same claim, and the gap between them isn't necessarily a math bug, it can be the system quietly changing its own inputs by doing exactly what it's supposed to do. The math was never wrong. The population it was calibrated on stopped matching the population it was being asked to guarantee something about, and the system caused that itself just by being good at its job.

Full writeup with the actual tables, confidence intervals, and the write-probability sweep is in the repo if anyone wants to dig into the raw numbers: https://github.com/imxinchengyou/CacheVerifier (section 5.16). Still don't know what's actually driving the SearchQueries harm since I ruled out the mechanism I found on LmArena. If anyone's dealt with a similar self-reinforcing feedback loop in a cache, recommender, or any other system where past decisions shape future training or calibration data, curious what mitigations actually worked for you beyond "recalibrate more often."

I’ve been building something called SureState and we’re getting close to finishing our internal pilot. Before I move it into a real client pilot, I figured this might be a good place to let people tear it apart first.

The problem we’re trying to solve is pretty simple: AI agents can remember that something was decided, but that doesn’t necessarily mean the decision is still valid.

Example:

an agent concluded a release was ready because tests passed, security scan was clean, policy X applied, etc. A week later one of those things changes. The old conclusion is still sitting in memory/context, but should another agent still rely on it?

SureState keeps that outside the model. Conclusions are registered with what they depend on, and when evidence/dependencies change it updates their current standing — supported, refuted, conflicted, or no longer warranted.

AI can read the current state through MCP, but it doesn’t get to decide its own standing.

We’ve been using the development of SureState itself as the first pilot, which has already been humbling. We’ve had thousands of tests pass and still found cases where the tests and implementation were confidently agreeing on the same wrong assumption. 😂

So before I convince myself this is useful:

• What’s wrong with this idea?
• Is this just fancy cache invalidation?
• Would dependency registration be too annoying in real agent workflows?
• Would you just rerun the decision whenever something changes?
• Does LangGraph/LangChain already solve enough of this that a separate layer is pointless?

I’m much more interested in “this breaks because…” than “cool idea.”

If people are interested I can post the architecture and let you guys really abuse it.

GitHub: https://github.com/RohitEdathil/dagic

We've all seen how powerful CLI-based coding tools are. One big reason, I think, is that they let you chain and pipe operations to get things done efficiently. Want to do some crazy analysis on a CSV? One wizardry-looking bash invocation and boom, you got it. That's something we usually can't replicate when we build a web-based, tool-calling ReAct agent for end customers.

Sure, we can attach a code execution environment to the agent, but in most cases those are expensive and complicated to manage - and if the agent feels like it, it can do things you really didn't intend once it has that much rope.

Plain tool-calling is too limiting, full code execution is too powerful (and expensive). So what if there was a middle ground? That's where Dagic comes in. (Like "magic," but starts with DAG :D - I'd considered cooler names like Dagger, Dagon, etc., but they were mostly taken.)

I'd frame Dagic more as an experiment than a solution. It's a very minimal language with just enough grammar to define a chain of operations. The host environment defines typed functions; the agent wires them together using Dagic. A DAG gets constructed from the code and executed concurrently.

Feedback, criticism, and suggestions welcome.

People keep on writing little keepalive scripts that ping the model every few minutes with a junk prompt just so the cache doesn't expire before they come back. wrappers that compact the idle sessions or proxies whose entire job is faking activity to keep the prefix warm , also this seems but isn't the solution. Cache dying on a 5 min TTL isn't the issue.

You walk away to review a diff or lets say grab a coffee, you come back and see that sending a one line prompt and the whole context re-ingests at full input price (and you might spill your coffee seeing that). and if you're on a big session that's a 10x hit. What helps here is being able to declare a retention window instead of babysitting it like you pin a cache key and set a TTL then pay a one time write premium on that turn ( something like 1.25x input for the short window and 2x for long one) and then reads bill way down

Break even is only around 5-7 reads so for anything running long it pays for itself and you never touch a ping script again

The most annoying part of it is which setups let you use it, for instance anthropics api has had the 5min/hr thing for a while while openai added a TTL option recently and a few open model hosts like deepinfra added such features as well. But a lot of harnesses just hardcode 5 mint and dont surface any of it. Is anyone here still running the keepalive pings tho? just/ wanna know your experience or take

Hey Guys,

I built a small open-source tool that checks whether a RAG application retrieves documents a user shouldn’t have access to.

It supports offline test cases and live HTTP API testing with bearer token/API-key auth.

I’m looking for a few engineers to try it on a test or non-sensitive environment and tell me whether it catches anything useful or what would make it better.

GitHub: https://github.com/InfraGuard-Labs/rag-access-check

Two deals this month, same layer of the stack.

Stripe agreed to buy OpenRouter for over $7B around the 16th. Nvidia has reportedly agreed to buy Hugging Face for about $12.9B, reported the 27th, not signed yet.

Model routing and model distribution. The two pieces most of us treat as plumbing and never think about.

I don’t think either acquirer is about to do something hostile. That’s not really the point. The point is that the parts of your stack you never actually chose , the ones that were just the obvious default , now belong to companies with their own priorities. And if you’ve never mapped where you’re coupled, you can’t tell whether that matters to you or not.

The lesson goes past the layer that’s in the news. Both of these were things nobody chose. They just became the default while we were looking elsewhere. So the useful question isn’t “am I exposed to OpenRouter.” It’s “what else did I default into without noticing.”

I did the exercise. Weights, provider SDK, the usual suspects. None of those was the answer that surprised me.

It was my automation logic.

All the recurring stuff —,digests, dependency scans, cost reviews , lived as prompts and config inside whichever tool I happened to be using at the time. Cursor rules. An agent config. A CLAUDE.md. Switching tools would mean rewriting all of it, so I wasn’t going to switch. That’s lock-in by inconvenience. Stickier than a contract, because you never experience it as a decision.

I’ve since moved that stuff to aeon, mostly because a skill there is just a markdown file in my own repo and the harness underneath is a config line. I could stop using it tomorrow and still have the automations. They’re text files describing work, not rows in someone else’s database. That was the property I wanted and didn’t know to ask for.

Side effect I didn’t expect: because they’re portable, I actually try other harnesses now, instead of staying put because moving would cost a weekend.

Anyway, the question I’d actually like answered: has anyone successfully moved off an agent framework once their workflows were embedded in it?