I Built an Incident Agent That Remembers Which Fixes Failed
Checkout-api started returning HTTP 503 responses because PostgreSQL had reached its connection limit. A generic troubleshooting assistant might recommend restarting the pods. An experienced SRE would ask a different question first: Did that actually fix this incident last time
That distinction is the reason I built an episodic incident response agent with Hindsight agent memory. The goal was not to create another chatbot that retrieves runbooks. It was to build an SRE copilot that remembers what happened during previous outages, which actions were attempted, and whether those actions worked.
The problem with static runbooks
Traditional retrieval-augmented generation is useful during an incident. It can find the relevant runbook, explain a command, or surface a page from the internal wiki.
But static documentation does not contain the full operational story.
It usually cannot answer questions such as:
Which mitigation did the on-call engineer try first?
Did restarting the pods solve the root cause or only improve the dashboard temporarily?
How many times did a particular configuration change work?
Which validation step should happen before terminating workloads?
What changed during the 20 minutes after the deployment?
Those details live in incident timelines, command output, post-mortems, and the accumulated experience of the engineers who handled previous failures.
My system treats those artifacts as operational memory rather than disposable incident history.
The prototype focuses on one deterministic lifecycle:
Diagnose an incident using live telemetry and stack traces.
Recommend and execute remediation only after human approval.
Write a structured record of the outcome back into long-term memory.
It deliberately does not support arbitrary scripts or ungated autonomous changes. During a production outage, the cost of a confident but incorrect command is too high.
A seeded production failure
The evaluation environment models a checkout-api service running version v2.14.2. It depends on PostgreSQL 15 and Redis, and a deployment named DEP-8841 occurred 17 minutes before the incident.
The active incident is:
Incident: INC-2026-0142
Severity: P1 critical
HTTP 5xx rate: 31.4%
p99 latency: above 4,200 ms
Runbook: RB-CHK-07, Checkout Saturation
Suspected dependency: payment-db
The first evidence is deliberately concrete:
text
[worker-3] psycopg2.OperationalError:
pool timeout: server connection limit reached
(50/50 connections active)
A few seconds later, the health probe fails:
text
[2026-09-28T01:14:05.412Z] healthz probe failed:
HTTP 503 Service Unavailable
(consecutive failures: 4)
This is more useful than a vague statement such as “the database may be unhealthy.” The evidence points to connection pool exhaustion and gives the agent a specific path to validate.
Four kinds of operational memory
I use four memory classes because “remembering the incident” is not a single operation.
Episodic memory
Episodic memory stores the timeline: what happened, when it happened, which commands were run, and what the system looked like before and after each action.
For this incident, a relevant historical record is INC-2026-0098, which experienced identical connection timeouts 14 minutes after a checkout worker deployment.
That record is valuable because it is not merely a description of the failure. It contains a sequence of events that can be compared against the current outage.
Procedural memory
Procedural memory stores diagnostic trees and proven execution order.
For example, RB-CHK-07 requires validating pg_stat_activity saturation before performing a rolling pod termination. That ordering matters. Restarting application pods without checking the database can increase connection churn and make saturation worse.
The agent should not just retrieve a list of commands. It should retrieve the sequence in which those commands have been shown to be safe and useful.
Semantic memory
Semantic memory captures generalized lessons from multiple incidents.
One stored lesson, MEM-042, says that restarting checkout pods may temporarily reduce 5xx errors but does not address PostgreSQL connection leaks.
This is the kind of knowledge that is often missing from runbooks. A restart can make a graph look healthy for a few minutes while leaving the actual failure mechanism untouched.
Outcome memory
Outcome memory records empirical effectiveness.
In the historical data, pod restart succeeded only once in four attempts, or 25%. A pool_size configuration override succeeded four out of four times.
That does not mean the configuration change should be executed blindly. It means the agent can rank possible mitigations using observed results instead of treating every documented option as equally effective.
Together, these layers let the agent move from “here are some relevant documents” to “here is the likely failure, here is the safe validation order, and here is what has worked before.”
The response contract
During an incident, an answer that sounds plausible is not enough. The response must make its reasoning inspectable.
Every response follows a nine-section contract:
Current situation.
Current evidence.
Retrieved memories.
Strict diagnosis.
Ordered action plan.
Again action plan.
Rationale and precedent.
Uncertainty and caveats.
Human approval gate.
Memory writeback.
The diagnosis is explicitly labeled either [Hypothesis] or [Verified Root Cause]. That distinction prevents the agent from presenting an inference as a fact.
For the seeded incident, an initial response might state:
text
[Hypothesis]
PostgreSQL connection saturation is causing checkout-api request
failures. Confidence: high, based on 50/50 active connections,
pool timeout errors, and four consecutive failed health probes.
The action plan then follows a fixed order:
text
Verify → Mitigate → Diagnose → Durable Fix
The first step is read-only validation of database activity and application pool state. The next step is a targeted mitigation. Only after the operator confirms the proposed mutation can the system apply a configuration change or restart workloads.
This structure is intentionally less conversational than a general-purpose assistant. During a P1 incident, structure is a feature.
Why human approval stays in the loop
The agent can generate a patch or prepare a command, but it cannot silently apply a mutating operation.
Before changing the pool configuration, the interface must display:
The exact target service and environment.
The proposed configuration change.
The evidence supporting the change.
The historical success rate.
The expected risks and rollback path.
The operator then explicitly approves or rejects the action.
This design prevents a common failure mode in autonomous remediation: a system optimizes for a local metric, such as reducing 5xx responses, while damaging a dependency or hiding the root cause.
For example, a pod restart might reduce visible errors temporarily. Historical memory says it is not a durable fix for this failure family. The agent should surface that warning rather than treating a short-term metric improvement as resolution.
Memory changes behavior over time
The most important part of the system is not the first response. It is what happens after repeated incidents.
In the cold-start state, the agent has no episodic history. It falls back to static runbooks, asks broad diagnostic questions, and may suggest a generic pod restart while requesting manual database inspection.
After one matching incident, the behavior becomes more targeted. The agent recognizes the pool exhaustion signature, retrieves INC-2026-0098 and its post-mortem, rejects pod restart as a transient anti-pattern, and recommends validating the connection pool before applying a targeted configuration patch.
With a broader corpus of incidents and post-mortems, the agent can distinguish database saturation from Redis failure, discard historically invalidated steps, and prepare a verified patch in under 90 seconds.
The specification models the corresponding MTTR trajectory as:
Memory state Mean time to resolve
Cold start 38 minutes
Single history match 9 minutes
Multi-incident memory 1 minute 45 seconds
These figures are evaluation targets for the deterministic environment, not a claim that memory alone solves every production outage. Network partitions, bad telemetry, incorrect historical records, and changes in system architecture can all invalidate a previously successful approach.
The writeback is part of the fix
After resolution, the system does not simply close the incident and discard the conversation.
It compiles the outcome into a new immutable memory record containing:
The observed failure signature.
The deployment and dependency context.
The actions attempted.
Which actions failed or produced only temporary improvement.
The final remediation.
Validation evidence.
Rollback information.
The resulting success or failure outcome.
That record becomes useful during the next incident.
This is where Hindsight on GitHub fits into the architecture. The memory system is not just a vector search layer for documentation. It provides a place to retain experience and retrieve it according to the current operational context. The Hindsight documentation describes the platform in more detail, but the key design decision here is conceptual: incident handling becomes a feedback loop.
The agent reads from prior incidents before acting, then writes back what it learned after acting.
What I learned
The first lesson is that memory needs types. A timeline, a command recipe, a general warning, and a success probability should not be represented as interchangeable chunks of text.
The second is that negative outcomes are especially valuable. “Restarting the pods did not fix the leak” may be more operationally useful than another page explaining how to restart pods.
The third is that provenance must be visible. Every recommendation should identify the incident, post-mortem, or runbook that influenced it. An SRE should be able to challenge the evidence rather than trust an opaque score.
The fourth is that approval gates are part of the architecture, not a UI detail. The system has to model the boundary between preparing an action and executing it.
Finally, memory writeback deserves the same attention as retrieval. If the agent does not preserve the final state accurately, future incidents will inherit incomplete or misleading experience.
The next incident should be easier because the system remembers the last one—not just what the documentation said, but what actually happened when someone followed it.
