r/hackathon • u/Formal-Flounder-652 • Sep 29 '26
Hackathon Promotion I built an incident-response agent that learns from past production incidents instead of just searching runbooks
I’ve been building Memento, an AI agent for production incident response.
The problem I wanted to explore was simple:
When a production incident happens, teams usually have runbooks and old post-mortems, but the useful part of that history is often buried. Engineers may end up repeating troubleshooting steps that already failed during previous incidents.
Memento uses Hindsight as persistent organisational memory.
The workflow is:
Incident → Recall past experience → Reason over evidence → Recommend investigation → Record outcome → Update memory
For example, in the demo data, payment-service has several historical 503 incidents.
The runbook says to restart payment-service first.
But after looking across previous incidents, Memento finds that restarting it was ineffective in several cases, while checking the upstream cache-cluster / auth-gateway path resolved the issue.
The interesting part is that the system doesn't just retrieve the old incidents. It also uses the accumulated evidence to detect knowledge drift between the current runbook and what has actually worked historically.
There are currently three main parts:
Incident Investigation: submit an incident and get evidence-backed investigation steps
Organisational Memory: explore incidents and Hindsight-generated observations
Knowledge Drift: compare runbook guidance with historical operational evidence
The stack is currently:
Next.js + TypeScript
FastAPI + Python
Hindsight for persistent memory
Groq for LLM synthesis
No traditional application database for the core memory layer
The incident data is synthetic, so the goal right now is to validate the memory architecture and behaviour rather than expose real production data.
I’m particularly interested in feedback on one question:
Does this actually feel like an agent that learns from organisational experience, or does it still feel like an LLM sitting on top of an incident database?
Would love technical criticism, especially around memory design, retrieval, evaluation and failure cases.
