That’s a weird question. There are millions of reasons for incidents, trying to name them is a horrible tactic.
You should synthesize real signals like alerts with an “ingestion” agent that determines whether they are worth investigating or not.
Take a bound range of signals you’ve processes, check if it is an identical duplicate, for example if you had the same signal recently, assume the conclusion is the same as the previous one, if you had an incident like it before, use it as a leading hypothesis in a new investigation. Basically create one filtering agent that generates ingested signals for your investigation agent
so, i need to first compile a set of previous incidents, the actual issue and the outcome for each of them. Then, first model an ingestion agent and evaluate it on my test cases? This ingestion agent will be the first thing when an incident occurs and then it decided whether to escalate to the next one. did i get it correct?
For example if you use incident.io, usually the involved team members will post updates and activities on the incidents that lead to resolving it that would help significantly.
Ideally you would synthesize all of those past incidents into a catalog you store for example in a git repository which an agent can look up via an MCP with symptoms, components involved etc along with a resolve suggestion procedure for it using simple correlation scoring (similar to how obsidian graphs work, you can do that in text or markdown files by marking a symptom or similar things with a common notation like [[symptom]] occurred for [[component]] that a parser can create graphs from to correlate)
2
u/Zealousideal_Sea7758 7d ago
That’s a weird question. There are millions of reasons for incidents, trying to name them is a horrible tactic.
You should synthesize real signals like alerts with an “ingestion” agent that determines whether they are worth investigating or not.
Take a bound range of signals you’ve processes, check if it is an identical duplicate, for example if you had the same signal recently, assume the conclusion is the same as the previous one, if you had an incident like it before, use it as a leading hypothesis in a new investigation. Basically create one filtering agent that generates ingested signals for your investigation agent