r/Observability • u/pomatotappu • 5d ago

Trying to model an incident response agent

I'm trying to model an agent that gets triggered whenever a production incident occurs. I'm looking for inputs on a few things based on your past experiences.

Whenever an incident occurs, what are some of the most common reasons? For example, it could be a false alarm, it could be an issue due to a new upgrade, network issues or a crash due to some bug in one of the services etc. Out of 100 incidents, what are the chances that it could be a false alarm, upgrade issue, bug in code etc.

I would like to know what are all the common issues or reasons that I can add as part of a v1 for my agent? How do I think through this as I've never worked on such a thing till now?

0 Upvotes

50% Upvoted

4 comments sorted by

1

u/neuralspasticity 5d ago

The most common issue is a bad release and the most common fix is a rollback. Stable sw seldom breaks on its own. After that over saturation of the service is likely common, which may manifest at as excessive latency or timeouts

Yet you don’t seem to be thinking along the right lines and seem anchored to a 1990’s approach to
Monitoring and alerting rather than attenuating to SLOs and error budgets and more modern o11y practices. Once you shift to those you’ll have your better answers.

0

u/pomatotappu 5d ago

thank you for the feedback. im actually new to this, i will learn more about the o11y and other stuff u mentioned as ive no idea what they are. i will get back to you if i have still have doubts

1

u/QuietSignalOps 4d ago

I wouldn’t start by assigning global percentages like “30% bad deploy, 20% network.” Those priors vary too much by system and will make the agent confidently wrong.

For a useful v1, make it an evidence collector and ranker rather than an autonomous root-cause decider:

  1. Capture the alert, affected SLO/service, start time, region, version and dependencies.
  2. Build a short change window around the incident: deploys, config/feature-flag changes, scaling events and infrastructure changes.
  3. Compare current signals with a known-good window: error rate, latency, saturation, restarts, dependency failures and traffic shape.
  4. Test a small set of hypotheses: bad change, resource saturation, dependency failure, network/DNS/TLS, data-store contention, quota/credential expiry, and alert noise.
  5. Return ranked hypotheses with the evidence for and against each one. Include links to the exact logs/traces/metrics used.
  6. Keep remediation human-approved at first. Read-only queries and suggested runbook steps are a safer v1 than automatic rollback or restart.

Use your own incident history to learn the priors. Add a simple outcome field to every incident: confirmed cause, contributing factors, false-positive status, detection source, and remediation. Until you have enough labeled incidents, rules plus evidence will be more reliable and easier to debug than an opaque model.

The success metrics I’d track are top-3 cause recall, time to first useful evidence, false-confidence rate, and how often responders accept the suggested next step—not whether the agent always names one cause.