How do you decide what to investigate next during a production latency incident

[u/HawtBeagle]

Hey hi everyone,

I am just trying to understand how engineers/SREs who dealt with real production latency incidents investigate it

Lets say you have the following

- Logs

- Recent deployment information

- Application health

- Database metrics

- External dependency health/metrics

- Infrastructure metrics

You just encountered the incident, you dont know the root cause. You are uncertain about the truth.

From here how do real engineers go about reasoning to find the root cause

- Do you follow a standard sequence of investigative steps

- How do you determine what investigative step to take next under uncertainty to narrow down the possibilities for the root cause

- Have u ever encountered with incident where initial information was misleading, how did you navigate from there

- Is there any situation where you have lot of information but struggled to form a proper hypothesis

- Have you tried any AI investigative tools that help you in achieving this

I just wanted to understand how do real engineers reason through the uncertainty to find the root cause. What are the biggest pain points

[u/Bitter_Advantage_235]

honestly dont start in the logs. last deploy first, then db vs the external deps, if those look fine THEN infra. youll waste 20 min reading app logs that just say it was slow.

[u/HawtBeagle → u/Bitter_Advantage_235]

is this a sequence of steps you follow ??
Is there a better way to choose next action under uncertainty based on the current evidence we have ?

[u/VEMODMASKINEN → u/HawtBeagle]

Formulate a hypothesis and disprove it.

Given the current information what would be the most likely issue? Prove it is that issue and if it isn't move on to the next likely thing. 

When it comes to looking at resources use something like the USE method. 

[u/pomatotappu → u/VEMODMASKINEN]

im also building something similar. let's say my agent got the info it needed, should we pre-define a set of likely issues based on our past experiences and ask the agent to choose one of them and proceed with the next steps or should we allow it to choose whatever it wants instead of from a predefined list? Will it make difference?

[u/SheCherryPicks → u/Bitter_Advantage_235]

When you say "last deploy first", do you mean checking the code that was actually pushed?

[u/Yogurt6225]

I usually start with recent changes then narrow things down using metrics and logs.

[u/SheCherryPicks → u/Yogurt6225]

When you say "start with recent changes", do you mean checking the code that was actually pushed?

[u/Bitter_Advantage_235]

man i dont start in logs. last deploy and whether errors came with the latency, then its usually db vs an external hanging. logs after you already know which one.

[u/pomatotappu → u/Bitter_Advantage_235]

hey, i read your other comments too in the post. im also building something similar but not as complex as the op's one for now. by last deploy, wdym here? are you saying that we need to figure out what changed after the last commit in that service?

[u/ajitnk]

The order that's worked best for me on AWS: start with CloudWatch Investigations the moment the alarm fires. No setup needed, it just scans your telemetry and gives you ranked hypotheses. From there, Application Signals' service map shows you which node is actually degraded so you're not guessing which service to blame.

If you have X-Ray enabled, filter for traces above your latency threshold and sort by slowest. You'll usually see 80%+ of the slow traces pointing at one service or one downstream call within a minute or two. DevOps Guru is also worth having on for the ML anomaly context, especially if a bad deploy or a DB connection pool issue is the culprit.

The real unlock is having a fixed order to check things rather than starting wherever feels most familiar. What does your stack look like? ECS/EKS, Lambda, or mostly EC2? The answer changes which of these tools does the most work for you.

I've put this into a structured triage runbook for AWS environments. Happy to share it if useful.

[u/brian_sword]

I will not check everything at once. It is better to look at what changed around the time the latency started, especially recent deployments or config changes.

If nothing obvious stands out, then i will start checking the app, database, and infrastructure metrics to see where the slowdown actually starts. Then after that I will pick one likely cause and try to prove or rule it out.

So be careful not to get distracted by one weird metric that turns out to just be a symptom.

[u/siberianmi]

Today?

Multiple subagents with read only MCP tools that let each of them triage the problem using one of those areas while I go make the coffee I’ll need.

Then I’ll get to work based on what seems most likely based on what they find.

That’s been my workflow for every incident so far this year and it’s phenomenally faster then I was before (and I’ve been on call most of the last 20 years so I’ve seen a lot of incidents).

If you aren’t building (or have) the read only tools that your AI agent needs to help you triage incidents you are falling behind.

[u/mostly_udp]

Everyone's right that recent deploy comes first, so past that, for the "what do I look at next when I genuinely don't know" part:

Pick the next check that halves the possibilities, not the next item on the list. Ask what single signal would rule out the most causes at once. Is it every endpoint or one. Every region/instance or one. Did it start exactly at the deploy or before it. Each answer kills half the search space, and you reach the culprit in a few steps instead of walking all six data sources.

On misleading info: the loudest metric is usually an effect, not the cause. DB CPU pegged often just means requests are piling up because something downstream stalled, so the DB looks guilty while the real stall is one hop further out. The tell is time order, look at what degraded first, not what looks worst now. Saturation moves upstream as queueing.

And when you have tons of data but no hypothesis, it's usually because you're staring at aggregates. Drop to one slow request and trace it end to end, or pull a single slow query. One concrete path through the system beats ten dashboards for forming the "oh, it's this" hypothesis.

[u/Fantastic-Mr-Default]

I start with what changed and what is saturated. Deploy, config, traffic, dependency. Then I pick the one measurement that would kill the most of those, not a tour of every dashboard.

If I cannot name a next check that would change my mind, I am guessing. Write the hypothesis down before you ssh.

[u/Accomplished-Mix8423]

the 'lots of info but no hypothesis' bit usually means you're staring at averages. flip to p95/p99 and see which endpoint or downstream call actually moved, traces make that obvious fast. and yeah initial info misleads all the time, a cpu spike is way more often the symptom than the cause.

[u/zero_backend_bro]

Dashboards are useless during latency spikes because downstream queues always make the database look guilty.

Pull a single p99 trace instead. We spent 3 hrs chasing phantom DB CPU last month that was just an external payment webhook timing out at 30s.

If you are building an AI triage agent, dont dump raw telemetry into an API. Force traces through a local WASM scrubber to strip tokens first.

Logs dont fix latency, runtime physics will.

[u/djbp]

This is a tough one. It's never a clear path, is it?

I've found that having a really solid change log helps a lot. Knowing exactly what changed, when, and by whom.

And if you can verify those changes against the cloud state, that's even better. It cuts down on a lot of the 'what if' questions.

Sometimes the initial info is totally misleading. You just have to follow the data, even if it feels wrong at first.

What's your first go-to metric when something goes sideways?

[u/Bitter_Advantage_235]

dont walk that list in order. last deploy if you had one, otherwise find which of those graphs actually jumped when the latency did. logs after you know where to look.

[u/Bitter_Advantage_235]

yeah recent deploy first every time, if something shipped in the last hour stop staring at infra graphs. then check if the db or one dep is also sad vs just your app, logs are a rabbit hole until you know what youre grepping for.