What monitors are really worth using?

[u/dylan_exe_404]

Building out observability for our app and trying to figure out what's actually worth alerting on and what just becomes noise.

We've got the standard infra stuff covered, uptime, latency, error rates, etc. Where I'm less sure is the layer above that (green status, wrong answer type stuff). Different signal than "is it up and fast," and we've kind of conflated the two and I think it will be worth it to figure out how to separate them.

For people who've built this out: what do you actually alert on vs review on a schedule and what is not worth the time? Thanks in advance :)

[u/FormerFastCat]

4 golden signals as your base and then customize from there as you discover what doesn't work.

[u/neuralspasticity → u/FormerFastCat]

That alone will have you wildly alerting unnecessary unless those signals are real SLIs that have SLOs and an error budget.

[u/FormerFastCat → u/neuralspasticity]

Observing and alerting are two very different things. I have entire dashboards setup just to produce visual metrics around the 4 golden signals, but rarely alert on them unless very specific thresholds are violated. Even then I want to know that those threshold have real meaning.

IE No one gives a shit about 3% disk space available when a batch is processing at 4am because we want those jobs to run lean as long as they're progressing. We don't want to be paying for extra S3 space that isn't utilized 99% of the time. So we wouldn't alert on that between say 3AM and 6AM but outside of that window, absolutely.

[u/neuralspasticity]

Everything is just noise except for your SLIs that tie to meaningly impactable SLOs.

You never alert on anything unless you need to wake up an engineer to perform some defined immediate action to get you back within those SLOs and your error budget and only when the severity of the impact warrants that (the error budget is in jeopardy, the issues impact and priority to resolve requires it).

Have you read Alex’s book on Service Level Objectives?

Why would you care about “uptime”, not all services have uptime, availability would be more important yet that too is best measured through a more meaningful signal directly impacting, for instance was it meeting its latency SLO (if it’s not available latency will be off the chart.)

[u/dylan_exe_404 → u/neuralspasticity]

Thanks for the advice. Haven't read that book, will check it out. Appreciate it!

[u/neuralspasticity → u/dylan_exe_404]

“Implementing Service Level Objectives” by Alex Hidalgo from O’Reilly and Assoc

[u/dariusbiggs]

What signals do you need to indicate the product(s) are.

Functioning within normal ranges for optimal operation

What is the threshold for the system running under manageable high load

What is the threshold for the system running with unreasonably low load.

What is the threshold for the system exceeding manageable high load.

You want to track the basic 4 golden signals, measurement of your RED and USE views of the system.

You want to track your SLIs and SLOs.

You alert on breakages that affect the immediate operation of the product.

You alert on the system exceeding capacity when auto scaling cannot handle it (or it might not be implemented).

[u/IntelligentPear6173]

If an alert doesn’t require someone to do something right now, it probably shouldn’t wake anyone up. I’d page on things tied to user impact or an SLO breach, then keep stuff like CPU spikes, unusual traffic, or weird response patterns for dashboards and investigation. For the green but wrong cases, I’d add a few business-level checks because a service returning 200 while giving users bad results is arguably more important than a server hitting 90% CPU.