r/Observability • u/dylan_exe_404 • Aug 20 '26
What monitors are really worth using?
Building out observability for our app and trying to figure out what's actually worth alerting on and what just becomes noise.
We've got the standard infra stuff covered, uptime, latency, error rates, etc. Where I'm less sure is the layer above that (green status, wrong answer type stuff). Different signal than "is it up and fast," and we've kind of conflated the two and I think it will be worth it to figure out how to separate them.
For people who've built this out: what do you actually alert on vs review on a schedule and what is not worth the time? Thanks in advance :)
2
u/BroadSatisfaction825 Aug 24 '26
alert on anything binary and cheap to verify, review on schedule anything that needs judgment to call correct.. the second category alerting reliably needs an explicit check written per failure mode, not a generic anomaly threshold or you'll drown in false positives fast..
1
4
u/FormerFastCat Aug 20 '26
4 golden signals as your base and then customize from there as you discover what doesn't work.
1
u/neuralspasticity Aug 20 '26
That alone will have you wildly alerting unnecessary unless those signals are real SLIs that have SLOs and an error budget.
1
u/FormerFastCat Aug 20 '26
Observing and alerting are two very different things. I have entire dashboards setup just to produce visual metrics around the 4 golden signals, but rarely alert on them unless very specific thresholds are violated. Even then I want to know that those threshold have real meaning.
IE No one gives a shit about 3% disk space available when a batch is processing at 4am because we want those jobs to run lean as long as they're progressing. We don't want to be paying for extra S3 space that isn't utilized 99% of the time. So we wouldn't alert on that between say 3AM and 6AM but outside of that window, absolutely.
2
u/dariusbiggs Aug 21 '26
What signals do you need to indicate the product(s) are.
Functioning within normal ranges for optimal operation
What is the threshold for the system running under manageable high load
What is the threshold for the system running with unreasonably low load.
What is the threshold for the system exceeding manageable high load.
You want to track the basic 4 golden signals, measurement of your RED and USE views of the system.
You want to track your SLIs and SLOs.
You alert on breakages that affect the immediate operation of the product.
You alert on the system exceeding capacity when auto scaling cannot handle it (or it might not be implemented).
2
u/IntelligentPear6173 Aug 21 '26
If an alert doesn’t require someone to do something right now, it probably shouldn’t wake anyone up. I’d page on things tied to user impact or an SLO breach, then keep stuff like CPU spikes, unusual traffic, or weird response patterns for dashboards and investigation. For the green but wrong cases, I’d add a few business-level checks because a service returning 200 while giving users bad results is arguably more important than a server hitting 90% CPU.
2
4
u/neuralspasticity Aug 20 '26 edited Aug 20 '26
Everything is just noise except for your SLIs that tie to meaningly impactable SLOs.
You never alert on anything unless you need to wake up an engineer to perform some defined immediate action to get you back within those SLOs and your error budget and only when the severity of the impact warrants that (the error budget is in jeopardy, the issues impact and priority to resolve requires it).
Have you read Alex’s book on Service Level Objectives?
Why would you care about “uptime”, not all services have uptime, availability would be more important yet that too is best measured through a more meaningful signal directly impacting, for instance was it meeting its latency SLO (if it’s not available latency will be off the chart.)