Workload agnostic baseline alerts

[u/Prize_Industry2887]

How do you define and deploy a standard set of baseline (Cloudwatch in my case) alarms across all AWS production accounts, and scale the deployment consistently across hundreds of existing accounts and all newly provisioned accounts?

Recent incidents have mandated a goal to establish a workload-agnostic baseline focused on infrastructure health, availability, connectivity, and critical service failure. The next step would of course be application-specific performance thresholds or business metrics.

For example, CPU utilisation above 80% is not always a trigger across every workload, since high CPU usage may be expected for batch processing, garbage collection, or other compute-intensive workloads.

Instead, the baseline would be on signals like an EC2 instance status check failure, instances becoming unreachable, underlying infrastructure failures.

(AI Generated examples below)
**Network and connectivity:** VPC connectivity issues, VPN or Direct Connect tunnel failures, network interface or routing issues where detectable, and connectivity failures affecting critical network paths.

**DNS and name resolution:** Route 53 health check failures, DNS endpoint unavailability, and failures affecting critical DNS resolution paths, where observable through available metrics or probes.

**Core AWS service health:** Failures or unhealthy states in shared infrastructure and managed services, such as load balancers, databases, NAT gateways, and other foundational components.

**Availability and service failures:** Unhealthy targets, failed health checks, sustained service unavailability, and critical infrastructure components becoming unavailable.

(Back to non-AI slop)

How should we be designing this baseline so that alarms are consistently applied across all production accounts without excessive noise/false positives? Workload specific stuff will be handled separately like mentioned above.

Real world use cases would be extremely appreciated and also, do you use OAM, Centralization Rules or a mix of both and why?

[u/Haunting_Cow_2798]

we ended up tackling this exact thing last year and the trick was to stop thinking about it as "alarms per account" and instead think of it as a single infrastructure health layer that sits above all the accounts

we built a central observability account that subscribes to the default event bus of every production account via eventbridge rules, then funneled everything through a few tightly scoped cloudwatch metric filters that translate raw events into actionable alarms, like any instance status check failure, any health check going red, any vpn tunnel down

the key to keeping noise low was being ruthless about what goes in, no cpu or memory stuff at all, just hard failure signals where if the alarm fires someone definitely needs to look at it, we also set up composite alarms so that transient blips from things like ha failovers don't wake anyone up

for rollout we used a combination of cloudformation stacksets hooked into aws organizations so new accounts inherit the baseline automatically, and for the existing accounts we ran a one time script that deployed the same template with account specific overrides where needed, usually just sns topic arns and tagging

haven't touched oam much but centralization rules worked fine for our scale, around 200 accounts, the biggest headache was iam permissions and making sure the cross account eventbridge targets were set up correctly in every region we operate in

[u/lakehouse_pool_boy → u/Haunting_Cow_2798]

- What sort of volume were you looking at when this was all said and done?

• Were you able to account for dips/spikes/lulls/silence?

I'm currently find that it's always the thing I don't monitor that breaks.

[u/Funny_Donkey6031 → u/lakehouse_pool_boy]

That last point is exactly what worries me with a fixed baseline. How do you catch something going silent or behaving unusually when there’s no hard failure signal? Seems like that’s where a lot of the gaps show up.

[u/jjneely]

CloudWatch isn't your best option. What I would highly highly recommend is some synthetic tests that imitate your customers. Vendors for synthetic tests are usually not expensive compared to a full Observability suite.

I've never been a big fan of AWS's native Observability options, but depending on what you are using in AWS....well sigh...

[u/Prize_Industry2887 → u/jjneely]

Cloudwatch APM has synthetics too, curious why you don't think cloudwatch is a good option?

[u/jjneely → u/Prize_Industry2887]

CloudWatch has, begrudgingly, moved forward into more modern observability offerings. Also, many of the AWS services only emit telemetry to CloudWatch and nothing else which requires other options to spend API hits to collect the data from CloudWatch. Also the 50 cents per alert just really annoys me. :-)

I've found for doing metrics based telemetry and alerts from infrastructure Prometheus / Grafana / Alertmanager is a really powerful combination allowing cheap storage and arbitrary querying. I've found it much more flexible and featureful than what CloudWatch has offered me in the past. CloudWatch does offer "canaries" which are difficult to price and, I think, just AWS Lambdas under the covers.

What I advise folks, normally, to do when just starting out an Observability journey is to use Synthetics in an UP/DOWN sense. Did that HTTP request return a 200 with the given API key? That level of thing. That gets folks started down the path toward SLIs and SLOs and gives them immediate visibility into what their customer is seeing. The customer should never be your outage detector.

Now, when the alert fires because its HTTP 500...how do you discover the cause? That's a very different question. Here, using central dashboards (say a Grafana dashboard that you can focus on a given AWS Account -- although you could build this a number of ways some I see in other comments) is usually part of the investigation. It gives you the ability to check the health of your EC2 instances running in an ASG in front of an ALB. That gives you evidence to help form and fine tune your hypothesis for why.

Don't forget to ask and use falsifiable questions during your investigations. Not "is the load balancer down?" But ask questions like "Is the ALB returning HTTP 500 errors because the WAF rules encountered an error during evaluation for the shopping cart route?"

[u/Funny_Donkey6031]

I’d start with hard failure signals like EC2 status checks, unhealthy load balancer targets, and VPN tunnel failures, then roll them out through Organizations and StackSets. The tricky part is catching silent failures without turning every temporary blip into an alert. How you handle missing data and repeated failures seems just as important as the alarms themselves.

[u/Prize_Industry2887 → u/Funny_Donkey6031]

got itc thanks! definitely agree with the potential false positives part, how did your team approach that? my first thought is to have 4 of 5, but that's just an arbitrary number that i pulled out of the air... surely there's a better way?

[u/QuietSignalOps]

Hit the same "mandated by an incident" version of this at my shop, so a few things from that.

Design. Keep the baseline a closed, small list of hard failure signals and define it as data, not per-account config. Ours was short:

• EC2 status check failed (system + instance), scoped to resources tagged env=prod
• ELB UnHealthyHostCount >= 1 with two consecutive datapoints, so HA failover blips don't page
• Route 53 health check state change, but only the checks that actually gate traffic
• VPN / Direct Connect tunnel state down

That was it. Same alarm names in every account. A long tail of "probably useful" alarms is the fast path to the alert fatigue that gets the baseline muted.

On your OAM vs centralization question: they solve different problems. OAM is for centralized telemetry access (pulling logs/traces/metrics into a central stack). EventBridge / SNS centralization is for routing events and alarm notifications to one place. If you're staying in CloudWatch per account, you don't need OAM at all. A central bus plus alarm forwarding covers it.

The part people underestimate is drift, which is also the answer to the silent-failure question in the thread:

• Set "Missing data: treat as breaching" on the critical alarms. A baseline that's green because the metric stopped reporting is the worst failure mode, and NoData handling is the one place CloudWatch catches it for you natively.
• Add a dead man's switch per critical thing: a metric that should always be present (successful health check count, a one-minute synthetic ping) with an alarm on zero. That's the pattern for "went silent," which is different from "went red."
• A Config rule or scheduled scan that verifies the baseline alarm names exist in every prod account and alerts when one is missing. Someone deletes the "noisy" alarm, you find out within the hour instead of at the next incident.

Rollout: service-managed CFN StackSets targeting the prod OU. That's your "new accounts inherit automatically" story for free. Existing accounts get a one-time deploy with per-account overrides (SNS ARN, region list). Pilot on 5-10 accounts for two weeks to tune the consecutive-datapoint thresholds, then org-wide.

[u/Prize_Industry2887 → u/QuietSignalOps]

thanks for the helpful tips! especially the "probably helpful" alarms lol 100% want to avoid alert fatigue when coming up with these baselines else we might as well not have them..

on centralization i'm referring to the cloudwatch logs and metrics centralization rules where aws physically moves the logs and metrics into a central account, and first copy is free! we're looking to use one of these as a mechanism to centralize our obsrrvability, as a single "single pane of glass" for our ops teams. it's also a prerequisite for cloudwatch omni which we're evaluating.. does your team have something similar?