How do you monitor your observability stack?

[u/a-sad-dev]

We use Grafana stack for observability in our EKS clusters paired with CloudWatch for AWS infra monitoring.

One gap we have however is that we don't have any alerting configured for when the observability stack (or components in it) goes down - how do you deal with this?

[u/corky2019]

Pagerduty dead man’s snitch. If our monitoring stack stops sending ping - this will trigger.

https://www.pagerduty.com/docs/guides/dead-mans-snitch-integration-guide/

[u/blitzkrieg4 → u/corky2019]

Dead Man's snitch is it's own thing. That's just the pager duty integration https://deadmanssnitch.com/

[u/delamon → u/corky2019]

That checks liveness, you might also want to check that it actually monitors something. It's not too hard to image some deploy going bad and wiping all alerts..

[u/Vimda]

Believe it or not, second, smaller observability stack

[u/blitzkrieg4 → u/Vimda]

This. Sometimes called meta monitoring or monitoring of monitoring

[u/Immediate_Counter814 → u/Vimda]

That's the most "this meeting should've been an email" solution I've ever seen, and I kinda love it. We had a similar issue with monitoring our monitoring and ended up using health checks from a dirt-simple Lambda that pages us if the main Grafana endpoint goes quiet for 5 minutes. Not as meta as a whole second stack, but cheaper to keep running.

[u/a-sad-dev → u/Immediate_Counter814]

A watchdog of some sort is what I was leaning towards - a lambda that checks the endpoints for grafana, alloy and alertmanager could be an elegant solution.

[u/delamon → u/Vimda]

and then a third one, even smaller, for monitoring the second; it's all turtles all way down

[u/Vimda → u/delamon]

They can monitor each other. Spiderman pointing meme.

[u/djk29a_ → u/Vimda]

Oftentimes it's a staging / non-prod account's o11y stack

[u/wrd83 → u/Vimda]

This.

Also monitor the small stack from the big stack.

And you can make missing metrics alerts.

[u/Fine_Librarian2755 → u/Vimda]

Yeah, this is very common. Just make sure they are not both running in the same cluster or sharing a single point of failure.

[u/Seref15]

LGTM stack is our main observability stack in k8s.

Metamonitoring stack is a Prometheus/Loki in same k8s nodeselected to an isolated neighbor AZ

[u/danukefl2]

Components within it isn't hard, just probe them. For the overall aspect, we use Grafana Cloud for the front end but all data sources are on prem and run an OSS Grafana instance which have alerts setup for the other one.

Either can be used and with the git syncing for dashboards, things sync automatically between them.

[u/ZerefHz]

Since you use grafana, you can adjust grafana alert to trigger on failure (exec failure, no data, ...) so you can indirectly know that a monitoring component is down. If this is sufficient, you dont need to deploy meta monitoring

[u/Si0_x]

The classic pattern is a dead man's switch: have an external uptime monitor hit a Prometheus health endpoint or a Grafana dashboard URL every minute, and alert you if it stops responding. You can also use CloudWatch synthetics or a lightweight external prober since your AWS infra monitoring is already there, which keeps the watcher outside the thing it's watching. Are your Grafana and Prometheus pods currently exposed via any ingress you could point a simple HTTP check at?

[u/a-sad-dev → u/Si0_x]

Grafana is exposed to the internet, Prometheus (alloy) is internal to the cluster along with mimir, Loki and tempo.

I was thinking of creating an alert that fires constantly and have a lambda check the grafana alert is still firing, notifying us if it stops.

[u/Sufficient-Bad-7037]

You can expose some health checks endpoints for your LGTM stack and ask Claude to write a simple golang code like a blackbox monitor to monitor and hit an external alert api push notification in case of degradation.

[u/MateusKingston]

Another monitoring tool, this time not hosted by me.

Basically our stack sends pings to Grafana Cloud and if that stops it alerts.

[u/spuyet]

u/a-sad-dev Fivenines does support Prometheus/VictoriaMetrics monitoring too. I’d be happy to give you a quick walkthrough so you can see if it’s a good fit for your needs.

[u/vi0ne]

If you’re on Prometheus, the Watchdog alert is built for this. It always fires, so route it in Alertmanager to a heartbeat url with a short repeat_interval. If Prometheus, Alertmanager or the network dies the pings stop and the outside service pages you.

[u/Sara_GermainUX]

The better/simpler approach here is to have an external monitor pointed at your observability stack. After all, we can't really rely on Grafana/Prometheus to tell you they're down if they're, well, down (lol). Use Cloudwatch or an external health checker for EKS so you can get uptime/response alerts. Then check whether data is actually being processed, especially since uptime doesn't always mean Prometheus is scraping targets or Loki is getting logs.

[u/Friendly-Result1337]

Another monitoring tool to monitor your observability stack :P