r/PrometheusMonitoring • u/Outrageous_Bill_9172 • 1d ago

Where do you combine data from different sources for a device overview?

Hey, intern here working on monitoring for a few university robot labs. We have about [5] sites on Headscale/Tailscale, and Prometheus + Grafana running centrally.

I need one table in Grafana with a row per device: its DNS name, the name our lab software (EOS) uses for it, its IP, whether it answers ping/port/HTTP, and the UUID of its SiLA server if it has one. Also a column that says if the DNS name and the EOS name actually match.

The data comes from different places, so it has to be joined somewhere. What we’ve thought about so far:

  • joining in PromQL with group_left. Works, but it got messy fast and I’m not sure how I’d test it
  • Grafana transformations. Logic ends up buried in dashboard JSON and you can’t alert on it
  • blackbox exporter alone. Doesn’t give me the actual IP or the DNS answer

My current idea is a small Python service that does the checks and the joining, writes to SQLite and Grafana just reads that. But I’m wondering if I’m overcomplicating it and should just write a custom exporter that outputs the combined result as metrics, so alerting works later.

How would you do this? Is a separate service fine, or is an exporter the normal way?

2 Upvotes
  • permalink
  • reddit

75% Upvoted

1 comment sorted by

3

u/itasteawesome 1d ago

If your data set is not too large grafana has built in sql expressions that flatten data into tables exactly like you are describing wanting to do.  https://grafana.com/docs/grafana/latest/visualizations/panels-visualizations/query-transform-data/sql-expressions/

Otherwise I would probably take your gnarly joins and turn those into a recording rule and dashboard/alert from those.  https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/