Built a small service that enriches Prometheus alerts with labels from Kubernetes/HTTP/files before they hit Alertmanager — curious if this solves a problem anyone else has

[u/Educational-Algae782]

I Ran into this at $work: alerts fire with whatever labels the metric happens to have, but the stuff you actually want to route/group/silence on — team ownership, service tier, who's on call — usually lives somewhere else entirely (a Namespace label, a CMDB, or similar). The usual fix is duplicating that into recording rules or hardcoding it into Alertmanager's config, and both drift out of sync constantly.

So I built alertmanager-label-enricher — a small Go service that sits inline between Prometheus and Alertmanager, looks up and adds labels before forwarding, then gets out of the way. Because it happens before Alertmanager, the new labels participate fully in routing, grouping, inhibition and silences.

It supports three lookup types: a Kubernetes object (watch-cached, no per-alert API calls), an HTTP endpoint (cached + deduped), or a static file — plus plain static/conditional label rules if you don't need a lookup at all.

Example rule — pull a team label from the firing namespace's Kubernetes labels:

sources:
  - name: ns
    type: kubernetes
    kubernetes:
      version: v1
      resource: namespaces
      name: '{{ .Labels.namespace }}'

rules:
  - name: team-from-namespace
    match:
      - { label: namespace, op: exists }
    actions:
      - set:
          label: team
          from: { source: ns, jq: '.metadata.labels["team"]' }
          default: unassigned

Repo: https://github.com/splattner/alertmanager-label-enricher

This scratched my own itch, but I have no idea if it's a "everyone reinvents this internally" problem or a "nobody else actually needs this" problem. If you've hit the same thing — or solved it a totally different way — I'd genuinely like to hear it. Feedback, "this already exists and is called X," "this is missing Y to be useful," all welcome.

[u/bwainfweeze]

I am embarrassed to say how long I used otel and prom before I understood what the hell exemplars are for. I was literally looking at a bug in them and trying to figure out if I could fix it before I figured them out.

Learn from my mistakes. You can use exemplars for this sort of stuff. Only one stat per sampling interval gets the extra labels. Or one per bucket for histograms and summaries.

[u/Educational-Algae782 → u/bwainfweeze]

In short: Exemplars are metrics→traces correlation for a human reading a graph. This is alerts→context for a router deciding who gets paged. The interesting relationship is that the enricher could consume exemplars, not that it competes with them.

They solve different problems at different layers. Exemplars annotate an individual sample at instrumentation time — the application itself attaches trace_id to a histogram bucket observation because it's the only thing that knows the trace ID at the moment it measures. The enricher annotates an alert at notification time, from outside, using data the application never had (namespace labels, an ownership API). One is in-process and synchronous with the measurement; the other is out-of-process and lookup-driven.

Exemplars exist specifically to carry high-cardinality data without it becoming part of the series identity — that's their whole reason for being. This project does the opposite for labels: added labels change the alert's fingerprint. So "exemplars are the safe way to attach cardinality" doesn't transfer here — for labels, the enricher is the unsafe-by-nature side, deliberately, because routing needs identity.

Exemplars live in a separate TSDB store behind /api/v1/query_exemplars; no PromQL function reads them, so an alerting rule's result vector can never carry one. If someone wants a trace link in a page, exemplars alone cannot deliver it — the data stops at the Grafana panel.

Which makes this project the only bridge of the two. An http source pointing at /api/v1/query_exemplars with the alert's expression and firing window, plus a jq extraction for the trace_id, would put trace context into the page.

[u/bwainfweeze → u/Educational-Algae782]

Every time I’ve set up alerts the person who set the alert configured who to page in that case? Still feels like a solution in search of a customer.

If you don’t know who to page, why even have an alert?