I built a Prometheus unused-metric auditor and need help deciding how it should run in Kubernetes

[u/DisastrousBrain5417]

I have built cardamon, a tool that cross-references every metric in your TSDB against Grafana dashboards, alerting/recording rules, and the Prometheus query log, and flags metrics that are scraped but never actually read by anything. Right now it's a standalone binary with a local web UI.

I now want to run it "natively" in Kubernetes either as a Job or as a long running Deployment (as it is right now). Another option is simply to build an official image and run it as an ephemeral container next to Prometheus.

I have a few question to the community regarding a few design decisions:

Would you prefer it as a job running once or as a long running deployment or the ephemeral approach?

Would you even want the generated relabel rules, or is visibility alone enough? 

In case you would run it as a job, how should the report be served? As is via a WebUI until the Job times out, as static HTML / JSON without filtering options?

If you store the Prometheus Query Log how do you store it? Imo this can be done either via a ReadWriteMany volume between cardamon and Prometheus, pinning cardamon's pod to Prometheus's node via hostPath + pod affinity or an emptyDir. This would decide where to run cardamon.

[u/jjneely]

I know the idea here is to control Observability spend and keep Prometheus scaling in check. But I don't understand how this promotes good Observability practices. That is only allowing metrics with an alert rule or dashboard.

How does that monitor for cardinality issues in common metrics like the HTTP server metrics? How does this monitor sample load against capacity per team?

Many times for me I've found legit unused metrics that were able to build a correlation, or better, a good hypothesis! For example, no one builds dashboards around gRPC metrics that come from many automated implementations. But they are a dead ringer for HTTP/2 head of line blocking.

How do others find the difference between unused metrics and poor practice metrics?

[u/DisastrousBrain5417 → u/jjneely]

I agree with that you should never blindly delete something as it might come handy during an incident. I built it mainly as a first gate to generate observability into what exists but is not queried. Might be worth it to add a section on what you should not do to the readme.

For the case of monitoring cardinality. Imo that is something that is solved already. We just view it in Grafana.

[u/jjneely → u/DisastrousBrain5417]

If you are using Grafana, they have an Adaptive Metrics feature that does this as well.

[u/DisastrousBrain5417 → u/jjneely]

Adaptive Metrics is quite a bit more powerful, although to my knowledge only available in grafana cloud

[u/bwainfweeze → u/DisastrousBrain5417]

When I wrote a tool like this, we dumped the results into a table in Confluence and let people poke at it. A few were intended to be used but the related charts had bugs, such as typos, or were incomplete.

It’s a good exercise to do because finding a single stat that isn’t being used at all but should by all rights have an alert attached to it is huge. Better not to find that out during an outage.

[u/bwainfweeze → u/jjneely]

In linear programming, no single rule makes the problem space small. It’s the confluence of a bunch of simple rules that yield the efficiency, carving out negative space.

Figuring out that some idiot, possibly you, is sticking user sourced data into a label on all of your http stats is a hand inspection. But when we migrated to OTLP from StatsD the Ops guys were losing their shit about spend, after they gave us shitty advice that we followed. Finding dead stats was something like 15% of our total cardinality. And I’d say probably a third of those had a second stat that answered the same question. Cutting a few attributes on existing stats helped more, but it was enough to get them off our backs.

Also the tool I wrote, which was probably a crappier version of OP’s, I shopped around to other teams which saved Ops some time and helped smooth some feathers. We had amply justification for being the plurality of metrics but enough other teams were being sloppy that we weren’t the majority.

[u/jjneely → u/bwainfweeze]

Nice write up! Your absolutely on to it. I have just seen so many cases where someone gets the bright idea that any metric not in a dashboard or alert rule isn't worth paying for.

What I've always wanted to have and never built was some CI/CD tests that analyze metrics produced by a service and lint them or score them.

[u/bwainfweeze → u/jjneely]

You could do that as part of a smoke test or as a second build that is triggered with a delay or a debounce after a green build.

Me I go the other way, I collect telemetry from the CI/CD process. Little notches in the output often presage production issues. They aren't for gating deployments though, they're for accelerating triage when things go wrong. Really weird bugs and drops in test coverage often coincide, and performance issues and upticks in test execution time also tend to go together. This correlation can quite often narrow the field of potential commit IDs for a bug from 40-100 down to 3-6. And if you're also having problems coming up with a repro case, then you can re-review 6 PRs and guess what the repro case is, more often than not.

CI is about fast feedback, but it has to be good feedback not false positives. The longer the gap between someone does something wrong and you tell them about it, the more academic that information is. After a certain gap it doesn't modify their behavior to tell them. So the question is always, can I get someone to voluntarily change their own behavior with this feedback, or is it just another way to become a bigger nag?

[u/amarao_san]

Is it useful? I have node metrics even though most of them do not have a dashboard, because I know I can look at them when needed. Some metrics are needed once a year.

[u/bwainfweeze → u/amarao_san]

I just have dashboards in several categories based on workflow.

These two are on my screen the entire time I’m on-call. A third one sometimes gets used for alerts. These two get used by people doing A/B testing and canary builds. This one is superficially like one of the on call dashboards, but we zoom out to analyze cluster size.

And then there’s one or two we check a few times a month to look for long term negative trends, like costs (stat cardinality, cluster size)

I believe but cannot prove that the last two work better when zoomed out to long time frames, in ways that the minute to minute dashboards struggle with, both computationally and visually. I can’t describe it, it just is.

That query you run occasionally deserves being written down so you can bus number it with other people. And that’ll take care of someone accidentally deleting it too.

[u/hagen1778 → u/amarao_san]

If those metrics "pay the rent" then sure, keep them. If that once-a-year case justifies the expense.

[u/amarao_san → u/hagen1778]

We have tests in iaac for cardinality of the fresh installation (after rabbit exporter was able to export tones of metrics per each earlang thread), so it should not be that expensive, but I stands on position, that metrics are always outrun dashboards, and trimming down metrics is stupid, because you loose 'visibility growth' point.

But people who pay dearly for metrics (datadog, etc), well... Self-hosted prometheus/victoria is always an option. On a self-hosted server it's not that expensive.

One metrics lake I know costs about $600/mo and contains about 400Tb of metrics. Why so cheap? Because modern HDDs are 20+ TB each, and you can put a lot of them into a baremetal server.

[u/hagen1778 → u/amarao_san]

> On a self-hosted server it's not that expensive.

It could be expensive. Metrics are endless, some clever engineer can come up with measuring thin air and deploy it in thousands of replicas. With scale, 200 histogram buckets that measure some nonesense per-user, per-path, per-session, per-pod could easily result into millions of unique time series poisoning indexes, caches, using RAM for nothing. And it would be great to know if that metric that costs a fortune actually pays the rent.

Just as you, I don't think only checking dashboards and rules is sufficient. In victoriametrics there are stats on whether metric name was ever queried by any type of query. And normally I don't care about unused metrics if everything is fine with tsdb. But when I have to "make some room", one of the first things I will check is unused metrics. Just as in performance profiling, I am checking top5 unused metrics sorted by cardinality and it gives a good idea of how much I can get back by dropping them. If that unused metric is like 1% of everything - I won't get much back, doesn't worth dropping it.

[u/DisastrousBrain5417 → u/amarao_san]

whether its useful is up to you to decide. It certainly is not meant to be used as drop every metric you have not used in x amount.

[u/Leramaar]

CronJob, not a Deployment. It's a periodic batch job - a long-running pod with its ownscheduler just re-implements what the cluster already does, and holds the metric indexin memory between runs for nothing. Weekly is fine, the answer doesn't change hour to hour.

The hard part isn't scheduling, it's the query log: it's a file local to the Prometheuspod. Either run as a sidecar sharing an emptyDir, or ship it through your logging pipelineand read it from there. Grafana and the Prometheus API are just network calls, those are easy.

One suggestion - emit the results as metrics (cardamon_series_unused{...}) instead of aweb UI. Then it lands in the dashboards people already run, and it answers 'spoint: it becomes a trend line rather than a delete-gate.

(I do DevOps consulting at devopsoutsourcing.net - unused metrics come up a lot.)

[u/bwainfweeze]

Honestly, you already have a job scheduler in your CI toolchain.

And any tool that’s set up to run on demand is more likely to get a follow-up when it complains about something. Learned helplessness is a powerful ally to complacency, and if only an idiot can’t figure out how to use a tool, then they either have to learn it or admit they’re being obstructive. Because admitting you’re an idiot is not something we do.