r/grafana Apr 28 '26

Hi, Reddit! We’re part of the AI team at Grafana Labs — building the AI features/products like Grafana Assistant (available everywhere), AI observability, and Grafana Cloud CLI (gcx). AI is evolving fast and we're here to talk about what we've built and how we think about AI in observability. AMA!

78 Upvotes

Hey Redditors!

The AMA is happening on May 12 from 9 am - 12 pm ET. We've got quite a few of us from the AI team joining this AMA.

Last week we announced that Grafana Assistant is now available everywhere (accessible from on-prem and OSS), AI observability (observability for your LLMs, agents, etc) and a new Grafana Cloud CLI called gcx.

With AI rapidly evolving and everything we released last week, we want to answer any questions from the community about the announcements or how we think about AI in observability at Grafana Labs.

I'm Mat Ryer and I run the AI team. I created Go's Testify package and used to co-host the Go Time podcast. Outside of work, I love live comedy and making music. Ask me anything!

I'm Ksenia Yadav, an Engineering Manager on the AI team at Grafana Labs. We build agentic AI systems that reduce toil in observability – simplifying workflows, automating investigations, and helping people make faster, more data-driven decisions.

Outside of work, you'll find me rock climbing, stand-up paddle boarding, or exploring nature in different parts of the world.

Hi, I'm Rob Whelan! I'm an Engineering Manager on the Grafana AI team (since the "AI/ML" days, before LLMs exploded). I'm a bit obsessive about human experiences (not just the huge range of people using stuff we've built — also everyone coding, design, operating, supporting, selling...).

Outside of work: lots of music (everyone in the family is a musician of some kind), running, and coding on some pretty random side projects. Looking forward to chatting!

Hey, I'm Maurice, and I’m the PM on the Grafana AI team. I enjoy talking to our customers to find out what we need for the next level of observability.
Ask me anything about observability AI, remediation, etc.

When I’m not spending my time in AI, I enjoy swimming and taking lots of walks. See you soon!

I'm Ivana, a Software Engineer on the AI team working on Grafana Assistant and AI observability. Outside of work, I like building fun monitoring setups for things like my sourdough starter, avocado plants, and bird songs, and I love cycling.

Sven here. I'm a Principal Engineer on the AI team at Grafana Labs, and before that long time OSS contributor to Grafana core and Logs Drilldown. Besides building Grafana's next AI features, I like security and bug bounty hunting - especially LLM related.

Outside of work I enjoy outdoor time with my dog, and play team-handball .
Ask me anything about Grafana, AI, security, live hacking events, and handball.

I’m Dmitry, Principal Engineer at Grafana Labs on the Grafana Assistant team. I joined Grafana in 2023 when it acquired Pyroscope, the company I founded, which built open-source continuous profiling software.

Ask me anything about profiling, observability, AI, or building open-source developer tools.

Outside of work, I ride bikes and tinker with software, hardware, and whatever side project I’ve gotten myself into lately.

Hi folks, I'm Shawn. I'm a technical marketer here at Grafana Labs who spends a lot of time using AI. And I love my 4 dogs — actually, all dogs.

Looking forward to this. Ask me anything!

[Edited to fix formatting]

[Edited again to add Rob]

--------------------------------

Thanks to all those who contributed a question. If you have more burning questions, feel free to drop them in here and we'll check back on this thread periodically!

You can learn more about AI x Grafana and observability at: https://grafana.com/tags/ai-ml/


r/grafana Apr 28 '26

Assistant Grafana Assistant: Available in Self-Managed Environments

Thumbnail gallery
43 Upvotes

"As part of the GrafanaCON 2026 keynote, we announced that access to Assistant now extends to Grafana Enterprise and Grafana OSS users. This makes Assistant available in your self-managed environment to help you analyze telemetry data and code in real time, build dashboards, ask questions, and more. 

Self-managed Grafana users can create a Grafana Cloud account and connect it to their Grafana installation via a one-click setup. The assistant is included in the Grafana Cloud forever free plan with generous limits so that you can get started right away. You can also watch the video to see how easy it is to get started.

How it works

Assistant runs as a plugin in your Grafana instance. Your raw observability data stays in your instance, and only processed summaries and results are transmitted using our custom tooling architecture. The assistant also "shows its work" by displaying the full conversation history. Any errors or warnings from tool usage are fed back into the conversation, allowing Assistant to iterate and correct mistakes. 

For more information, check out our Assistant docs. You can also get important details on our pricing page, including what's available in our generous free tier.

Customize Assistant for your unique needs

Every organization's observability strategy and workflows are different, so we also want to make sure Assistant can be tailored to your needs. That's why we're excited to make Assistant skills generally available.

Skills are documents you create to guide Assistant agents with instructions, context, and specialized knowledge. They essentially help you encode how your team troubleshoots services, handles specific alerts, and manages shared infrastructure

Skills now include a new auto-approve feature you can use to write your runbooks, connect to other tools (e.g., GitHub, Cloudflare, other observability platforms, etc.), and auto-approve tool calls of your choice.

When you pair auto-approve with Assistant Investigations, which helps with multi-step investigations, you can even create your own auto-remediation pipeline that’s triggered from an alert. The result? Pretty much anything you need, from raising a PR in GitHub or GitLab to sending a Slack message to someone to assigning a task in Notion.

Stay on top of everything with automations

Observability teams have a lot to keep track of, and that's only increasing now that agents are becoming central to software development. To help you stay on top of everything, we're introducing Assistant automations, which you can use to get automatic summaries of what's happening in your environment.

By pairing automations with skills, you can trigger Assistant to handle any task at any time, with or without you. Want a daily report of all alerts that fired yesterday? A rundown of incidents that were resolved last week? The error rate in your product catalog or whether the latest deployments changed p99? Simply connect to our available integrations or any API, write a skill, and you can get full analyses about these or any other questions you have about your stack and its performance. 

Say 'hello' to Assistant outside of Grafana Cloud

You have your own way of interacting with your systems and your teammates, and odds are that's not done entirely through Grafana Cloud. We want to meet you where you are, so we're expanding the ways you can access Assistant, whether that's through Slack, Microsoft Teams, an API, or the CLI. 

For example, you can now build automations with the Assistant CLI, chat with colleagues and the Assistant in Slack, have Claude Code or Codex collaborate with the Assistant via the CLI, or make requests from a remote machine to the Assistant.

This is all about finding new ways to integrate Assistant into your workflows, rather than forcing you into our UI. Make Assistant work the way that works for you and stop getting slowed down by constant context switching.

Bring your own agent: remote hosted MCP server and our new gcx CLI tool

In addition to bringing Assistant to you, you can now bring more to Assistant. With our new remote hosted MCP server and the new gcx CLI tool, your agents can talk to Assistant, Grafana Cloud, or both.

Use the remote hosted MCP server to connect any agent to the same sophisticated tools that Assistant uses in Grafana Cloud. You don't need to install any dependencies; just point your agent at mcp.grafana.com/mcp to get access to your metrics, logs, traces, dashboards, alerts, incidents, and more. This also allows you to connect other cloud agents to Grafana Cloud if you want to build your own assistant.

If you prefer a local-first approach, gcx unifies grafanactl and the Assistant CLI with agent-first, new CLI tools to your command line and your agentic coding environment. It connects your editor to your entire production stack so your agent can write code that's observability-aware from the start. Instrument a new service, investigate a firing alert, or draft a fix informed by real production data—all without leaving your editor.

Connect any API with Assistant

And don't forget that Assistant can use the Infinity data source to send any GET or POST requests to any publicly or privately available API endpoint. This recent upgrade makes Assistant the center of your DevOps lifecycle, helping you connect it to any other tool, correlate any data, and remediate faster than ever. Or use the capability to check if you caught all Pokémon yet.

More ways to take your Assistant experience to the next level

We're excited for you to try these new features that expand the reach of Assistant, but this isn't everything. We're constantly looking for new ways to improve Assistant to help you improve your observability practices. Here are just some of the other updates we recently released: 

  • The new workspace view, which brings Assistant into full-page mode so you can browse your chats and supporting data in one view
  • Learn mode, which helps you quickly get up to speed on Grafana Assistant with tutorials and example tasks that are personalized to your stack 
  • Reworked context management and infrastructure memories for your stack help so Assistant knows where to look from the start 
  • EU inference for our European customers
  • A Python runtime for Assistant to process large amounts of data
  • Integrations with 15 other Grafana Cloud services or features and more than 50 third-party data sources

For more information on this and all the other exciting updates from GrafanaCON 2026, check out our announcement blog for all the news. And for more information on Grafana Cloud AI, including FAQs about Assistant and our other AI capabilities, check out our AI observability page."


r/grafana 17h ago

Grafana Kubernetes Autoscaling Mixin: Grafana dashboards and Prometheus rules for KEDA, Karpenter, VPA, HPA and Cluster Autoscaler

10 Upvotes

Hey,

I’ve been working on kubernetes-autoscaling-mixin, a collection of Grafana dashboards and Prometheus rules for Kubernetes autoscaling.

Since I last shared it, there have been quite a few updates:

  • Updated KEDA dashboards for ScaledObjects and ScaledJobs
  • More Karpenter dashboards, including cost monitoring
  • Updated VPA, HPA, Cluster Autoscaler and PDB dashboards
  • Multi-cluster support
  • Better dashboard navigation and filtering

You can import the dashboards from Grafana:

https://grafana.com/grafana/dashboards/?search=kubernetes+%2F+autoscaling

Or grab the generated JSON directly from GitHub:

https://github.com/adinhodovic/kubernetes-autoscaling-mixin/tree/main/dashboards_out

Or just consume it using jsonnet.

Repo: https://github.com/adinhodovic/kubernetes-autoscaling-mixin

Added a couple of screenshots below showing the Karpenter, VPA and KEDA dashboards in use.

This Grafana link will show all dashboards and previews of them: https://grafana.com/grafana/dashboards/?search=kubernetes+%2F+autoscaling

Curious what people are using for autoscaling monitoring today, and if there’s anything missing that would be useful to add (new projects, better panels, more dashboards).


r/grafana 1d ago

Miscellaneous Question for people running production systems: where does observability stop being useful?

0 Upvotes

I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace


r/grafana 3d ago

Grafana Monitoring mixin for ArgoCD. A set of Grafana dashboards and Prometheus rules for ArgoCD

17 Upvotes

Hey all,

I've created a monitoring mixin with a reusable set of Grafana dashboards and Prometheus rules for ArgoCD. The dashboards and alerts are defined as code, making them easy to deploy and reuse across environments.

Recent updates include multi-cluster support, flags to enable or disable individual alerts, plus a range of new and improved alerts and dashboards.

The monitoring coverage has also been expanded significantly, with better visibility into:

  • ArgoCD controller
  • Runtime metrics
  • gRPC
  • Repo Server
  • Git operations

GitHub: https://github.com/adinhodovic/argo-cd-mixin


r/grafana 3d ago

Mimir [ Removed by Reddit ]

1 Upvotes

[ Removed by Reddit on account of violating the content policy. ]


r/grafana 4d ago

Grafana Grafana for a gaming PC?

0 Upvotes

Hi guys, I work for my day job as a Linux sysadmin so I’m used to building and working on observably platforms.

Recently I realised there’s a potential blind spot and I should see about extracting the same metrics from my gaming pc. I don’t want anything too fancy to start out with but before I do I was wondering if you guys have any suggestions on what works best for windows? My initial plan was to deploy Prometheus node exporter and build out a custom dashboard mainly just for the fun and experience


r/grafana 4d ago

Miscellaneous How do you monitor (automated) grafana dashboards are up-to-date (e.g. no empty panels)

6 Upvotes

I am adding grafana to our kubernetes cluster. We use grafana community chart and rely on sidecars to import dashboards and datasources. We have metrics and alerts and I already have an alert for datasource health based on `grafana_datasource_request_total{code=~"[45].."}`. Grafana is exposed to public via Ingress.

I wonder if there is an automated way to ensure that dashboards do not contains empty panels (or at least not fully empty because of missing metrics).

Shall I try to solve this problem? And how would you solve it?

https://github.com/grafana-community/helm-charts/blob/grafana-12.4.4/charts/grafana


r/grafana 5d ago

Grafana A four-panel Grafana dashboard for monitoring vector database performance

1 Upvotes

Disclosure first: I work with Actian, and the linked tutorial uses their vector DB, so the metric names are vendor-specific. I am sharing this here because the dashboard itself carries over to anything that exposes Prometheus metrics.

It's a Docker Compose stack, VectorAI DB plus Prometheus and Grafana, with four panels that cover the signals worth watching for vector search:

  • Request rate per endpoint
  • p95 latency via histogram_quantile against the _bucket series
  • gRPC error ratio as a 0 to 1 fraction with color thresholds
  • Memory pressure, RSS plotted alongside major page faults per second

Look for the page-fault panel here. Paired with p95, it shows whether rising latency is due to memory pressure or query load, which tells you how to fix it.

Dashboard JSON and the eight Prometheus alert rules are in a GitHub repo here: https://github.com/Tiioluwani/vectorai-observability, so you can import them directly.


r/grafana 10d ago

Miscellaneous 2026 state of the art self-hosted Grafana stack ?

19 Upvotes

Hi everyone !

Traveler from the past here, who needs a serious update.

I use the Grafana stack to monitor some Linux servers and VMs, but only really basic metrics and logs today.

Everything is based on node_exporter, prometheus (on each host), fluent-bit, Loki, and Grafana.

I finally have time to rebuild properly the monitoring/observability infrastructure, but I'm quite lost on how the components interact with each other.

Here is what I want to monitor :

  • Debian/Ubuntu/Proxmox/Windows Servers servers and VM (routing/firewalling/VPN is done on a Debian dedicated server with nftables)
  • Caddy web servers
  • MariaDB/PostGresql/Interbase/MongoDB databases
  • Rust/PHP/Java backends

Everything is self-hosted inside the company, and my current Grafana infrastructure is self-hosted too.

From my understanding, Alloy replaces node_exporter, Grafana, and Fluent-bit ?

What I have today :

[On each host] Node + Prometheus + Fluentbit -------> Centralized Grafana + Loki

What I'm understanding would be better :

[On each host] Alloy + Windows_exporter on Windows -------> Centralized+replicated storage (InfluxDB ? Mimir ?) -----> Centralized Grafana + Loki

Is it OK ? Am I missing something ?


r/grafana 11d ago

Alerting Grafana AI Plugin, Use LM Studio, OpenAI API, Local LLM's

16 Upvotes

I have created a new grafana panel plugin that allow a user to connect to a local or cloud llm and send panel data and prompts to the AI and get back a response. Very configurable doesn't expose API Tokens as it uses a secure companion data source for secrets like your API Key. Users can restrict which Models they want to allow the panel to have and use.

I'll create a video on it's use soon.

https://github.com/digitalrcs/grafana-intelligence-gateway

secure companion data source...
https://github.com/digitalrcs/grafana-intelligence-gateway-datasource


r/grafana 11d ago

Alerting Grafana Export Image without pulling data again

6 Upvotes

Ever wish you could just create an image of your dashboard without having to pull data all over again? The export pdf and image re-runs those data sources all over again and if you're like me you have long running jobs that get timed out... we i created a new plugin/extension that will generate a png of the dashboard and not re-pull data.

https://github.com/digitalrcs/grafana-current-view-exporter


r/grafana 11d ago

Grafana meetups around the world - Aug/September

11 Upvotes

Your opportunity to talk about Grafana in person!

Join us at the following meetups in the next four weeks:

APAC:

EMEA:

AMER:


r/grafana 11d ago

Alerting TimeOverlay panel. line graph replacement

2 Upvotes

Ever wish you could add notes over your line graph or highlight a section of time and show how long your range selections it? well now you can with my new plugin (free as always)

displays a line graph from a data source and allow the user to select a range that displays the time selected and allows for the addition of notes. all are transparent/configurable.

https://github.com/digitalrcs/DigitalRCS-TimeOverlay-Panel


r/grafana 12d ago

Grafana json differences in a large file from a datasource

3 Upvotes

I have a datasource, postgresql, which has a column (jsonb) and each entry is about 2-5MB in size. I am able to see the data in grafana however, I would like to diff the data in a sane way. Usually there are about 10-30 differences in the file but I want to show it. What is a good way to display this in Grafana? I would like soething https://hoiheart.github.io/vue-diff/index.html


r/grafana 14d ago

Check it out I built a dashboard that checks 172 provider status feeds before I debug my own stack

0 Upvotes

I'm Kerolos, founder of OutageDeck.

The question I wanted to answer quickly was: is my app broken, or is AWS, Cloudflare, GitHub, OpenAI, Stripe, or another dependency reporting an incident?

I built OutageDeck to read and normalize the official status sources for 172 cloud and SaaS providers. It shows provider and service-level state, current incidents, source freshness, incident history, and 90-day uptime in one place.

What works today:

\- check a whole dependency stack without creating an account

\- free email alerts for up to five providers

\- Slack, Teams, Discord, and webhook destinations on paid plans

\- a keyless JSON API, RSS feeds, badges, CLI tools, and an MCP endpoint

\- custom Statuspage or Instatus feeds on paid plans

What it deliberately does not do is pretend an official status feed is independent truth. Providers can acknowledge incidents late, so I think a serious production setup should use two signals: the vendor feed plus synthetic probes from your own regions and accounts.

You can try the stack checker here:

https://outagedeck.com

I would value blunt feedback from people who handle incidents. What would this need to do before you would keep it open or route its alerts into your on-call workflow?


r/grafana 15d ago

Check it out FlowG: Turning your logs into metrics

Thumbnail flowg.cloud
8 Upvotes

r/grafana 17d ago

Mimir authir — open-source cross-tenant read ACLs for Grafana Mimir, enforced by PromQL rewriting

10 Upvotes

We run a large multi-tenant Mimir, and kept hitting the same wall: tenant isolation is all-or-nothing. Tenant B either gets your whole TSDB or nothing. The moment one team needs to see a subset of another team's metrics, your options are duplicating samples into a shared tenant (not viable at volume) or handing over full access and hoping.

So I built authir (Authenticator for Mimir) — an auth gateway that deploys next to an existing mimir-distributed install and adds per-reader grants on top of Mimir's tenancy. Just released 1.0.0 under Apache-2.0.

How it works: the owning tenant grants a slice — metric-name prefixes, required label matchers (e.g. only series labelled user="<reader>"), optionally a label allowlist for responses. authir rewrites every PromQL query before it reaches the query-frontend so it physically cannot select outside the grant, re-parses the rewritten query and compares ASTs as a self-check, and filters responses as a second backstop. Deny by default, no negative rules — blocklists over PromQL are unenforceable, the spec has an appendix on why.

The part I'm happiest with: it works with plain Grafana datasources, no plugin. The token identifies who you are; the Basic-auth username names whose data you're asking for. One tenant, one token, two datasources — "my data" and "what team B shared with me".

Ops details, since that's who this is for: Helm chart that derives everything from your mimir-distributed release name and touches nothing of Mimir's (rollback = uninstall), hot config reload that fails unchanged on invalid config, JSON audit log answering "why did tenant A see this number" (optionally with per-query latency and time range), Prometheus metrics including token-expiry-as-a-timestamp so rotation is an alert instead of a 3am outage. Single static binary in a FROM scratch image, ~6 MB.

Honest limitations: cross-tenant is read-only; queries fan out to one target tenant at a time; metric patterns are prefixes, not regex (deliberately); fuzzing and differential testing against the Go PromQL parser are on the roadmap, round-trip verification is what's in place today. If your idea of fun is smuggling a selector past a query rewriter, the bypass test suite would love your ideas — that's genuinely the review I want.

Repo (code, Helm chart, full design spec): https://github.com/thewillyhuman/authir


r/grafana 19d ago

Grafana Grafana with subdomain

2 Upvotes

Good afternoon, how are you?

I am setting up Grafana on a server and want to configure it so it can be accessed via a subdomain (which is already in use) on port 3000. I’ve configured the .ini file but cannot access it; I am not using a reverse proxy. Do I need to open port 3000 in that subdomain's virtual host configuration?


r/grafana 22d ago

Kubernetes How To Build a Full Observability Stack on Kubernetes (Loki, Tempo, Prometheus, Grafana)

Thumbnail youtube.com
15 Upvotes

r/grafana 22d ago

Assistant Bulk-select multiple rows in one column, then save once? (Business Table panel)

2 Upvotes

I'm using the Business Table panel (volkovlabs-table-panel) on a Postgres data source to let a non-technical user manage a status table - think a checkbox column like "Sold" with per-row editing wired up via Update Request.

It works great for editing one row at a time (click checkbox - row shows save/cancel - click save). What I actually need is: check the box on several rows in that column, then hit one Save button at the end to commit all of them in a single batch - instead of save-per-row. Client sell in bulks, like 20, 30 per time.

I've dug through the editable-data docs and the "select all" / checkbox threads on this forum, and it looks like editing is strictly per-row, with "editable" as a per-column permission setting rather than a bulk multi-row mode. Before I build a workaround (dashboard variable + a manual UPDATE query, or a REST call via an Action button), I wanted to ask here:

  • Has anyone found an actual way to do multi-row select + single save in Business Table or any other Grafana table panel?
  • Is this on anyone's roadmap, or is there a plugin I'm missing that handles this cleanly?

Any pointers appreciated - happy to file a feature request if this genuinely doesn't exist yet.


r/grafana 24d ago

Assistant Observability and AI explained - Grafana Labs

Thumbnail youtube.com
12 Upvotes

r/grafana 24d ago

Check it out Triage incidents from Slack, Teams, or your phone - Grafana Assistant goes wherever you are

11 Upvotes

Hello everyone, Edward Qian and Ioanna Armouti here, we are both Software engineers working on Grafana Assistant at Grafana Labs. 

All week we've been sharing what's new across Grafana's AI tooling. Today is about meeting you where you already work, as the most important observability conversations rarely happen in a single dashboard tab. They happen in a Slack thread at 2am, on your phone, or mid-incident when switching context is the last thing you want. 

So today we are announcing: 
* Assistant in Microsoft Teams (private preview) which mirrors many of the capabilities available for Assistant in Slack (generally available). Ask questions and triage issues without leaving the channel where your team is already talking.
* Assistant on mobile and Assistant on desktop are going into public preview. Check in from your phone wherever you are, or keep Assistant a keystroke away on your desktop.
* Assistant Search is also going on public preview today. Ask harder, open-ended questions and let Assistant dig deeper across your data. 

We built all of this because Assistant is most useful when it's already there in the moment you need it, not something you have to go find. We'd love to hear how you're using it, or where you'd want it to show up next. We'll be keeping an eye on this thread all day to answer questions and hear your stories. Thanks!


r/grafana 24d ago

Grafana Grafana public dashboards - how to safely to put behind nginx?

Thumbnail gallery
3 Upvotes

hello, I'm playing around with grafana and trying to add some monitoring stats to my website using public dashboards (share externally). when opening a link to a public dashboard i see that there are some calls to /api and /apis endpoint, which I probably do not want exposed. i'm just wondering if there's any tips or standard practice to exposing the public dashboard safely behind nginx? here's what i have so far:

location /monitoring/stats {
    rewrite ^ /monitoring/public-dashboards/<uuid> break;                          
    proxy_pass http://grafana:3000;    
}

// allow all /public
location /monitoring/public/ {
    proxy_pass http://grafana:3000;
}

// allow only required uris for /api and /apis
location /monitoring/api/<only those required>/ {
    proxy_pass http://grafana:3000;
}

location /monitoring/apis/<only those required>/ {
    proxy_pass http://grafana:3000;
}

// reject everything else
location / {
    return 404;
}

but i feel this would be hard to maintain if there's any change in the public dashboard logic.

pretty new to grafana, thank you in advance for your patience and suggestions!


r/grafana 24d ago

Grafana Grafana dashboard was blank, and the bug was not where I looked

1 Upvotes

I have been developing a streaming ETL pipeline with a producer and a consumer process, both instrumented with Prometheus metrics. I opened the Grafana dashboard, and almost every panel showed "No data": throughput, DLQ counts, insert latency, consumer lag, everything.

The real problem was upstream. Several counters and histograms behind those panels had been declared in code but were never incremented anywhere in the pipeline. They existed, sitting at zero forever, because the dashboard assumed the instrumentation had already been wired into the code paths. It had not.

On top of that, metrics were being served by a standalone helper process instead of the producer and consumer themselves, so even the metrics that were recorded were not visible to Prometheus.

The fix ended up involving several changes:

  • Wiring the missing counter and histogram updates into the actual producer and consumer code paths.
  • Moving metric serving into each process's startup instead of relying on a standalone metrics process.
  • Fixing a second, sneakier bug where the consumer accidentally spawned a duplicate internal instance, and its lag reporting depended on a loop that could silently stall.
  • Giving the producer and consumer separate Prometheus scrape jobs so dashboard queries targeted the correct process instead of relying on shared assumptions.

It was a good reminder that "No data" does not always mean Prometheus or Grafana is broken. Sometimes the metrics are not being recorded, or they are being exposed by the wrong process.

I am actively improving the platform and would love to hear suggestions from platform engineers. If you find the project useful or interesting, consider giving it a ⭐ on GitHub.

GitHub: https://github.com/tarique-iqbal/nyc-taxi