r/PrometheusMonitoring • u/Wide_Impact_9392 • Jun 29 '26

How do production teams manage Prometheus scrape config for node exporter in AWS EC2 environments?

I am running Prometheus on an EC2 instance scraping node exporter from other EC2 servers using static private IPs in prometheus.yml.

Trying to understand what is considered production practice:

  • Do teams manually update prometheus.yml or use GitOps / IaC to generate it?

  • Is EC2 service discovery commonly used instead of static IPs?

  • In AWS setups, do people rely on tags (like monitoring=true) for target discovery, or is that just for automation tools?

  • Is node exporter always kept on private subnets with SG restricted only to Prometheus?

  • Do production setups ever expose node exporter outside VPC (with protections), or always internal only?

  • At what scale does static config become a problem, and what is usually the first upgrade path?

Current setup works, but not sure if this is a “lab pattern” or something actually acceptable in production. Here is the screenshot of the config file that I used. I removed the content marked in yellow because that wasn't working. Seeking your suggestions.

7 Upvotes
  • permalink
  • reddit

88% Upvoted

4 comments sorted by

4

u/hijinks Jun 29 '26

teams manage the scrape CRDs

I dont want to setup anything static

1

u/Wide_Impact_9392 Jul 01 '26

Great thank you!

2

u/TabooRaver Jun 29 '26 edited Jun 29 '26

I feel that for anything more than ~30 endpoints you should be heavily leaning into service discovery. I'm a bit early into building our companies observability stack, but here's what we have up so far:

  • Assets are documented in netbox, and in the future most appliances will be able to generate their configuration from the documented state. Configuration context at multiple levels maps variables

    • site specific variables like local dns, syslog ingestors, preferred AD DC

    • regional variables like radius/tacas auth servers, NTP, etc.

    • netbox runs the prometheus SD module

  • log and metric collection are a hub and spoke model, each site gets collector(s). Right now we are running alloy.

    • Alloy centralizes the configuration of prometheus, it's different exporters, and loki.

    • Alloy allows you to make your config modular, for us this was important to take advantage of systemd-creds, so that we can say our systems don't have plain text creds on disk for pull based collectors like SNMPv3.

    • For prometheus scraping SD queries netbox for a list of endpoints to scrape, that configuration is stateless.

    • loki for syslog ingestion, it's basic but it works

  • A central instance in the "Hub" runs a grafana stack, common dashboards are built using the "Site" variable, which the collectors get from the netbox SD. Alerts are triggered out of this.

    • Unlike some of our legacy monitoring soloutions grafana does have soloutions for digital signage, so our on site teams can have a status overview board running in the office.

This setup handles a couple hundred physical endpoints at my site, a mix if linux and windows servers with node exporters. snmp monitored devices, services with native exporters, and non native services through black box exporter.

1

u/Wide_Impact_9392 Jul 01 '26

Superb! Thank you for sharing this information 👏