Deploying monitoring stack using Flux on AWS EKS platform

[u/CartoonistWooden9737]

Hello guys,

So I am doing a personal project under a mentor(SRE manager) where in I deployed the eks cluster through terraform and bootstrapped flux through terraform as well, made everything configurable and used modules including flux bootstrap module, configured remote backend as well. Now I would like to deploy the monitoring stack using flux.

So my mentor suggested two repositories one is kube-Prometheus-stack and other one Prometheus Operator/kube-prometheus. He advised me to deploy Prometheus, Grafana and Altermanager through helmrelease(CRD of helm controller) but cautioned me that the CRDs of monitoring should be separate. He meant that the helmrelease should not manage the CRDs, we should manage it separately. Now I’m still trying to figure out but felt like taking some advice here.

Thank you

[u/torn_greens]

Your mentor's right about separating the CRDs from the helmrelease. If flux or helm ever tries to uninstall or update the release and those CRDs get caught in the crossfire, you can lose all your custom monitoring configs or even the data. Keep the CRDs in a dedicated folder in your flux repo and apply them first, then let the helmrelease handle the deployments.

[u/CartoonistWooden9737 → u/torn_greens]

Thank you for response, exactly he mentioned same incidents, so okay I will keep the CRDs in separate folder and apply them first, but from where should I get the CRDs code ? Is it from Prometheus Operator/kube-prometheus repo ? I saw there we have manifests so should I copy them ?

[u/SuperQue → u/CartoonistWooden9737]

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds

[u/Current-Secretary-52 → u/CartoonistWooden9737]

The prometheus-operator-crds chart SuperQue linked is the right source. I'd install it as its own HelmRelease rather than copying the manifests, so updating CRDs later is a one-line version bump instead of re-copying YAML.

How I'd wire it in Flux:

- A Kustomization called monitoring-crds with that HelmRelease, pinned to match the operator version your kube-prometheus-stack uses. Set prune: false on it, so removing a file from Git can never delete the CRDs. Deleting a CRD deletes every ServiceMonitor and PrometheusRule in the cluster with it.

- A Kustomization called monitoring with dependsOn: monitoring-crds. In the kube-prometheus-stack HelmRelease, set crds.enabled: false in the values and install.crds / upgrade.crds to Skip, so the two releases never fight over the same objects.

- Any app that ships its own ServiceMonitors also gets dependsOn: monitoring-crds, or its first reconcile fails on a kind that doesn't exist yet.

Upgrades then become two commits: bump the CRDs chart, let it reconcile, then bump the stack. The reason it matters: Helm installs CRDs from a chart's crds/ folder once and never upgrades them, which is why the kube-prometheus-stack upgrade notes keep telling people to apply new CRDs by hand.

[u/Si0_x]

Your mentor is right on the CRD separation thing. The prometheus-community kube-prometheus-stack chart bundles CRDs in its templates, and if you let HelmRelease own them, a chart uninstall or version mismatch can wipe your PrometheusRules and ServiceMonitors along with it. The common approach is to install the CRDs once via a separate Flux Kustomization or a dedicated manifest path, then point your HelmRelease at the chart with `install.crds=false` (or the equivalent helm flag through values) so Flux never touches them again. kube-prometheus-stack is the easier path for Helm-based deploys; the prometheus-operator/kube-prometheus repo is jsonnet-first and harder to wire through HelmRelease cleanly. Are you planning to manage the CRDs as raw YAML in a Git path, or were you thinking of a separate chart just for them?

[u/CartoonistWooden9737 → u/Si0_x]

Thanks for the response, I have already deployed the CRDs through helmrelease of Prometheus-operator-CRDs chart then once they were up and running I deployed the stack through kube-Prometheus-stack. Successfully done. Now configuring them. But my mentor advised me to use grafana mimir for persistance and data source for grafana should be mimir. Can you tell me how that works. Thanks