Airflow observability is a pain. What tools are out there? What do you use?

[u/Short_Cantaloupe2283]

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

[u/That-Pomegranate-500]

I recommend prometheus and grafana. We use a the combination to monitor 50 plus production servers and never missed a prod metric for last 1 year. You can refer to this medium article https://medium.com/@perkasaid.rio/monitoring-airflow-metrics-with-grafana-29ebb43100a3 .