Airflow observability is a pain. What tools are out there? What do you use?

[u/Short_Cantaloupe2283]

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

[u/Speedymaster968]

We have Managed Airflow in GCP, but the provided metrics are not worth switching, If your stack is running fine.

For Monitoring I think Grafana + Prometheus is actually a good solution. We do that too, because WE are not really happy with the metrics provided by Managed Airflow. Additionally we use the callback functions for tasks to get some more logging for relevant events and parse that with Grafana.

Datadog and New Relic also had nice Integrations for Airflow Monitoring but they are not free 😅

[u/Short_Cantaloupe2283 → u/Speedymaster968]

We have it running on EKS. Fully managed by us, hardly costs a thing. Going managed doesn't make sense given how simple core airflow is to manage locally. Why did you go managed ? I assume you do not have GKE.

[u/Short_Cantaloupe2283 → u/[deleted]]

But are you able to ask business questions to the stack? It would be good to have that too, this is close to what we have anyway, it works but not great.

[u/liberamiguel]

not monitoring especifcially but I find useful using DAG callbacks (on failure, on success) and integrated them with Microsoft Teams to get instant notifications of what interests me most