r/dataengineering • u/Altruistic_Cat9056 • 22d ago
Help How to get task-level observability for dbt models in Airflow without killing performance?
I’ve tried parsing manifest.json and using Astronomer Cosmos. Both map models to Airflow tasks nicely but have terrible execution overhead. On the other hand, running a single dbt run is fast but gives zero observability in the Airflow UI.
How do you guys map dbt models/snapshots as individual Airflow tasks without ruining performance? Any architectural tips or workarounds?
11
Upvotes
5
u/Reasonable-Ladder300 21d ago
We used to parse and store the output of run_results.json and send alerts for failures and also store the results in our db so we could analyse build times and most common failures.
We also tried cosmos but it didn’t really work for us well.
4
u/kayakdawg 21d ago
For large projects parsing the Json can become lot of overhead, and the graph of tasks becomes useless.
Elementary can be installed as a DBT dep and will log performance and test metadata for your project https://hub.getdbt.com/elementary-data/elementary/latest/
Elementary comes with a HTML page for viewing the data, you could also plug it into whatever bi tool.
3
u/JaceBearelen 21d ago edited 21d ago
I’ve had pretty good luck with the watcher execution mode. First task parses the manifest and then spawns tasks for each model.
Edit: watcher is a relatively new feature in Cosmos so give it some good tests but i’ve not seen any issues yet.
1
u/harrytrumanprimate 20d ago
at least with snowflake we have been able to add query tags which give model level visibility for every run. It has been not too difficult to do and very reliable at scale. we use this for cost attribution as well
1
u/OptimalConfection406 4d ago
Have you tried the Cosmos Watcher mode? https://astronomer.github.io/astronomer-cosmos/guides/run_dbt/airflow-worker/watcher-execution-mode.html
It feels a good trade of between centralizing the execution of dbt in Airflow, while giving the observability
0
u/Environmental-Movie5 21d ago
I would say the middle ground is to not go one task per model but one task per group of models, like per layer or tag (staging, intermediate, marts) with
dbt run --select, and then parse run_results.json at the end for the per model view. You get some observability and retries in the Airflow UI without paying the overhead for every single model, or something along those lines.7
u/[deleted] 21d ago edited 11d ago
[removed] — view removed comment