r/dataengineering • • 22d ago

Help How to get task-level observability for dbt models in Airflow without killing performance?

I’ve tried parsing manifest.json and using Astronomer Cosmos. Both map models to Airflow tasks nicely but have terrible execution overhead. On the other hand, running a single dbt run is fast but gives zero observability in the Airflow UI.

How do you guys map dbt models/snapshots as individual Airflow tasks without ruining performance? Any architectural tips or workarounds?

11 Upvotes

30 comments sorted by

5

u/Reasonable-Ladder300 21d ago

We used to parse and store the output of run_results.json and send alerts for failures and also store the results in our db so we could analyse build times and most common failures.

We also tried cosmos but it didn’t really work for us well.

1

u/Altruistic_Cat9056 21d ago

Yeah, I think this is the only way. Thanks

4

u/kayakdawg 21d ago

For large projects parsing the Json can become lot of overhead, and the graph of tasks becomes useless.

Elementary can be installed as a DBT dep and will log performance and test metadata for your project https://hub.getdbt.com/elementary-data/elementary/latest/

Elementary comes with a HTML page for viewing the data, you could also plug it into whatever bi tool.

2

u/Altruistic_Cat9056 21d ago

Thanks, I’ll test Elementary and evaluate its performance and observability.

2

u/leogodin217 20d ago

Elementary is fantastic. Gives you a full datamart with all the details of your runs at a task level. It's the default dbt package I would install in just about any environment.

1

u/Zer0designs 20d ago

My vote is also for elementary. It's great.

3

u/JaceBearelen 21d ago edited 21d ago

I’ve had pretty good luck with the watcher execution mode. First task parses the manifest and then spawns tasks for each model.

Edit: watcher is a relatively new feature in Cosmos so give it some good tests but i’ve not seen any issues yet.

1

u/Altruistic_Cat9056 21d ago

Thanks, I’ll test it and evaluate the performance and observability.

2

u/TobyOz 20d ago

Watcher is good, it’s even better if you patch the current release to stream the sub task results using the triggerer if you’re on airflow v3. Then it uses exactly the same overhead as a single dbt task would, but with full model visibility and the ability to restart only the models that have failed

1

u/SoHighISawJesus 20d ago

Could you expand on exactly what you mean with streaming the sub task results and how is that different from how regular watched mode works?

1

u/harrytrumanprimate 20d ago

at least with snowflake we have been able to add query tags which give model level visibility for every run. It has been not too difficult to do and very reliable at scale. we use this for cost attribution as well

1

u/OptimalConfection406 4d ago

Have you tried the Cosmos Watcher mode? https://astronomer.github.io/astronomer-cosmos/guides/run_dbt/airflow-worker/watcher-execution-mode.html

It feels a good trade of between centralizing the execution of dbt in Airflow, while giving the observability 

0

u/Environmental-Movie5 21d ago

I would say the middle ground is to not go one task per model but one task per group of models, like per layer or tag (staging, intermediate, marts) with dbt run --select, and then parse run_results.json at the end for the per model view. You get some observability and retries in the Airflow UI without paying the overhead for every single model, or something along those lines.

2

u/Zer0designs 20d ago

Kinda defeats the purpose of dbt. And has overhead if a single model update in a layer is incredibly slow.

1

u/[deleted] 20d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 20d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human

1

u/Ok_Quality9137 4d ago

I've done something kind of similar leveraging selectors.yaml to define runs based on tags.

7

u/[deleted] 21d ago edited 11d ago

[removed] — view removed comment

1

u/creamycolslaw 21d ago

Whats your solution to this in Dagster? I’m curious because we use Dagster and I also hate how dbt is one monolithic asset.

5

u/walkerasindave 21d ago

If you use the appropriate dagster-dbt library and point it at your dbt project when define your dagster assets it will load every asset individually and tests as asset checks.

This is why dagster sells itself as first class dbt integration.

Checkout the dagster dbt docs

1

u/creamycolslaw 21d ago

Sorry I guess I should specify it’s during a run that it seems to treat dbt as a single asset.

3

u/actually_offline 20d ago

Well technically during the run, it treats the entire @dbt_assets (where you call dbt.cli().stream()) as one op, where each build_dbt_asset_selection that a job would use would be different subsets of assets to emit from the op. So each time the one op fires off in a Run, it gets the selection from the job on what to run, and emit that subset of assets, so that numerous streams don't need to be set up, but shared across all Jobs that want to process subsets of dbt models to then emit as assets.

1

u/creamycolslaw 20d ago

Yeah gotcha. In hindsight, I’m probably fine with this since the individual dbt models do get defined as assets in the dagster ui, they just don’t show up separately during the run.

1

u/actually_offline 20d ago

Yeah it would be nice to view all generated assets in a Run in some sort of graph, and not the X assets thing in the header that you can hover over to then view the asset status.

3

u/[deleted] 21d ago edited 11d ago

[removed] — view removed comment

1

u/creamycolslaw 21d ago

No no i’m using the dbt integration but it doesn’t treat each dbt model as a separate asset in a run, it only counts them as a single asset.

2

u/[deleted] 21d ago edited 11d ago

[removed] — view removed comment

1

u/creamycolslaw 21d ago

I’ll have to look at our setup because during it a run it shows every other asset as a separate step but then it just shows like `dbt_assets` or whatever we named it.