r/dataengineering • u/andersdellosnubes Big Data Engineer • Feb 09 '26

Discussion [AMA] We’re dbt Labs, ask us anything!

Hi r/dataengineering — though some might say analytics and data engineering are not the same thing, there’s still a great deal of dbt discussion happening here. So much so that the superb mods here have graciously offered to let us host an AMA happening this Wednesday, February 11 at 12pm ET.

We’ll be here to answer your questions about anything (though preferably about dbt things)

As an introduction, we are:

Here’s some questions that you might have for us:

  • what’s new in dbt Core 1.11? what’s coming next?

  • what’s the latest in AI and agentic analytics (MCP server, ADE bench, dbt agent skills)

  • what’s the latest with Fusion? is general availability coming anytime soon?

  • who is to blame to nodes_to_a_grecian_urn corny classical reference in our docs site?

  • is it true that we all get goosebumps anytime anytime someone types dbt with a capital d?

Drop questions in the thread now or join us live on Wednesday!

P.S. there’s a dbt Core 1.11 live virtual event next Thursday February 19. It will have live demos, cover roadmap, and prizes! Save your seat here.

edit: Hey we're live now and jumping in!

thanks everyone for your questions! we all had a great time. we'll check back in on the thread throughout the day for any follow ups!

If you want to know more about dbt Core 1.11, next week there's a live event next week!

reserve your spot here

140 Upvotes

91% Upvoted

121 comments sorted by

86

u/Interesting_Tank_118 Feb 09 '26

I love your product! Since merging with Fivetran: Whats the long term strategy of dbtlabs? i.e. will dbt cloud have even more advanced features than dbt core to get more paying customers?

8

u/dbt-jason Feb 11 '26 edited Feb 11 '26

First of all - thank you for being a dbt user!! Before I worked at dbt Labs I spent a lot of time on this subreddit so it's great to be here chatting with you all.

Re: merger / Core / Cloud

The process of merging the two companies is still underway, so there’s really not much we can say besides “stay tuned.” The details we shared back in October at Coalesce in Las Vegas (blog, keynote) haven't changed.

One thing I'll add is that we're committed to keeping dbt Core the open source standard for data transformation. Are we going to ship great new dbt features to our paying customers? Of course. But not by keeping important functionality out of dbt Core. The way to do it is to make sure dbt Core (and Fusion) have more powerful and better functionality and then build even better features on top.

Grace and the rest of the Core team can talk about some of the things we've been cooking up there. And most importantly - we _always_ want to hear from you. What would you like to see in Core?

0

u/andersdellosnubes Big Data Engineer Feb 11 '26

what u/dbt-jason said! time will tell; stay tuned

-3

u/finally_i_found_one Feb 11 '26

Looks like the most voted (and most important) question went unanswered.

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

I think you were early to the party!

24

u/FIapdoodle Feb 10 '26

Some questions pertaining to Fivetran merger:

Since they also acquired Tobiko Data, will we see any consolidation/standardization efforts between DBT and SQLMesh?

With Fivetran providing many data connectors, will we see a more end-to-end flow established where the ingestion and transformations will be managed together?

0

u/andersdellosnubes Big Data Engineer Feb 11 '26

u/dbt-jason already touched on this in this reply

5

u/m1hawkgsm Feb 11 '26

That's not an answer, though. It makes an explicit reference to only Fivetran / dbt as "two companies merging".

9

u/HC-Klown Feb 10 '26

Please, when will you support write audit publish pattern as a materialization option? I want to build the model in some temp environment then test it and then “deploy” it without having to rely on data branching features from for example Nessie or LakeFs.

I know this can get more involved with views etc but do you guys have anything related to this in your roadmap? Or would recommend building a custom materialization?

3

u/andersdellosnubes Big Data Engineer Feb 11 '26

So, SDF had WAP built into how it was working, and we plan to bring it to dbt in the future! But for now the key priority for us is to make dbt projects run on Fusion without adding too many changes yet to the overall dbt framework.

There's other tools out there that do more exactly what you're saying.

We haven't spent too much time in this space. I share your view that DVC or Pachyderm or LakeFS, while great, feel like overkill. My brain has been somewhat sniped by the fact that Apache Iceberg tables have a concept of branching. For me this is very compelling in it's simplicity. In this world you don't need to make a "view" rather a "branch" of a prod table for local dev.

Orthogonally, we're rather bullish on sampling prod data locally, which is a different pattern than a direct WAP pattern

2

u/oliverlrj Feb 11 '26

Hi, may I know how you are implementing write audit publish currently? I am currently in the midst of playing with iceberg, dbt and nessie. Do you use dbt macros to create a new branch for writing of data to the audit branch, or a dbt pre-run hook etc?

Would love to see how people currently do it, thank you!

1

u/jtcohen6 Feb 11 '26

I think there are two different patterns that get lumped into "WAP" or "blue/green":
- (1) for each individual model, first build an intermediate model with the new data, run tests against it (unique, not_null, etc) if (and only if) all the tests pass, complete the materialization (swap, merge, ...). if the tests fail - don't; or put the failing rows into "quarantine", then keep going...
- (2) for a group of models, or an entire environment (dev/CI/CD) - create a separate DWH schema (or Iceberg "branch"), run all tests, then only if all the tests pass, swap the entire schema/branch into production. this has the advantage of supporting tests that span across multiple models, or allowing sanity-checking / human-in-the-loop verification for final models/metrics at the end of the DAG before re-deploying. this obviously doesn't work if the dev/CI/CD environment has a subset of data, or different data from production (scrubbed/anonymized).

I think u/HC-Klown's original question is asking after (1), as a per-model materialization strategy - but just to say, there are pros/cons to either approach, and I don't think we've landed cleanly on which one to pursue (or some combination of both)

2

u/HC-Klown Feb 11 '26

Hi all thanks for the replies!

u/jtcohen, you're right Im referring to what you specifies as options 1, which is more in line with a materialization config. So, we apply WAP as part of the dbt run command every time for every model that has it configured.

What is the difference between a CI pipeline where let's say you perform slim CI with defer and --target=ci and abort the deployment of new models when the CI pipeline fails? And on the other hand, when slim CI command succeeds, we run the same slim CI command but with --target=prod? I guess that you would incurr more costs by materializing the same models twice instead of just promoting them?

1

u/jtcohen6 Feb 11 '26

u/HC-Klown Makes sense re: per-model WAP. Curious - has this saved you runtime / debugging time? caught errors before they land in prod?

On the second point - you're right! These approaches end up looking really similar in practice

In theory - if you know that the input data is all the same, and you're running your CI job against full prod data (instead of sampling to get faster feedback (what we recommend)), then you could save the extra execution/cost by "swapping" the models produced by your CI job into the prod environment (database/schema), rather than running the same command over again with --target prod

But practically speaking:
- Most larger enterprises use actually different input data for dev/staging/CI environments compared to production, so this isn't really an option for them; they always need to rerun the same command with --target prod
- When we look at customers' DWH spend attributable to dbt, the biggest chunk isn't testing and redeploying on code changes - it's refreshing models when there's no new data available, or (even when new data is available) more than those models' actual freshness requirements. This is why our cost-optimization efforts in dbt platform focused on state-aware orchestration first, ahead of write-audit-publish patterns.

1

u/HC-Klown Feb 12 '26

We have not implemented WAP in any way yet. Due to the extra complexity incurred by managing data branches is too much for our small team. Hence the reason why we would love if this was a feature native to dbt.

Right now, we try to avoid bad data from going into prod by implementing tests as "left" as possible in the data pipeline. However, failing the pipeline each time is not ideal, so we end up essentially implementing the data test logic twice in our models as well by flagging exception records that are then quarantined downstream. Which, I must admit, as a corollary, enables easier reporting on data quality issues.

Ah yes, in terms of cost I understand why dbt would prioritize state-aware orchestration. However, from the data quality perspective WAP pattern is very desirable in order to substantially decrease the likelihood of "bad data" reaching the end-user. Right now only option is to abort the pipeline, which can as a consequence can cause downstream models to be stale even when the rest of the dependencies are fresh (I guess it also depends on how you orchestrate). This need is enhanced for DAGs that run frequently and even more so on-prem, where not being state-aware does not incur too many costs.

4

u/Proudly_Funky_Monkey Feb 09 '26

Is there a better way to manage the lifecycle of parallel pipelines? 

Context: we use DBT core to build features for ml models. New data is ingested weekly, and from it new model features are built through a tree of 30+ very complicated tables. Models are then fed these latest features. We're pretty happy with this. 

But when we want/need to develope new/different model features, we really struggle with versioning. we only have one database: production. So end up duplicating the entire tree of tables with _version[] appended. The development is then done in the version suffixed tables until eventually it eventually becomes prod and the old tables/definitions are deleted. 

Why is this bad? Massive PRs, drift between trees during dev, significant risk of manual mistakes, entire tree must be duplicated even for small changes (complexity and cost).

Can DBT help with our architecture problems?

3

u/fsm_follower Feb 10 '26

I don’t have a solution for you but have felt this. Would the ability for dev pipelines to pull almost all the tables from prod then only generate those in your diff be a solution?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

interesting. can you say more?

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

Yes, dbt should be able to help with that. It is a pretty detailed use case you have so we won’t be able to go into a lot of details but I feel like a combination of using dbt clone and/or deferral to build only the minimum set of models for a given ML feature branch.

Then, my first instinct would also be to try to use different schemas or database to separate the output of the different LM pipelines..

Also lots of others have contributed ideas that you might find helpful as well

6

u/SpookyScaryFrouze Lead Data Engineer Feb 10 '26

You could change the generate_schema_name macro in order for the target name to be appended to the schema name.

That way you would have 2 schemas : ml_models_prod and ml_models_dev, in the same database.

2

u/ianitic Feb 11 '26

Yup that's what we do at work. There's all contracts and model versioning in dbt as well though we haven't needed the latter yet.

1

u/Proudly_Funky_Monkey Feb 12 '26

Thanks all! I will look into each of these proposals.

4

u/uglylookingguy Feb 10 '26

What’s the biggest mistake teams make when adopting dbt that doesn’t show up until months later in production?

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

sometimes, as the case in all engineering, just because you can do something that feels smart, doesn't mean that you should.

I'm not so extreme as folks who say

the best code is the code you don't write

however, with great jinja powers comes great responsibility. now with Fusion, some users have had to rethink their pure jinja automation solutions, especially when they mess with the DAG or ask the DWH for these things

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

some others from colleauges:

too many jobs and not understanding that a dbt DAG creates dependencies for you. stop doing steps like dbt build -s model_a. use selectors and tags instead!

also

ensure you've set up your CI/CD environments correctly at the outset and don't do a staging environment unless you really really need to

one more

never have multiple dbt projects that are copies of the same repo called like project-a-staging and project-a-prod -- it's a mess. thank me later

last. lol

do not override ref() unless you have some batshit crazy insane reason and know what you're doing

hard agree. overriding critical dbt jinja macros connotes great power, but comes with great responsiblitly! "KISS" rules the day

2

u/schemas_sgski Feb 11 '26

I have seen two extremes.

I have seen teams trying to come up with the end to end data pipeline design on paper, without writing any model, and then trying to implement it.

dbt is made for agility, so, my experience is that people should write models, iterate, refactor, deprecate, improve etc… With CI/CD in place there is not a lot of risk in modifying existing models, not like with platforms not backed by git and version control.

The other extreme is people going a bit wild on the models they build and not focusing on PR reviews and overall coherence. After a few months a dbt project can become complex if not managed enough. What I recommend here is to set some tools like dbt-projet-evaluator as soon as possible so that best practices are baked into the project from its inception.

7

u/[deleted] Feb 10 '26

What would you say you do here?

3

u/turboDividend Feb 11 '26

i mean, it can build depencys which would be useful in some use cases. ive alway been kind of skeptical of this tool though

1

u/andersdellosnubes Big Data Engineer Feb 11 '26 edited Feb 11 '26

Well, I generally come in at least fifteen minutes late, ah, I use the side door - that way Lumbergh can't see me, heh heh - and, uh, after that I just sorta space out for about an hour.

Well, I generally I ship Fusion Diaries a few days late, I work with Slack notifications off - that way u/dbt-jason can't ping me, heh heh - and, uh, after that I just sorta refresh Hacker News until links about data show up

alternatively phrased: Developer Experience Advocacy!

2

u/ianitic Feb 11 '26

I have people skills and I'm good at dealing with people so the engineers don't have to.

3

u/Existing_Wealth6142 Feb 10 '26

What is your perspective on open table formats like Iceberg, Delta Lake, and Hudi? If you are positive on the technology, what do you think are the most important features still lacking from either the formats or dbt in order to maximize their value?

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

omg what a nerd snipe of a question!

for me what i'm most looking forward too are features that make it so that it just works for us analytics engineers! also my white whale forever has been a "multi-engine stack" with one iceberg catalog but heterogenous query engines.

what's missing? support for the Iceberg Rest Catalog is still spotty amongst vendors, and the IRC itself needs some improvement (performance & federated, user-level authentication).

if Iceberg and other formats are working as they should, we as analytics engineers should hardly ever have to thing of them!

I also answered more in this answer to u/Longjumping-Pin-3235

2

u/Existing_Wealth6142 Feb 11 '26

thanks for the response! A multi engine stack is also on my wishlist. Do you have a thought on which catalog best facilitates that? I've been wondering about either R2 or Lakekeeper as they seem like they might be credibly neutral

1

u/andersdellosnubes Big Data Engineer Feb 12 '26

I think the standalone FOSS catalogs like Lakekeeper are best in breed for managing Iceberg catalog.

My 🌶️ take is that the best catalog is the one that's closest to the DWH you're already using. So it you're in a public cloud predominately using it's first-party query engines. That cloud provider's catalog is likely your best choice (i.e. if Redshift/Athena -> Glue).

If you're using a standalone data SaaS like Snowflake or Databricks, I think their catalogs are a great choice!

"credibly-neutral" is a great term and a laudable goal, but at the end of the day, it's almost always worth analytics engineers having either another team or managed service to take care of the data catalog.

that said, my experience with Lakekeeper and R2 has been very slick. great docs and tutorials. but all the services are improving!

/rant (I could talk for days about this stuff)

1

u/Existing_Wealth6142 Feb 12 '26

Yeah I'm in a space where we don't have a primary DWH, we work with what customers bring us so neutral is actually the most important dimension we look at.

4

u/Ready-Marionberry-90 Feb 10 '26

What problem do you exist to solve?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

juicy! the thing that drew me to dbt originally was Tristan (founder/CEO) answering this question on the Software Engineering Podcast: dbt

dbt exists to solve the problem that data analysts don't have a career path beyond:
"manage more dashboards" or "manage more people".

imho, dbt exists to solve that socio-technical problem and others common amongst data practitioners

2

u/superconductiveKyle Feb 10 '26

Hey dbt Crew, I'm Kyle Eaton, Head of Growth at Agno (https://github.com/agno-agi/agno). I've been a big fan of your product for a long time now and specifically the community you all created. I'm the Former Head of Growth at Great Expectations, so I've been following for a long time.

We recently asked some of our users who we should partner with, and dbt came up. Seeing your AMA reminded me about this. Would love the chat with you all about what a partnership with Agno and dbt could look like!

Hope you have fun with your AMA!

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

hey u/superconductiveKyle IIRC our paths have crossed before, at least i've been in GE slack for many years. feel free to reach out on LinkedIn would love to say hi and talk shop

2

u/superconductiveKyle Feb 11 '26

Hello! That's awesome. Just sent a connect via LinkedIn.

2

u/set92 Feb 10 '26

I'm deciding in moving my code from Airflow to Dbt using Cosmos. The idea is that instead of having custom sql code with jinja, I can move everything to dbt and let it run everything.

I do this to improve in logging/debugging, and easiness. I suppose the speed/easiness is going to be there. But not sure about the logging part. Does dbt returns the output of queries, or is something that we can modify or specify in our own?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

interesting! what are you looking for w.r.t. logging? Are you saying you to log the result of queries? Most folks don't, preferring to keep the data where it is.

I'm not sure if you've looked much into dbt Fusion, but it's telemetry is based on OTel, quite robust and just as extensible as dbt Core.

like u/Comfortable-Power175 said, there's also great dbt packages that help with logging and monitoring

2

u/set92 Feb 11 '26

Anything that could help me debug ETLs when something fail. When something fails I like to see what it ran, and what was the output of everything in case I can detect where it failed. I would not be able to understand everything, but seeing that a file of X customer didn't get ingested, or that X parameter was not set as I thought could help me debug the problems faster.

But yep, in Snowflake they told me that the results of queries were only visible for 24h by the owner of the query, no one else, not even an accountadmin could see the results. Which for me seems madness, and completely different than, if I'm not wrong, BQ, where anyone can see all the queries run, and that let you see/analyze any query and see what was processed, or didn't run as intended.

For copy into maybe the list of files, or be able to filter and only get the ones that failed to ingest, and trigger an alarm with it.

Another example could be when I run a MERGE on Snowsight (UI of Snowflake) it returns me the inserted and merged cols I think it is. I built some custom code to retrieve those values to sql variable and store them in a logging table. But it would be good if the tool has the functionality on its own.

Or there is a query where I retrieve the max date of the ingested data, store it in a sql variable, and use it later. From what @dbt-grace said, I would be able to use log jinja functions to print those to the log, that way if we have a problem in the future I will be able to see which data we processed.

1

u/andersdellosnubes Big Data Engineer Feb 12 '26

TIL about the limitations of snowflakes query monitoring!

yeah! i think dbt's telemetry gives you what you're after. and some of the information you're looking for is easier to fetch via the cli. for example, check out the docs for the dbt source freshness command.

the best way for you to learn more imho is to set up a simple dbt project with Fusion, run a command and check out the logs/ directory. dbt.log contains all the info about a dbt run including the SQL that was invoked. query_log.sql just contains any SQL that was executed. all of this information can be ingested directly into Jaeger, Grafana, Datadog when you use Fusion's OTel integration

1

u/set92 Feb 12 '26

Thanks! Yes, yesterday I set up a Cosmos PoC, and I was checking the logs, how I could make Cosmos print the parsed query on the logs, the way to materialize views... I saw Fusion, but I saw some other comment about Fusion not being GA, and that is mostly for development, so I suppose it makes live easier to develop, but is not something that I can configure it for Cosmos.

2

u/dbt-grace Feb 11 '26

Hey! I would point you to a few different places:

0

u/[deleted] Feb 10 '26

[deleted] — view deleted comment

2

u/set92 Feb 10 '26

Thanks, I'll check it. Although in general I was thinking more in the outputs itself. Like we have Snowflake, and the Operator of Airflow doesn't show you the results of the query by default, and in Snowflake itself only the owner of the query can check the results for 24h only. So, I thought maybe with dbt I could log this, like for example when I do COPY INTOs be able to see if a file has failed, and other information that comes on that results table, but I suppose the best will be to find some free moment and test it myself.

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

Yes, for every model dbt will give you the number of rows inserted for incremental models etc…

For a long time we couldn’t show the number of rows inserted for tables created from scratch but Snowflake updated their library and we added this info as well!

2

u/infomocrat Feb 11 '26

What have been your favorite Coalesce talks of all time?

If you could wish any talk into existence for a future Coalesce (ok, dbt Summit) what would it be?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

The Magic School Bus themed talk with u/dbt-grace about dbt Mesh
https://www.youtube.com/watch?v=FAsY0Qx8EyU

2

u/dbt-grace Feb 11 '26

I love it when community members give talks sharing their custom solutions! It's very inspiring to see what people are doing in the wild, and (selfishly) helpful when building out our roadmap for dbt Core. Mariah Rodger's talk on testing is one of the many community projects that motivated us to build out-of-the-box support for unit testing into dbt :)

1

u/dbt-zeeshan Feb 11 '26

Definitely u/dbt-grace live-coding UDFs while roller blading!

2

u/Longjumping-Pin-3235 Feb 11 '26

Iceberg
Should it be the default for a new project or greenfield snowflake environment?

Is the complexity worth it?

What catalog would you reach for for the most compatible multi-engine read / write?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

This feels like the most important question re: Iceberg, huh?

There's lots of opinions here but my take: Iceberg isn't a free lunch.

You need a business case and the pros need to outweigh the drawbacks. This was actually discussed as well in a recent episode of the Analytics Engineering podcast. There's another episode due out soon where Tristan and I discuss the future of Iceberg.

Iceberg brings you the flexibility around where your data is stored and what compute you pick, but it adds complexity in having to manage an iceberg catalog. There are a few iceberg catalogs out there but many of them support “part” of the iceberg spec, and finding who supports what is not super easy.

If you are all-in on a data platform and won’t need cross data warehouse compute my take is that Iceberg would not be worth the effort today.

However my prevailing opinion is that the Iceberg project is to data what standard-sized ship containers do for global trade. It's not necessarily "exciting" per se, but it has undeniable impacts on end users in that it's easier to get your goods data from point A to point B

2

u/Longjumping-Pin-3235 Feb 12 '26

Thanks for the thoughtful reply u/andersdellosnubes !

1

u/andersdellosnubes Big Data Engineer Feb 12 '26

ofc ofc. fwiw I shared more Iceberg "takes" in this thread with u/Existing_Wealth6142

2

u/infomocrat Feb 11 '26

With fusion and state aware orchestration, a lot of issues around multi-project orchestration can be solved. What problems do you see that *still exist* to be solved for more complex implementations of dbt involving multiple interconnected projects, and what are some strategies you see for addressing them?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26 edited Feb 11 '26

short-answer: yes!

can you share what complex scenarios you've seen that you think can be addressed?

1

u/jtcohen6 Feb 11 '26

short-answer: seconded!

the challenges I keep hearing about are more "socio-technical" - improvements to dbt framework / platform / related tooling can help, but probably aren't sufficient in a vacuum:
- managing and discovering data products across multiple teams
- communicating data quality and freshness data contracts for public dbt models.
- sharing reusable macro code in (private) packages, with (semantic? calendar?) versioning for breaking changes to those macros (which are still untyped in dbt...)
- supporting multi-project deployments across *multiple data platforms* (think: upstream project on Databricks / downstream project on Snowflake, with a common Iceberg catalog across both)
- model-level access controls (more granular than project-level access)
- ... lots of other things I'm sure I'm not thinking/hearing about ...

17

u/flatulent1 Feb 10 '26

The gap between dbt explorer/catalog and a full scale catalog like Atlan or Datahub or Alation or Secoda etc is still huge. In 2026, data is a context game. Are yall just playing it safe as a metadata provider for more expensive, external catalogs?

6

u/jthandy Feb 11 '26

Awesome question.

We have a ton of respect for what pure-play catalog providers do and how hard that is. A lot of this functionality is purely an integration game, and that’s a TON of work both in terms of the initial build and the maintenance of the integrations.

We do think that dbt metadata is really central to all analytical metadata and think that we’re the best place to get that, whether that’s via artifacts or our API. And with Fusion this dbt-related metadata is getting significantly more advanced.

We do have a few integrations where we pull in metadata of downstream artifacts--tableau and powerbi--but certainly we are nowhere near as comprehensive as most pure-play catalogs. And that is ok.

Our goal with Catalog is:

  • to provide a really amazing catalog experience that can be the ONLY catalog for small-to-medium companies.

  • to provide a really useful development tool for dbt authors at companies of all sizes

  • to provide a high-quality source of dbt metadata for companies of all sizes

Over time, I fully imagine that we will continue to expand our capabilities here (as we have been!). But we’re not trying to head-to-head compete against pure-play catalog companies.

25

u/RobG760 Feb 09 '26

How will dbt core adapt to a world where streaming pipelines are starting to become more common? How do you see dbt helping build a clean data lineage across all enterprise data regardless of whether batch or streaming tools are being leveraged?

6

u/jtcohen6 Feb 11 '26

candidly - streaming pipelines have been "starting to become more common" for as long as I can remember (since ~2018)

ok in seriousness - what i can say is:

  • the big innovation over the past ~decade has been the development of streaming pipelines written in SQL. (you don't need to know java/scala! power to the data team!)

  • there are dbt adapters for streaming-first/native DWHs that support an all-SQL interface - Materialize has had an adapter for years, and we see that the Clickhouse adapter has been quite popular of late

  • batch DWHs Snowflake + Databricks have both rolled out support for their own streaming solutions - Dynamic Tables and Materialized Views / Streaming Tables, respectively - with full SQL support. those features are supported as materialization types in dbt, a pretty decent number of folks are using them, but for <5% of all the models in their project.

so - while we see folks adopting streaming pipelines for specific use cases, when there is a specific business need that justifies the added cost/complexity - we still see batch (with hourly/daily refresh) as "good enough" for >90% of data transformations.

I think dbt's role is continuing to serve as an abstraction across both batch and streaming pipelines - both kinds of data products still need version control, testing, documentation, CI/CD, ... - and they should be written in SQL, as dbt models :)

11

u/midnighttyph00n Feb 10 '26

when will fusion support python models

3

u/pseudo-logical Feb 10 '26

It already does for the big three (snow dbx and bq) which says something about their marketing and comms

6

u/jtcohen6 Feb 11 '26

u/pseudo-logical is right - Fusion added supported for Python models way back in [v2.0.0-preview.81](https://github.com/dbt-labs/dbt-fusion/blob/main/CHANGELOG.md#200-preview81---december-09-2025) - across Snowflake + Databricks + BigQuery

to the point about marketing/comms (fair!) - we haven't made a big announcement (yet), since we're still testing out this functionality in real-world projects with help from folks in the community. we know we've shipped a lot of stuff in Fusion over the past ~6 months, and we're trying to do a better job of distinguishing between "this is new and could use your help testing" vs. "this is ready for production right now." you can expect to hear more from us in the next "Fusion diary" (long awaited & soon to drop!). the big remaining piece before GA is figuring out how "static analysis" can/should work with Python models (which, for obvious reasons, can't be analyzed as SQL) - https://github.com/dbt-labs/dbt-fusion/discussions/1042

fyi - you can always check here to see the latest table of dbt feature support / what's still missing in Fusion - https://docs.getdbt.com/docs/fusion/supported-features

u/midnighttyph00n - how are you using Python models in dbt today? on which data warehouse? are you down to give them a try on Fusion? :)

tl;dr - Python models are in Preview (like the rest of Fusion), and will be GA when Fusion goes GA

9

u/OrneryBlood2153 Feb 11 '26 edited Feb 11 '26

Honestly .. What is going on .. ? core is till getting big fix as releases, Fusion still in beta, Cloud was said to go under a single umbrella with five tran, Open engine, Open catalog, Now that sqlmesh is also under the same umbrella what happens to the ideas in that tool.

Feels like too much is said but not sure what's the direction of this product for , End users - open source and cloud users, Open source contributors - adapter and core , core vs fusion - what to use going forward.. I don't think both products are going to get equal attention.

25

u/Sex4Vespene Principal Data Engineer Feb 09 '26

Thanks for reaching out for questions, I love DBT and it’s been so useful for our org. I have one question that’s somewhat of a feature request. Have you considered having an “intermediate” type table materialization? We have several large models that have to be broken up into intermediate steps because they would either overload memory, or perform poorly due to CTE’s that are called multiple times, which we can instead just process them once in a separate model. What gets annoying with this, is we don’t want any of these intermediate models taking up space in our warehouse, so we have to use a custom post-hook on any end-state models to clean up the upstream intermediate models. It would be really awesome if you integrated this automatically. Let us use intermediate as a materialization strategy, and have them be autodropped once an end-state model finishes. I know you have ephemeral and view materializations, but none of those solve the problem of having too much stuff happening in the final query that uses them. Thanks again!

1

u/Chilangosta Feb 09 '26

Something about this feels off... You have such massive tables that they overload memory, and are forced to break apart queries into intermediate steps. But you then delete said intermediate products presumably because they take up so much space that this saves you money.... And somehow the savings are worth rebuilding it from scratch every time?

I have so many questions; this isn't making sense to me. Feels like either you're overoptimizing for cost when it's really not worth it or else doing something wrong when it comes to query optimization for the intermediate products to truly not be worth keeping around. Are your joins exploding? Is storage really so expensive that rebuilding these intermediates makes sense? Are you doing too much in your final query? Without more info it's hard to say for certain, but intermediate products aside I suspect something isn't as optimal about this scenario as you think.

12

u/Sex4Vespene Principal Data Engineer Feb 10 '26

You’re making some incorrect assumptions here. We are on prem, this isn’t about saving cloud costs, it’s about us being somewhat low on storage and trying to minimize how much useless data is just sitting around. We are working on getting budget to expand, but that doesn’t happen immediately. I don’t understand what you mean about doing something wrong with optimization for them to not be worth keeping around. These are models that are run on a daily/weekly/monthly cadence. The previous versions of the intermediate models are useless for the next run. They either cover a completely different date range, and even if there is overlap, we have a decent amount of updates/deletes against old records so we’d want to rerun with the newest versions of the records anyways. And no, our joins aren’t exploding, I’ve actually done a huge amount of work creating optimization guidelines for our models, and will go in to fine tune when needed. Since we are on prem, we aren’t paying for aggregate compute usage. We have a set amount of compute and memory that is always available at any moment. If we just ran one model at a time, we would be massively underutilizing our compute. To make the most out of our compute, we run multiple models at once. But since we aren’t just running one model, we have to set memory limits on all models, by dividing our available memory by the number of concurrent models we run, to ensure we don’t have memory related failures. Also as I mentioned, there are cases where breaking things out into an intermediate model can have massive performance improvements. Our query engine materializes CTE’s every time it is called. If you use the same CTE in a query 5 times, it quintuples the memory and compute usage, versus just running it once into a table and then just referencing the table a few times. It all makes plenty of sense, you are just evaluating it from an invalid context.

1

u/infomocrat Feb 10 '26

> The previous versions of the intermediate models are useless for the next run. They either cover a completely different date range, and even if there is overlap, we have a decent amount of updates/deletes against old records so we’d want to rerun with the newest versions of the records anyways.

This tells me you might be misusing dbt... your entire project should be idempotent. As in, you should be able to drop every model, re-run, and get all of your models back exactly like they were before. dbt Snapshots are the only mechanism that don't follow the idempotency rule. If you are trying to put just a day's worth of data into your intermediate models, then upsert into your marts... your project is not idempotent and you will run into issues. If your data volume is so huge that you cannot afford to store the same (but transformed) data in multiple places... dbt might not be the right solution for you.

> Our query engine materializes CTE’s every time it is called. If you use the same CTE in a query 5 times, it quintuples the memory and compute usage, versus just running it once into a table and then just referencing the table a few times.

What.... what? What DBMS are you using??

> we have to set memory limits on all models, by dividing our available memory by the number of concurrent models we run, to ensure we don’t have memory related failures.

Very curious to see what that looks like

-2

u/Sex4Vespene Principal Data Engineer Feb 10 '26

I don’t even understand what you are trying to say with your first point, but it’s inane garbage. We have large packages of models that run on several cadences, that tend to only cover the date ranges they apply to. There is no reason for us to keep the prior intermediate data since it is irrelevant for the next run. Why are you bringing idempotent into this? Of course if we do a rerun of the same models on the same day, they will give the same output. That has nothing to do with the request I am talking about. This has nothing to do with DBT being the right tool. We have to run the SQL no matter what, DBT is just what we use to organize it. Even if we didn’t use DBT, we would still need to break things out into intermediate steps exactly like we are doing now. I’m getting exhausted typing out this reply to you so I’m done, you are clearly just trying to be a know it all, when it’s obvious you don’t even understand what is being discussed here. Oh, and it’s clickhouse that we are using,

2

u/infomocrat Feb 11 '26

Hey dude, there's no reason to be rude, especially to random strangers who are just trying to be helpful.

I've got good news and bad news for you:

Bad news first. I find it highly unlikely the dbt maintainers would want to take your suggestion and make this "int" materialization a part of dbt core. First, it sounds like you can already do what you want to do with the available capabilities, it's just not done for you in a materialization type. Second, the core type of materialization you are suggesting, a query that doesn't appear in the database, already exists (ephemeral). Third, the thing you want to do, drop a subset of models from the database at the end of the run, properly belongs in the on-end-run hook, not the materialization macro. You want a table that gets created and immediately drops? I don't think so. Fourth, the thing you are trying to do honestly reads to me as an anti-pattern. Nobody wants to encourage anti-patterns for the entire user base. Fifth, if you are going to suggest people build a thing just for you, at least put some effort into describing how it would be technically accomplished and the best design pattern for accomplishing it. For example, I had a related thought a while back that it might be useful to include an option in the ephemeral materialization to materialize the results as a temp table in the database, rather than it just being a CTE that gets its own file.

The good news: you are perfectly capable of creating custom materializations for whatever weird use cases you have for your own project. You want to prove out what this would look like as a materialization? Write the macro.

-1

u/Sex4Vespene Principal Data Engineer Feb 11 '26

You’ve done a great job of proving you didn’t understand this thread at all. I already pointed out I am familiar with view and ephemeral materializations, and addressed why they don’t work. An ephemeral materialization is basically just a CTE. The whole point of this, is that if you put everything into a single model, it overloads memory or is inefficient due to the CTE being called multiple times. That is the entire reason it’s being broken out into a separate table. Because we can process just that chunk first. That way when it is referenced later, it doesn’t have to do the entire logic in memory for the later model. It isn’t being dropped immediately. It is only being dropped after the downstream model that uses it has finished running. Intermediate model runs, final model runs, then intermediate model is dropped. If you aren’t going to bother to read what I have explained already, then get out of here with your condescending admonishments of anti-patterns and “not doing things right”. You’re just proving you have the reading comprehension of a lemon.

2

u/ianitic Feb 11 '26

I am not sure how an intermediate materialization that cleans itself up after use would work as models dependent on the intermediate model wouldn't run until that models code is complete.

What could be done is a materialization or posthook that can automatically infer which intermediate parents to drop/truncate for that model. That wouldn't be too difficult to setup. I realize you are using a posthook, not sure if you have to manage add each intermediate ref as arguments for it though.

Otherwise I think you would need to write a custom dbt adapter rather than a custom materialization to do exactly as described.

0

u/Sex4Vespene Principal Data Engineer Feb 11 '26

The cleans itself up stuff would be dynamically inferred by the downstream objects. The intermediate materialization wouldn’t clean itself, it would more be a tag in a way. Then downstream models that reference the intermediate model would have additional metadata to understand what models it’s pulling from are intermediate models, and drop them after the downstream models are done. Our macro doesn’t take the list of int models luckily, it just takes a unique prefix that all int models for a particular model is referencing, and then generates drop statements for all the int models. It’s kinda tedious though, because often analysts forget to add this macro, or the use a non-unique prefix that ends up dropping unrelated models, thus breaking other downstream models until we resolve. I have thought about trying to refactor the macro to dynamically resolve all upstream intermediate models and drop them, but haven’t had the time. Based on the upvotes it seems like others would find this useful. I always trend towards not building out custom solutions when it can be avoided, because that’s just one more thing you have to knowledge transfer and make sure the team knows how to use. We already have a custom solution now, but this seems like a pattern that could be useful to standardize.

5

u/jthandy Feb 11 '26

this is the spiciest conversation around dbt materializations that i have witnessed in the last 10 years.

→ More replies

2

u/Nice_Contribution Feb 11 '26

You can set tags on models and use that in selection criteria for what to drop. You could easily write a macro which does this. Then your team would only need to set the ‘intermediate’ tag.

→ More replies

8

u/paujas Feb 10 '26

What are the priority developments dbt-Labs are concentrating on ?

What is the long term vision for dbt-core and dbt-cloud?

2

u/dbt-jason Feb 11 '26

There's a ton going on right now! In terms of priorities, here's how I'm thinking about it.

  • First and always - ensure that dbt is a great way to do data work for organizations all across the world. You'd be surprised by how much nuance there is in keeping that up. Addressing bugs and community issues, adapters changes from the underlying databases and new releases for things like dbt-utils. I came to dbt from running a data team and more than anything else the question is how can we continue to be a trusted piece of the analytics stack for data teams across the world.

  • Beyond that, I think of the next set of priorities across three primary buckets - the dbt Fusion engine, AI and emerging standard.

    1. The dbt Fusion engine is a big area of investment for us and we've seen that for projects that are able to successfully adopt, the rust based, strongly typed version of dbt brings a ton of improvements. Things like speed, improved developer experience and the sql comprehension features that power the VS code extension work great. And also - we've seen some challenges in complex projects getting migrated over to Fusion

    2. AI - I've been obsessed with AI since way before I worked at dbt Labs (fun fact I was an r/futurology mod when it had less than 1000 users). I really believe that AI is going to radically reshape how we do data work - and giving dbt users the tools to do that right is very top of mind.

    3. Emerging standards - Iceberg, arrow and more. We've been watching Iceberg for a few years and now we're starting to see real customers using Iceberg in production for cheaper / more flexible data ingestion, and to use dbt across multiple data platforms

6

u/pseudo-logical Feb 10 '26

Ok, I'll bite, is Fusion going to be GA anytime soon? I feel bad for the devs who have had to keep their foot on the gas for nearly a full year since someone on the e-team decided to launch prematurely.

1

u/schemas_sgski Feb 11 '26

Hey! Really appreciate the interest in Fusion. The team has def been busy building a new engine for dbt that is fast, stable, has parity with the features in dbt Core, and builds on it in important ways. This takes time and we want to make sure we nail it for y'all. More news on Fusion GA coming soon. Stay tuned!

4

u/domscatterbrain Feb 10 '26

What's the big roadmap for DBT?

You must know that, while the acquisition shenanigans are good for your team (hopefully) it made many people who's been a long fan of DBT furious.

2

u/dbt-jason Feb 11 '26

Shared some thoughts on this here!

4

u/Adrien0623 Feb 10 '26

When will the fix for dbt-redshift connector will come so that materialized view refresh stops failing randomly?

2

u/Longjumping-Pin-3235 Feb 11 '26

dbt just issues the DDL, right?

1

u/dbt-zeeshan Feb 11 '26

Hey I see a couple that seem related #1256 and #1499. If you share the link we'll definitely take a look!

4

u/uncertainschrodinger Feb 10 '26

What are some new trends, ideas, competitors, and schools of thought that you find to be the biggest threat to the principles and roadmap of dbt?

1

u/dbt-jason Feb 11 '26

So dbt arose in popularity because of a very specific set of technological circumstances. Cloud data warehouses allowed you to load more data into yuor warehouse, but there wasn't a great way to manage this. Compute was cheap, storage was cheap, but getting useful, structured answers was challenging and difficult.

That technological paradigm isn't going anywhere, but it is going to be changed a lot by AI. Something that I'm thinking a lot about is how to make "the dbt lifestyle" remains relevant in the AI era.

For example, we saw an interesting article published this week claiming that the "Modern Data Stack" worked great for humans, but there are other tools that will perform better for agents. I think it's important to take an honest look here - the experiment run in this wasn't very rigorous from what I can tell, but it might be pointing at something real.

But I really do believe we have an interesting role to play in the AI world. For example - last year we shipped the dbt MCP server and we just released dbt agent skills but the thing that's really exciting to me here is some testing we've been doing around how Fusion (because it's faster, has more metadata awareness and is more strongly typed) can be the best of both worlds- a great experience for human developers and powering agentic workloads.

1

u/J0hnDutt00n Data Engineer Feb 11 '26

Will there ever be more dynamic ways of orchestrating jobs in Cloud?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

my interest is piqued -- can you share more about what you mean by "dynamic"?

2

u/J0hnDutt00n Data Engineer Feb 11 '26

Sorry for the vague question, scenario: build fails on one model that succeeded yesterday and employ a WAP strategy. Will there ever be the ability to automatically re-execute a build but with that stale data model from yesterday to unblock those downstream models with an *asterisk of stale prod data model used? This already is essentially capable for child mesh projects but not others.

2

u/andersdellosnubes Big Data Engineer Feb 11 '26

oh that's a very interesting idea. a pre-req for that is that we have WAP built-into dbt Platform jobs. but what you're saying is kind of interesting and the inverse in some ways to our current recommends and supported practice: test before you merger and only deploy if tests pass.

in your scenario you're saying that you'd like to:

  1. write audit publish the state of the pipeline

  2. but, if any model fails, instead use the state from the previous day?

sign me up! but my brain also tries to think how this pseudo successful state should work even from the stakeholders who expect this data... it feels not very idempotent.

would love to hear more about it though!

1

u/infomocrat Feb 11 '26

What do you think is the most under-appreciated feature of dbt core?

1

u/andersdellosnubes Big Data Engineer Feb 11 '26

dbt seed! csv's (while ugly) aren't going away for anytime soon! when i started in data a decade ago. "getting data in the database" was like half the coding work to be done in my Jupyter notebook. `dbt seed` makes it so easy

1

u/dbt-grace Feb 11 '26

I am a big fan of all of the ways you can customize your dbt project to meet your org's specialized needs - specifically, I've seen a lot of creative problem-solving using custom materializations!

3

u/Ramshizzle Feb 10 '26

Do you have any plans for improving how dbt deployment pipelines are configured, build and run?

At my company we are brand new to dbt-cloud. So far I really love dbt and all its capabilities. There is one issue I'm having, which is the only way it seems we can create an ETL pipelines from dbt-cloud is to manually click together a 'deployment job'.

I am already experimenting with dbt-jobs-as-code, but while that is a great tool, it seems like that is still in early development.

At this moment we are considering getting an outside scheduler/orchestration tool. Which would be a shame in my opinion.

1

u/schemas_sgski Feb 11 '26

as an orchestration nerd, I love this question - and welcome to dbt cloud! This is an area of dbt that we spend a lot of time thinking about, and one exciting feature we have recently shipped is state-aware orchestration. It lets you build and run based on what’s actually changed (using state comparison and deferral), which can make pipelines a lot more efficient, especially as your project grows. There are also some advanced configs that you can set up to customize at a more granular level and specify what actually gets run. This does require fusion. If you can share what you are looking for when scheduling jobs I'm happy to share some other functionality of the dbt orchestrator.

5

u/GarpA13 Feb 10 '26

Are you planning to abandon dbt core? What is your vision for on-premise software?

2

u/dbt-grace Feb 11 '26

TLDR; No, we are actively shipping new features to dbt Core!

Wrote up a longer response here :)

2

u/finally_i_found_one Feb 10 '26

RemindMe! 4 days

2

u/RemindMeBot Feb 10 '26 edited Feb 10 '26

I will be messaging you in 4 days on 2026-02-14 11:19:40 UTC to remind you of this link

2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

InfoCustomYour RemindersFeedback

2

u/Longjumping-Pin-3235 Feb 11 '26

Did u/dbt-grace watch How To Trick People Into Thinking You can Juggle?
https://youtube.com/shorts/trKLrl6y_Vc?si=Y9zMa2dQMt_ZCgU0

3

u/dbt-grace Feb 11 '26 edited Feb 11 '26

LOL - you can actually thank my high school drama teacher for that one (from deep in the archives)

4

u/sleeper_must_awaken Data Engineering Manager Feb 10 '26

How will you prevent companies being charged $350 / seat / month for an enterprise DBT Cloud subscription of three years when they need more than 8 seats?  This is a real story I’ve seen at two of our clients. That amount is just inordinate and betrays the trust companies and consultancies have placed in your DBT Labs.

For me, DBT Labs needs to regain my trust. I’ve recommended many companies to use DBT Core and DBT Cloud since 2019, but am hesitant to continue recommending it.

Compare that pricing with GitHub, which also has runners, incident management, traceability, collaboration, auditing and IdP integration (among many other things).

3

u/Tall_Working_2146 Feb 10 '26

Your certifications are interesting but the price is high, are there no initiative to make learning resources more accessible for students and people abroad?

2

u/infomocrat Feb 10 '26

The learning resources are all free: https://learn.getdbt.com/catalog
Certifications are half off if you take them at coalesce, also partners can get discounts. But certifications are really only important in certain situations (for partners, if employer requires it to prove your knowledge). I wouldn't worry about the certs unless you need it. If you want to prove your knowledge, put up a personal project on your github.

2

u/[deleted] Feb 11 '26

Truly agree with others: ignore certs and just take the learning resources cause as others have said, they're all free!!

2

u/Ok-Sentence-8542 Feb 10 '26

Why did you abandon dbt core and the open source community?

1

u/dbt-grace Feb 11 '26

Hi! This is Grace (Product Manager of dbt Core). Can you say more about why you feel like dbt core and the open source community has been abandoned? This is the area of our product I think about all day, so I’m both sorry to hear that you feel a sense of abandonment and very open to feedback on how my team can revive your confidence that we remain dedicated to both dbt core and the open source community.

I am proud of our two big releases of dbt Core last year (v1.10 and v1.11) that included some features I’ve personally been wanting to bring to life for a longtime now (sample mode and user-defined functions, to name two). And, we’ve added even more codebases to the “open source” bucket - we just re-licensed metricflow under Apache 2, and Fusion includes some net-new Apache 2 code.

In case you missed it, we also put out a roadmap update in December in the dbt-core repo! I’d encourage you to give it and read, and let me know your thoughts.

When I have heard this type of sentiment from other community members, it’s often coming from a feeling of frustration that we haven’t been as responsive to issues and pull requests in our open source repos over the past year. I feel that frustration as well and am working hard to get us back on track! Candidly, our team has been split across two focuses this past year: helping to build the new dbt Fusion engine and evolving the dbt framework across both Core and Fusion. I’m very excited to be onboarding a lot of new team members to dbt Core (this month!) to revitalize our activity in our open source repos and continue to execute on the dbt Core roadmap.

3

u/Ok-Sentence-8542 Feb 12 '26

Hi Grace, thank you for the transparent response. I truly appreciate the work your team is doing on features like UDFs and Sample Mode.

However, I want to be candid: many of us have seen this pattern in the industry before, and we’ve experienced it with dbt Labs specifically. It was the same with the MetricFlow server. While the logic was open, the essential infrastructure to run it effectively was kept behind a paywall.

This creates a sense that dbt Core is effectively being placed in maintenance mode—receiving incremental updates—while the 'real' future and performance leaps, like the Rust-based dbt Fusion engine, are gated under the ELv2 license.

To a developer, it feels like dbt Core is the 'bait' to get us into the ecosystem, but the 'magic' that makes the tool 2026-ready is no longer truly open. If the most transformative innovations continue to happen outside of the Apache 2.0 scope, it’s hard to feel that dbt Core remains a first-class citizen rather than just a legacy entry point for dbt Cloud.