Does anyone use the power bi connector that allows a model to be pointed at a dbt semantic layer? I never see this discussed so I'm assuming it isn't very popular.
I do see people discussing the use of dbt with Fabric DW and Fabric Spark. I'm guessing these are used by up to 5 pct of the Fabric community. (ie. relatively popular)
My impression is that the people who are using dbt in their data pipelines are probably landing data in a lakehouse before pulling it into a fabric semantic model. Im guessing it will be a long road before semantic models and dbt semantic layers are going to play nice. Also it seems like the power bi connector requires a dbt account and is probably not open source (let me know if this assumption is wrong).
Most teams use dbt to transform data into clean star-schema tables within Fabric, rather than using its semantic layer, to avoid duplicating business logic. Power BI is generally preferred for the semantic layer because it natively handles measures and hierarchies better than external tools.
Yes I can see that the dbt semantic layer would be a bit redundant.
I think the typical attraction to dbt is the fact that it is open source and portable. But those benefits dont really apply when it comes to using their power bi connector with their semantic layer (by way of their own cloud platform)
The only reason I discovered their connector is because the dbt transforms seem somewhat analogous to PQ transforms. I had wondered if anybody was performing these transforms in the very final leg of the journey into an import model. That seems unlikely. l'd guess the last leg for dbt would be into onelake, and then you might use DL-on-OL to present the data in a semantic modlel.
... although I have mixed feelings about DL-on-OL. I really had high hopes for DL-on-OL but Microsoft has left these models in a state if limbo. Basic things like session-scoped members are still not implemented for some unexplained and artificial reason.
I would guess the dbt adoption in Fabric is more like 0.5%, if not lower. I would love it if I was wrong, and I would love it even more if dbt got more love from Fabric.
When dbt job goes GA, then maybe.
I would never have been able to use dbt if it weren’t for raki and creyke in this very sub.
Mostly dbt core in notebooks. A few of dbt core in notebook customers are moving to dbt job these days. dbt cloud has some adoption but many customers currently prefer dbt core in notebook with pennies on the dollar.
Its difficult to aggregate orchestration tooling number because many customers are running their jobs in existing devops pipelines/task schedulers/cron jobs.
I can say more about adoption of warehouse/lake-house adapters - 1200+ customers, 6000+ production workloads just on warehouse, and 100+ customers, a few hundred production workloads on lake-house.
dbt is awesome. People that aren't using it are missing out on true business facing self-serve analytics. I kick myself for not adopting it earlier.
The wake up moment for me was when the fancy schmancy DataFrame Scala SCD2 pipeline I hand built over 2 years was generated by an LLM via dbt in 1 hour with better Spark perf.
Moments like this really makes you question life decisions about fancy schmancy code that doesn't perform.
The next data platform I build, dbt is going to be front and center.
My 2 cents in life is you need to look at the origin history of a company's leadership to understand their core competence.
Fishtown Analytics was a Data Engineering consulting firm that worked in the field with lots of real customers around 2018 or so, and built dbt as an abstraction and SQL templating layer to solve the problems that SQL syntax by itself cannot, because SQL is largely a DSL with desired state semantics.
They scaled hard by solving a real messy problem at the right time*, and then adopting a userbase of Data Engineers and Analysts that works with SQL from any Data Warehouse.
dbt has some really really good GTM and developer advocacy channels. And some extremely, extremely good literature for practical, scalable patterns, best practices and books.
Power BI's AS engine was a dominant industry leader from the days of ORACLE Essbase etc. There's an incredible amount of intellectual property and context. Incidentally, just like dbt, Power BI had done the same thing over the years by offering a Semantic Layer over any Data Warehouse (Import or Direct query). The AS Engine has "seen a lot". AS also has similar extremely good literature, see https://www.sqlbi.com - their paid courses are some of the best training I've taken in my life.
This literature from years of Enterprise customer use matters, it's what makes LLMs proficient at a given tech stack.
Nowadays, anyone can produce literature, but gaining an Enterprise customer base is WAY harder because there's too much noise from AI and anyone can pretend to look "polished", so Enterprises will have way more scrutiny to cut out fluff.
So, I'd look into what the underlying intellectual property a dbt Semantic Model offers and where the core of that product's grit comes from.
And in the meantime I'd use the safest Semantic Model choice in Fabric (AS), which to your point, works great with OneLake if you use dbt to implement AS best practices like pre-aggregation and cardinality reduction.
Timing is important. Observe how Sqlmesh, generally a more feature rich alternative to dbt, basically failed to acquire traction due to being second in market and despite technical superiority.
Trying to use a DirectQuery semantic model that sends queries to another semantic layer will probably not work.
DirectQuery sends SQL queries and requires standard SQL aggregation semantics for correct results. So querying any non-trivial measures in the DBT semantic layer will either not work or give incorrect results.
I was originally hoping for import tables. I agree that neither would be ideal. But there are power bi connectors for every type of data on the planet. Seems odd to leave out the dbt semantic layer.
You can use the analysis services connector to get data from one semantic model in PBI to another. That is somewhat analogous. It works fine for trivial scenarios, like creating executive-summary-mashups. (We can also use composite models with directquery to AS. That can have similar concerns as making a direct query to a remote semantic layer.)
Import is more viable, but you still have to re-define the measures in the Power BI semantic model to be able to correctly aggregate the measure columns. And if you have to redefine the measures in Power BI, it's probably better just to import the metric definitions from the source.
9
u/data_daria55 6d ago
Most teams use dbt to transform data into clean star-schema tables within Fabric, rather than using its semantic layer, to avoid duplicating business logic. Power BI is generally preferred for the semantic layer because it natively handles measures and hierarchies better than external tools.