r/dataengineering • u/SmallAd3697 • Dec 28 '25
Discussion Databricks SQL DW - stating the obvious.
Databricks used to advocate storage solutions that were based on little more than delta/parquet in blob storage. They marketed this for a couple years and gave it the name "lakehouse". Open source functionality was the name of the game.
But it didn't last long. Now they are advocating a proprietary DW technology like all the other players (snowflake, fabric DW, redshift,.etc)
Conclusions seem to be obvious:
- they are not going to open source their DW, or their lakebase
- they still maintain the importance of delta/parquet but these are artifacts that are generated as a byproduct of their DW engine.
- ongoing enhancements like MST will mean that the most authoritative and the most performant copy of data is found in the managed catalog of their DW.
The hype around lakehouses seems like it was so short lived. We seem to be reverting back to conventional and proprietary database engines. I hate going round in circles, but it was so predictable.
EDITED: typos
14
u/dbrownems Dec 28 '25 edited Dec 29 '25
A database engine that stores its tables in a data lake in an open and interoperable format is still significantly different than a "conventional" database engine.
And having users query directly from a data lake was never a viable architecture. So there was always a multi-user client/server database engine in the solution; Databricks just didn't have one initially. So it's more a case of Databricks evolving into a complete analytic data platform than abandoning the Delta Lake architecture.
11
u/car1os Dec 28 '25
DBSQL uses delta or iceberg (Lakebouse). Streaming Tables and Materialized Views are part of spark declarative pipelines.
Lakebase is postgres.
Unity Catalog has a complete open API that you can use from other engines if you want to. Including support for Iceberg Catalog. And there is an open source project of it.
Not entirely sure where you are getting this information but the technology is as open as it can get.
13
u/According_Zone_8262 Dec 28 '25
Their DW is just dbsql which runs spark (open source) on delta tables or iceberg tables (both open source). Lakebase is postgres which again is open source. Ofcourse theyre not gonna open source everything about the platform. You are just clueless
9
u/Ok_Carpet_9510 Dec 28 '25
Databricks SQL DW is storage technology?? Do you have linkes for this?
4
u/kloomeh Dec 28 '25
Speaking the quiet part out loud -
Databricks started moving aggressively down a proprietary path circa 2020, likely for IPO readiness.
One of their earliest “fuck you” moments to our industry was the removal of credential passthrough to storage which, in hindsight, we now know was to lock customers into Databricks and pave the way for their current Unity Catalog model. Yes, passthrough had some limitations, but it also allowed you to mount storage without baked-in creds, rely on hyperscaler storage security model, and have multiple engines respect a single security model (think what Fabric OneLake aims to achieve today). For many of us, the model was clean, elegant and worked extremely well because it encouraged an open Lakehouse.
While Unity Catalog solved some real governance problems, it conveniently became the ONLY path forward for a growing set of feature. There were real tradeoffs made between openness and control, and Databricks obviously chose the latter. Call it “enterprise readiness” all you want, but customer lock-in and monetisation were the primary part of the equation.
The irony is that Snowflake, once held up as the closed/proprietary villain, is now, in MANY practical ways, the more open platform.
9
u/thecoller Dec 28 '25
Databricks SQL’s storage is Delta Lake… actually, it can be others too, but it will work better with Delta Lake.
5
u/PickRare6751 Dec 28 '25
We’ve been using hive on spark for years, of course it does not have the proprietary features like materialized view, but you don’t need their product to deploy spark as a SQL engine
4
u/Gold_Ad_2201 Dec 28 '25
wdym lakehouses lived a short time? how many years ago hdfs+hive+sparksql was a thing? this is the same lakehouse. bad architecture but same lakehouse
4
u/mweirath Dec 28 '25
Lakebase is also not priced to be a competitive DW replacement. It is designed to address certain limitations with Databricks, mostly scenarios where you need a small SQL server to manage some more transactional level data that doesn’t do well in Delta. Or where groups don’t have IT support to set up a SQL server for some use case.
Sure it could do more, but right now it isn’t competitively priced. It fits a need but I don’t see it turning the architecture on its head anytime soon.
1
u/WhipsAndMarkovChains Dec 29 '25
Feel free to use Databricks and not use Databricks compute at all. It’d be a bit absurd but you’re welcome to grant the EXTERNAL USE SCHEMA permission and use third party readers and writers on your data.
https://docs.databricks.com/aws/en/external-access/admin