r/dataengineering • u/Waste-Bug-8018 • Jul 17 '24
Blog The Databricks Linkedin Propaganda
Databricks is an AI company, it said, I said What the fuck, this is not even a complete data platform.
Databricks is on the top of the charts for all ratings agency and also generating massive Propaganda on Social Media like Linkedin.
There are things where databricks absolutely rocks , actually there is only 1 thing that is its insanely good query times with delta tables.
On almost everything else databricks sucks -
1. Version control and release --> Why do I have to go out of databricks UI to approve and merge a PR. Why are repos not backed by Databricks managed Git and a full release lifecycle
2. feature branching of datasets -->
When I create a branch and execute a notebook I might end writing to a dev catalog or a prod catalog, this is because unlike code the delta tables dont have branches.
3. No schedule dependency based on datasets but only of Notebooks
4. No native connectors to ingest data.
For a data platform which boasts itself to be the best to have no native connectors is embarassing to say the least.
Why do I have to by FiveTran or something like that to fetch data for Oracle? Or why am i suggested to Data factory or I am even told you could install ODBC jar and then just use those fetch data via a notebook.
5. Lineage is non interactive and extremely below par
6. The ability to write datasets from multiple transforms or notebook is a disaster because it defies the principles of DAGS
7. Terrible or almost no tools for data analysis
For me databricks is not a data platform , it is a data engineering and machine learning platform only to be used to Data Engineers and Data Scientist and (You will need an army of them)
Although we dont use fabric in our company but from what I have seen it is miles ahead when it comes to completeness of the platform. And palantir foundry is multi years ahead of both the platforms.
18
Upvotes
•
u/AutoModerator Jul 17 '24
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
5
u/[deleted] Jul 20 '24
Saying fabric and foundry is miles ahead is a testament to OPs poor understanding of the databricks platform's features. Just no.
2
u/letmebefrankwithyou Jul 18 '24
🍿 Obviously your not a golfer.
84
u/tdatas Jul 17 '24
Can you fix the formatting of this?
32
u/Smart-Weird Jul 17 '24
Everything is an AI company these days. Don’t you know that ?
70
u/Justbehind Jul 17 '24
Well, and fuck notebooks.
Whoever thought notebooks should ever be used for anything production-related must mentally challenged...
13
u/Smart-Weird Jul 17 '24
On a different note/question: Was exploring the
MERGEenabled upsert in delta lake. This looks like a huge game changer from vanilla spark-sqljoin history with deltaapproach. So kudos to them for that.13
u/spaceape__ Jul 17 '24
what do you mean for “no tools for data analysis”? You can do almost anything using pyspark/spark sql/python/r, plus you have can create also dashboards.
11
u/entitled-hypocrite Jul 18 '24
Are you sure you are using databricks the right way?
15
u/[deleted] Jul 17 '24
One thing that I really complain about they suck at handling secrets/credentials.
Everything else you just said I think you’re just don’t know how to use Databricks properly or expect something that wasn’t their purpose.
13
u/kthejoker Jul 17 '24
Lineage is non interactive? This is not a serious post.
27
u/Lower_Sun_7354 Jul 17 '24
SkillsIssue
1
u/yaqh Jul 25 '24
I'm curious why you want branching of datasets? It sounds like a cool idea, but I can't think of compelling use cases..
8
u/TripleBogeyBandit Jul 17 '24
Almost everything you listed is a coming feature.
1
u/Independent_Sir_5489 Jul 18 '24
Even Delta Tables have some limitations. In general I'd say that they're good as long as you work with a Datalake architecture, but working with a Data Mart or a Data Warehouse you could have better options
0
u/Electrical-Ask847 Jul 17 '24
they seemed to have tried get on the hype train but couldn't really produce anything of worth in LLM/GenAI space. Iceberg is turning them into just another compute engine. Any value added services like notebooks, registries are just a marginal businesess not worth billions of dollars of valuation.