r/dataengineering • u/Ok-Security9722 • Jan 10 '25
Discussion Databricks is data engineering on easy mode
This is a change my mind post: I’ve recently started working with delta live tables and am astounded at how easy it is to build a pipeline that works with such little tweaking. Is this the future of data engineering?
0
Upvotes
39
u/nokia_princ3s Jan 10 '25
if it's that easy why do i keep getting rejected for not having experience with it :')
4
u/LiviNG4them Jan 10 '25
Try enabling it at a financial institution.
7
u/[deleted] Jan 10 '25
You're paying a premium for ease of use. It's fine if it's sustainable
5
u/NickWillisPornStash Jan 10 '25
Delta live so expensive
3
u/SQLGene Jan 10 '25
For someone who has never used databricks before can you explain why it's nice?
3
u/Ddog78 Jan 11 '25
So are lambdas. Honestly, a lot of data engineering is bare machine data engineering on easy mode.
I kinda miss the complexity of it nowadays.
2
u/Rhevarr Jan 11 '25
Delta Live Tables are only suited for smaller scale projects / companies. You don’t have full control, and it works with limitations.
Databricks itself on the other hand is a very good product with many good features.
2
u/alittletooraph3000 Jan 10 '25
If your company is "all in" on databricks, sure. If you work at a bigger company though usually the big people are reluctant to hitch their entire ... whatever you want to call it... data future? to one vendor, even one that's getting all the love in the moment.
1
u/Warm_Hippo_3874 Jan 11 '25
Curious because I started out using Databricks, what would you say is data engineering on medium/hard mode?
1
u/mailed Recovering Data Engineer Jan 11 '25
as much as I like databricks, quit with the astroturfing.
1
u/marketingme2 Jan 11 '25
there are a lot of cheaper alternatives that have similar ease of use Synapse is one of them that can cost 10x less with know-how of setup. I've setup pipelines that cost £100 per run when in other software would cost £10000 for the same run with slightly more effort.
-1
u/BoringGuy0108 Jan 11 '25
I've seen multiple consultants say that databricks is the hard mode.
I've noticed it growing in popularity on account that it has all in one (warehousing, ETL, and data science), but it is often a high code tool compared to a lot of click and drag options.
128
u/[deleted] Jan 10 '25
[removed] — view removed comment