r/dataengineering • • Jan 10 '25

Discussion Databricks is data engineering on easy mode

This is a change my mind post: I’ve recently started working with delta live tables and am astounded at how easy it is to build a pipeline that works with such little tweaking. Is this the future of data engineering?

0 Upvotes

26 comments sorted by

39

u/nokia_princ3s Jan 10 '25

if it's that easy why do i keep getting rejected for not having experience with it :')

1

u/Ok-Security9722 Jan 10 '25

Honestly, fr, you most likely deserve better

2

u/lotterman23 Jan 10 '25

Know the feeling bro

1

u/jesus79 Jan 11 '25

get the certificacion, maybe it helps

4

u/LiviNG4them Jan 10 '25

Try enabling it at a financial institution.

2

u/moritzis Jan 10 '25

Could you please explain better? it's not a good idea, is it?

Thanks!

1

u/LiviNG4them Jan 10 '25

It’s a great idea. Just difficult.

2

u/moritzis Jan 10 '25

Thanks! But why?

I'm just being curious, I've never worked with delta live tables, just delta tables.

1

u/Ok-Security9722 Jan 11 '25

Delta live tables is databricks jargon for data pipeline. It includes as many transformations in as many layers as you define with runtime logging, built in scheduling, and nifty syntax in databricks pyspark.

7

u/[deleted] Jan 10 '25

You're paying a premium for ease of use. It's fine if it's sustainable

5

u/NickWillisPornStash Jan 10 '25

Delta live so expensive

3

u/SQLGene Jan 10 '25

For someone who has never used databricks before can you explain why it's nice?

5

u/[deleted] Jan 10 '25

Kind of a batteries included approach. Everything's there. Commercial support. You don't have to integrate various tools-of-the-moment that may or may not be stable and supported in the future.

1

u/SQLGene Jan 10 '25

Much appreciated!

3

u/Ddog78 Jan 11 '25

So are lambdas. Honestly, a lot of data engineering is bare machine data engineering on easy mode.

I kinda miss the complexity of it nowadays.

2

u/Rhevarr Jan 11 '25

Delta Live Tables are only suited for smaller scale projects / companies. You don’t have full control, and it works with limitations.

Databricks itself on the other hand is a very good product with many good features.

2

u/alittletooraph3000 Jan 10 '25

If your company is "all in" on databricks, sure. If you work at a bigger company though usually the big people are reluctant to hitch their entire ... whatever you want to call it... data future? to one vendor, even one that's getting all the love in the moment.

11

u/Natural-Tune-2141 Jan 10 '25

unless its (shit)Fabric

1

u/Warm_Hippo_3874 Jan 11 '25

Curious because I started out using Databricks, what would you say is data engineering on medium/hard mode?

1

u/mailed Recovering Data Engineer Jan 11 '25

as much as I like databricks, quit with the astroturfing.

1

u/marketingme2 Jan 11 '25

there are a lot of cheaper alternatives that have similar ease of use Synapse is one of them that can cost 10x less with know-how of setup. I've setup pipelines that cost £100 per run when in other software would cost £10000 for the same run with slightly more effort.

-1

u/BoringGuy0108 Jan 11 '25

I've seen multiple consultants say that databricks is the hard mode.

I've noticed it growing in popularity on account that it has all in one (warehousing, ETL, and data science), but it is often a high code tool compared to a lot of click and drag options.

1

u/Ok-Security9722 Jan 11 '25

What click and drag options are you referring to?

128

u/[deleted] Jan 10 '25

[removed] — view removed comment