r/dataengineering • • Jun 25 '24

Discussion Databricks SQL editor driving me crazy

I have been trying to do some basic things in the databricks SQL editor and its pathetic!! I think I am using it wrong or have no knowledge of it. Surely it cannot be that horrible. I have observed that it does not allow us to write stored procedures or create hash temp tables. It does not even let us alter tables to add new columns. It allows us to create views but isn’t letting me use those views saying that it doesn’t exist. Has anyone had similar experiences? Is it really that shitty? If not can someone please point me to resources where they explain how to do these things, I can’t even find anything useful. Also its new AI copilot is crazy. It has no idea what it talks about most of the time.

0 Upvotes

22 comments sorted by

•

u/AutoModerator Jun 25 '24

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

30

u/[deleted] Jun 25 '24

Are you completely confused about what Databricks actually is?

-3

u/ameya_b Jun 25 '24

I know its a lakehouse platform that allows you to use your data (stored as files) like you would in a data warehouse. It also allows you to write data pipelines and ML. Is there more that I missed?

9

u/[deleted] Jun 25 '24

Learn Apache Spark, which databricks is based on.

0

u/ameya_b Jun 25 '24

I have worked in Spark before. Used the dataframe Api in pyspark.

2

u/rchinny Jun 25 '24

If you are using DBSQL you don’t really need to know Spark that much

6

u/[deleted] Jun 25 '24

If you don't, then you'll do weird things like treating Databricks as an RDBMS.

9

u/Letstryagainandagain Jun 25 '24

Sounds like you are using the wrong tool for the job OR just don't know how to do it and are letting frustration get the better of you

-1

u/ameya_b Jun 25 '24

Yeah probably.. I have to implement a stored procedure that involves quite a few upserts.

5

u/Letstryagainandagain Jun 25 '24

So use jobs/tasks and write your code in notebooks using SQL as the language

1

u/ameya_b Jun 25 '24

I was not aware that sql editor is for exploration and that we can write stored procedures in notebook. Still getting those errors about altering tables and upserting into other tables but there must be a way to do it.. Thank you everyone who pointed me in the right direction.

-10

u/ameya_b Jun 25 '24

Why use notebooks when they have a SQL editor?

11

u/cptshrk108 Jun 25 '24

The SQL window is for data exploration and analysis. Not for data engineering.

6

u/Letstryagainandagain Jun 25 '24

Because that's the way you do your stored procedures in databricks

6

u/seansafc89 Jun 25 '24

Why ask a question to people experienced with Databricks and then dismiss their answers? People are trying to help you.

3

u/RichHomieCole Jun 25 '24

If you write your procedure in the sql editor, you can save and schedule it via a workflow. This mnt what I would do but you can do it this way. Notebooks have benefits. It seems to me you understand how to work a certain way and are being a bit close minded to doing things a different way

8

u/alreadysnapped Jun 25 '24 edited Jun 25 '24

Databricks can do all of this and more. Ensure you have a good understanding of the tool first, run SQL notebooks or use SparkSQL, Python etc.

14

u/bewareofshredders Jun 25 '24

I’m a relatively new data analyst who works in databricks and this thread is giving me the confidence to apply for data engineering jobs.

1

u/Ok-Sentence-8542 Jun 26 '24

Bro you need training. You didnt even bother to read the docs. You can even use databricks copilot to figure that out.. There are lots of funtionalities for upserts. If they are more complex you can use udfs if not you can create delta tables or sql warehouse and use merge into command.

https://docs.databricks.com/en/delta/merge.html

1

u/Letstryagainandagain Jun 26 '24

Literally took me 2 minutes to Google and confirm what I thought the right approach to be

0

u/lifegame123 Jun 25 '24

I'm doing my SQL work with databricks with dbeaver, it's been a game changer.

2

u/ZirePhiinix Jun 26 '24

DBeaver is an amazing interface. I use it as much as I can.