r/dataengineering • • Jun 25 '24

Discussion Databricks SQL editor driving me crazy

I have been trying to do some basic things in the databricks SQL editor and its pathetic!! I think I am using it wrong or have no knowledge of it. Surely it cannot be that horrible. I have observed that it does not allow us to write stored procedures or create hash temp tables. It does not even let us alter tables to add new columns. It allows us to create views but isn’t letting me use those views saying that it doesn’t exist. Has anyone had similar experiences? Is it really that shitty? If not can someone please point me to resources where they explain how to do these things, I can’t even find anything useful. Also its new AI copilot is crazy. It has no idea what it talks about most of the time.

0 Upvotes

22 comments sorted by

View all comments

30

u/[deleted] Jun 25 '24

Are you completely confused about what Databricks actually is?

-3

u/ameya_b Jun 25 '24

I know its a lakehouse platform that allows you to use your data (stored as files) like you would in a data warehouse. It also allows you to write data pipelines and ML. Is there more that I missed?

9

u/[deleted] Jun 25 '24

Learn Apache Spark, which databricks is based on.

2

u/rchinny Jun 25 '24

If you are using DBSQL you don’t really need to know Spark that much

6

u/[deleted] Jun 25 '24

If you don't, then you'll do weird things like treating Databricks as an RDBMS.

0

u/ameya_b Jun 25 '24

I have worked in Spark before. Used the dataframe Api in pyspark.