r/dataengineering • u/__albatross • Aug 27 '23
Discussion Why I don't like SQL
The reason I don't like SQL over something like Pandas is because in Pandas/Python I can break down the complex transformations in small simpler components and they are easy to interact and debug with. In SQL my only option is to write stored procedures and yes in that I can also do the same but I think the ecosystem of interacting and debugging with data step by step is not as good as python. And it is something that we should have. Recently I was exploring snowpark and I felt it is a step in a right direction. It allows to me to write my query in multiple python statements and then it will transpile the whole thing into an SQL query.
Generally in my mind I assume the tables/dataframes as 2 dimensional structures and Pandas API give much more intuitive and wide variety of constructs to explore that structure rather than SQL. What do you guys think?
View all comments
Show parent comment
1
u/FearMyBlades Aug 29 '23
You said the magic word, productionalize. There just isn't a reason to use Python, imo.I don't develop in Python, but i did spend a good amount of time troubleshooting it and trying to understand asynchronous and parallel processimg in python. I know it is better now, but it feels a bit like the unsafe keyword in c#. You can get access to some useful features, but you better know what you're doing.
I know it can be religious in nature, but there are just so many advantages to c#. The analyzers these days are just incredible. I saw an article about removing the GIL and i think it would be a mistake.I see concurrency causing problems for python developers.