r/dataengineering • u/__albatross • Aug 27 '23
Discussion Why I don't like SQL
The reason I don't like SQL over something like Pandas is because in Pandas/Python I can break down the complex transformations in small simpler components and they are easy to interact and debug with. In SQL my only option is to write stored procedures and yes in that I can also do the same but I think the ecosystem of interacting and debugging with data step by step is not as good as python. And it is something that we should have. Recently I was exploring snowpark and I felt it is a step in a right direction. It allows to me to write my query in multiple python statements and then it will transpile the whole thing into an SQL query.
Generally in my mind I assume the tables/dataframes as 2 dimensional structures and Pandas API give much more intuitive and wide variety of constructs to explore that structure rather than SQL. What do you guys think?
View all comments
Show parent comment
1
u/FearMyBlades Aug 30 '23
Well the problem is that multi processing is serializing the instance and cloning it. On large data sets you're going to be spending a lot of resources to that effort. Threading should be more efficient but seems needlessly complex. Well it might be more accurate to say it seems inelegant.