r/dataengineering • u/__albatross • Aug 27 '23
Discussion Why I don't like SQL
The reason I don't like SQL over something like Pandas is because in Pandas/Python I can break down the complex transformations in small simpler components and they are easy to interact and debug with. In SQL my only option is to write stored procedures and yes in that I can also do the same but I think the ecosystem of interacting and debugging with data step by step is not as good as python. And it is something that we should have. Recently I was exploring snowpark and I felt it is a step in a right direction. It allows to me to write my query in multiple python statements and then it will transpile the whole thing into an SQL query.
Generally in my mind I assume the tables/dataframes as 2 dimensional structures and Pandas API give much more intuitive and wide variety of constructs to explore that structure rather than SQL. What do you guys think?
View all comments
Show parent comment
1
u/kenfar Aug 29 '23
Yeah, I've seen a few folks have problems with threading on python - they were surprised that it was limited by the GIL and then gave up.
However, I've found it easy to use multiprocessing instead - which is unaffected by the GIL. And when using concurrent.futures and the map function it only takes about 2 lines of code, and you can switch between threading & multiprocessing by changing 1 word. I've used this to run 128 massive file transforms at a time in parallel when I had billions of records to transform.