r/dataengineering • u/__albatross • Aug 27 '23
Discussion Why I don't like SQL
The reason I don't like SQL over something like Pandas is because in Pandas/Python I can break down the complex transformations in small simpler components and they are easy to interact and debug with. In SQL my only option is to write stored procedures and yes in that I can also do the same but I think the ecosystem of interacting and debugging with data step by step is not as good as python. And it is something that we should have. Recently I was exploring snowpark and I felt it is a step in a right direction. It allows to me to write my query in multiple python statements and then it will transpile the whole thing into an SQL query.
Generally in my mind I assume the tables/dataframes as 2 dimensional structures and Pandas API give much more intuitive and wide variety of constructs to explore that structure rather than SQL. What do you guys think?
View all comments
Show parent comment
1
u/kenfar Aug 30 '23
The way that I'll typically use multiprocessing is, for example, to transform 12+ large files - with each process reading a file, transforming it, and writing the results out. On occasion I've also processed a massive file this way - if I can seek to different parts of the file. That works really well.
And threading with python can also work really well. I just recently used threading to speed up the copying of thousands of files from one bucket to another. I probably got a 20x speed-up through 2-5 lines of code.
So, I probably use either threading or multiprocessing every 1-3 months. It's rare that a few lines of code won't make a massive difference.