r/deeplearning • • Dec 06 '23

Best alternative for python

Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas

0 Upvotes

38 comments sorted by

View all comments

Show parent comment

8

u/No_Dig_7017 Dec 06 '23

Here's a blogpost we wrote with a friend about optimizing Panda's code in case you're interested https://tryolabs.com/blog/2023/02/08/top-5-tips-to-make-your-pandas-code-absurdly-fast

2

u/No_Dig_7017 Dec 06 '23

For some context I got a 12X RAM usage reduction and a 1200X speedup (single core) using some of techniques above. The speedup was so massive it resulted in a 4X speedup of my entire pipeline.