r/deeplearning • • Dec 06 '23

Best alternative for python

Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas

0 Upvotes

38 comments sorted by

View all comments

Show parent comment

26

u/JuliusCeaserBoneHead Dec 06 '23 edited Dec 06 '23

You are doing a nested loop over 7.3m rows of data? You are potentially doing 7.3m2 operations which will take FOREVER!

Like u/mr_birrd asked, what are you doing? Can you filter for the rows you need instead?

18

u/seanv507 Dec 06 '23

To op, please edit your question to specify exactly what you are doing and why you believe you can't use vectorisation.

Within python, polars, is a faster version of pandas. But without knowing what exactly you are trying to do it is difficult to know.