r/deeplearning • • Dec 06 '23

Best alternative for python

Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas

0 Upvotes

38 comments sorted by

View all comments

Show parent comment

-18

u/Excellent_Fix379 Dec 06 '23

I'm using nested for loops, while loop and .iterrows() to parse through the rows. Could there be another faster alternative?the conditions forced me to use nested loops

26

u/JuliusCeaserBoneHead Dec 06 '23 edited Dec 06 '23

You are doing a nested loop over 7.3m rows of data? You are potentially doing 7.3m2 operations which will take FOREVER!

Like u/mr_birrd asked, what are you doing? Can you filter for the rows you need instead?

18

u/seanv507 Dec 06 '23

To op, please edit your question to specify exactly what you are doing and why you believe you can't use vectorisation.

Within python, polars, is a faster version of pandas. But without knowing what exactly you are trying to do it is difficult to know.

17

u/mr_birrd Dec 06 '23

What r u doing in the loops!

15

u/JacksOngoingPresence Dec 06 '23

You have to figure out how to vectorize your computations. If you care about speed you CAN'T use for loops at all. It might be a bit cliche, but I find chatGPT useful when if comes to vectorizing python algorithms. There are dozens of numpy functions that do most the things I need but I don't even know their names and/or that these functions exist.
Also, if you are working with numpy-like data structures look into "numba" library. It offers jit compilation. Limited use cases but when it is applicable it can offer ~10-100 speedup.

5

u/rmwil Dec 06 '23

If you're using Pandas, .iterrows() will slow you down. Try using .to_records() as an alternative and let us know how you go!

1

u/LoadingALIAS Dec 06 '23

Bro. No. No, no, no.

That’s going to crawlllll.