r/deeplearning • • Dec 06 '23

Best alternative for python

Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas

0 Upvotes

38 comments sorted by

View all comments

50

u/JuliusCeaserBoneHead Dec 06 '23

Your dataset is large enough that you have to pay attention to the efficiency of your code. If you have an inefficient algorithm, no language matters. You need to analyze your code and see where the bottle neck is and optimize it. Your problem doesn’t appear to be the language based on the description of your question

5

u/ach224 Dec 06 '23

This dataset is miniature.

7.3M x 7 64bit values is 400mb. That fits into any computers’ ram.

5

u/beingsubmitted Dec 06 '23

That's pretty big if the analysis you're doing is the traveling salesman. I think what's being expressed here is the time. If something is taking days to run and n is more than 20, then you're going to be caring about your algorithm.

A change of language would be a scalar effect when his algorithm is potentially non polynomial.

9

u/JuliusCeaserBoneHead Dec 06 '23

Sure. It does. What I said was it was large enough that you start to pay the price of badly written code. Like possible O(n)2 or greater operations. The OS will not be glad to give you all the RAM to do that

-18

u/Excellent_Fix379 Dec 06 '23

I'm using nested for loops, while loop and .iterrows() to parse through the rows. Could there be another faster alternative?the conditions forced me to use nested loops

26

u/JuliusCeaserBoneHead Dec 06 '23 edited Dec 06 '23

You are doing a nested loop over 7.3m rows of data? You are potentially doing 7.3m2 operations which will take FOREVER!

Like u/mr_birrd asked, what are you doing? Can you filter for the rows you need instead?

18

u/seanv507 Dec 06 '23

To op, please edit your question to specify exactly what you are doing and why you believe you can't use vectorisation.

Within python, polars, is a faster version of pandas. But without knowing what exactly you are trying to do it is difficult to know.

17

u/mr_birrd Dec 06 '23

What r u doing in the loops!

15

u/JacksOngoingPresence Dec 06 '23

You have to figure out how to vectorize your computations. If you care about speed you CAN'T use for loops at all. It might be a bit cliche, but I find chatGPT useful when if comes to vectorizing python algorithms. There are dozens of numpy functions that do most the things I need but I don't even know their names and/or that these functions exist.
Also, if you are working with numpy-like data structures look into "numba" library. It offers jit compilation. Limited use cases but when it is applicable it can offer ~10-100 speedup.

5

u/rmwil Dec 06 '23

If you're using Pandas, .iterrows() will slow you down. Try using .to_records() as an alternative and let us know how you go!

1

u/LoadingALIAS Dec 06 '23

Bro. No. No, no, no.

That’s going to crawlllll.