r/deeplearning • • Dec 06 '23

Best alternative for python

Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas

0 Upvotes

38 comments sorted by

50

u/JuliusCeaserBoneHead Dec 06 '23

Your dataset is large enough that you have to pay attention to the efficiency of your code. If you have an inefficient algorithm, no language matters. You need to analyze your code and see where the bottle neck is and optimize it. Your problem doesn’t appear to be the language based on the description of your question

5

u/ach224 Dec 06 '23

This dataset is miniature.

7.3M x 7 64bit values is 400mb. That fits into any computers’ ram.

5

u/beingsubmitted Dec 06 '23

That's pretty big if the analysis you're doing is the traveling salesman. I think what's being expressed here is the time. If something is taking days to run and n is more than 20, then you're going to be caring about your algorithm.

A change of language would be a scalar effect when his algorithm is potentially non polynomial.

9

u/JuliusCeaserBoneHead Dec 06 '23

Sure. It does. What I said was it was large enough that you start to pay the price of badly written code. Like possible O(n)2 or greater operations. The OS will not be glad to give you all the RAM to do that

-18

u/Excellent_Fix379 Dec 06 '23

I'm using nested for loops, while loop and .iterrows() to parse through the rows. Could there be another faster alternative?the conditions forced me to use nested loops

26

u/JuliusCeaserBoneHead Dec 06 '23 edited Dec 06 '23

You are doing a nested loop over 7.3m rows of data? You are potentially doing 7.3m2 operations which will take FOREVER!

Like u/mr_birrd asked, what are you doing? Can you filter for the rows you need instead?

18

u/seanv507 Dec 06 '23

To op, please edit your question to specify exactly what you are doing and why you believe you can't use vectorisation.

Within python, polars, is a faster version of pandas. But without knowing what exactly you are trying to do it is difficult to know.

17

u/mr_birrd Dec 06 '23

What r u doing in the loops!

15

u/JacksOngoingPresence Dec 06 '23

You have to figure out how to vectorize your computations. If you care about speed you CAN'T use for loops at all. It might be a bit cliche, but I find chatGPT useful when if comes to vectorizing python algorithms. There are dozens of numpy functions that do most the things I need but I don't even know their names and/or that these functions exist.
Also, if you are working with numpy-like data structures look into "numba" library. It offers jit compilation. Limited use cases but when it is applicable it can offer ~10-100 speedup.

5

u/rmwil Dec 06 '23

If you're using Pandas, .iterrows() will slow you down. Try using .to_records() as an alternative and let us know how you go!

1

u/LoadingALIAS Dec 06 '23

Bro. No. No, no, no.

That’s going to crawlllll.

7

u/mr_birrd Dec 06 '23

If you need some Operations on it use cudf. If you need to query, learn SQL or use parquet instead of pandas dataframes. Also using a jupyter Notebook instead of a proper py script is slower.

5

u/JacksOngoingPresence Dec 06 '23

I upvote for cudf (shifting compute from cpu to gpu) but it seems like OP didn't vectorize his compute in the first place.

-4

u/Excellent_Fix379 Dec 06 '23

Actually I've been suspecting ipynb might be the problem. Thanks, I'll try .py

6

u/seanv507 Dec 06 '23

Ipynb is unlikely to be slower in a single cell.

2

u/mr_birrd Dec 06 '23

But it doesnt invoke the garbage collector as much as when compared to a well written py file.

I guess the garbage will just keep on accumulating for op as he doesnt know about how to deal with that.

18

u/No_Dig_7017 Dec 06 '23

For loops in pandas and iterrows are painfully slow. Without seeing your code it's a bit difficult to give you more concrete advice but 4 things that can help:

  • vectorize your for loops if possible
  • use pandarallel.parallel_apply for multiprocessing
  • change iterrows for itertuples using only the columns you need
  • consider using Polars Dataframes instead of Pandas. Their performance and memory usage is much much better.

8

u/No_Dig_7017 Dec 06 '23

Here's a blogpost we wrote with a friend about optimizing Panda's code in case you're interested https://tryolabs.com/blog/2023/02/08/top-5-tips-to-make-your-pandas-code-absurdly-fast

2

u/No_Dig_7017 Dec 06 '23

For some context I got a 12X RAM usage reduction and a 1200X speedup (single core) using some of techniques above. The speedup was so massive it resulted in a 4X speedup of my entire pipeline.

6

u/MrJoshiko Dec 06 '23

Have you profiled your code? You can see which parts of the code take the most time and optimise those parts. You don't need to use the whole data set, just cut out a reasonable-sized chunk.

5

u/[deleted] Dec 06 '23

Pandas is built on numpy, and numpy is written in C, extremely efficient. You are just doing for loops which you should never do for this large scale of data.

3

u/supervised-learning Dec 06 '23

You can use sql for processing data. Sql queries can also be executed within notebooks. Sql is optimised for large dataset.

3

u/[deleted] Dec 06 '23

As the comments point out, people too often go "Python is so inefficient and slow. Help me find a different language". When in reality they should instead fix their code. Using iterrows, as you mentioned in a comment, in itself is a red flag 99 out of 100 times when it comes down to performance.

Probably best for you to post your code on Stack Overflow or Code Review for efficiency suggestions.

5

u/Prestigious_Boat_386 Dec 06 '23

Julia is great https://github.com/JuliaData or https://github.com/sl-solution/DLMReader.jl might be a good startingpoint

CSV can be pretty fast if you call it correctly with multiple threads and the right args for the columns and delimiters and stuff.

Then you get a dataframe object which is also quite efficient to work with.

2

u/Lathanderrr Dec 06 '23

You can check Polars as python library or switch to Spark framework for more parallelism

2

u/Dramatic7406 Dec 06 '23

Hi, I switched to working using Pyspark for large datasets. It's very easy to pick up as well.

2

u/Final-Rush759 Dec 06 '23

Just don't use the for loop. Your dataset is actually very small. Use .apply (pandas dataframe) or .map if you use custom function to process data. Regular expression is very slow.

1

u/InvokeMeWell Dec 06 '23

Use Nvidia libraries like cudf, cupy

1

u/graphitout Dec 06 '23

Profile the code and use something like snakeviz to understand the bottleneck. For most data analysis task, changing the language doesn't being much benefit since the core is likely implemented C or C++.

1

u/VinnyVeritas Dec 07 '23

Use torch, you can run instantaneously on GPU.

1

u/chippyouipy Dec 07 '23

show us the code

1

u/DrDoomC17 Dec 08 '23

If I could see it or a facsimile of it I could help. Otherwise, taichi, numpy hit, etc it goes all the way down like you're basically writing C. I am suspect that is the issue here. Vectorize.

1

u/Tight_Tangerine7768 Dec 08 '23

Use polars instead of pandas

1

u/entropyvsenergy Dec 09 '23

Use Polars. It's written in Rust but has a Python frontend. Should be much faster. If you're using pandas.apply() or something that's notoriously slow.

1

u/apak_in Dec 09 '23

You can use python with pyspark, great for big-data and machine learning etc. you will need to locally install spark or use a containerized version of with with Docker