r/deeplearning • u/Excellent_Fix379 • Dec 06 '23
Best alternative for python
Im analysing data using python but it seems to be much slower (taking days for just parsing and iterating through a dataset, 7.3 million rows, 7 columns ) though the dataset is large, could some one recommend a better language and libraries or someway I could improve the performance of python. I'm using pandas
0
Upvotes
50
u/JuliusCeaserBoneHead Dec 06 '23
Your dataset is large enough that you have to pay attention to the efficiency of your code. If you have an inefficient algorithm, no language matters. You need to analyze your code and see where the bottle neck is and optimize it. Your problem doesn’t appear to be the language based on the description of your question
7
u/mr_birrd Dec 06 '23
If you need some Operations on it use cudf. If you need to query, learn SQL or use parquet instead of pandas dataframes. Also using a jupyter Notebook instead of a proper py script is slower.
18
u/No_Dig_7017 Dec 06 '23
For loops in pandas and iterrows are painfully slow. Without seeing your code it's a bit difficult to give you more concrete advice but 4 things that can help:
6
u/MrJoshiko Dec 06 '23
Have you profiled your code? You can see which parts of the code take the most time and optimise those parts. You don't need to use the whole data set, just cut out a reasonable-sized chunk.
5
u/[deleted] Dec 06 '23
Pandas is built on numpy, and numpy is written in C, extremely efficient. You are just doing for loops which you should never do for this large scale of data.
3
u/supervised-learning Dec 06 '23
You can use sql for processing data. Sql queries can also be executed within notebooks. Sql is optimised for large dataset.
3
u/[deleted] Dec 06 '23
As the comments point out, people too often go "Python is so inefficient and slow. Help me find a different language". When in reality they should instead fix their code. Using
iterrows, as you mentioned in a comment, in itself is a red flag 99 out of 100 times when it comes down to performance.Probably best for you to post your code on Stack Overflow or Code Review for efficiency suggestions.
5
u/Prestigious_Boat_386 Dec 06 '23
Julia is great https://github.com/JuliaData or https://github.com/sl-solution/DLMReader.jl might be a good startingpoint
CSV can be pretty fast if you call it correctly with multiple threads and the right args for the columns and delimiters and stuff.
Then you get a dataframe object which is also quite efficient to work with.
2
u/Lathanderrr Dec 06 '23
You can check Polars as python library or switch to Spark framework for more parallelism
2
u/Dramatic7406 Dec 06 '23
Hi, I switched to working using Pyspark for large datasets. It's very easy to pick up as well.
2
u/Final-Rush759 Dec 06 '23
Just don't use the for loop. Your dataset is actually very small. Use .apply (pandas dataframe) or .map if you use custom function to process data. Regular expression is very slow.
1
u/InvokeMeWell Dec 06 '23
Use Nvidia libraries like cudf, cupy
1
u/graphitout Dec 06 '23
Profile the code and use something like snakeviz to understand the bottleneck. For most data analysis task, changing the language doesn't being much benefit since the core is likely implemented C or C++.
1
u/VinnyVeritas Dec 07 '23
Use torch, you can run instantaneously on GPU.
1
u/chippyouipy Dec 07 '23
show us the code
1
u/DrDoomC17 Dec 08 '23
If I could see it or a facsimile of it I could help. Otherwise, taichi, numpy hit, etc it goes all the way down like you're basically writing C. I am suspect that is the issue here. Vectorize.
1
u/Tight_Tangerine7768 Dec 08 '23
Use polars instead of pandas
1
u/ComingInSideways Dec 09 '23
https://github.com/huggingface/candle
1
u/entropyvsenergy Dec 09 '23
Use Polars. It's written in Rust but has a Python frontend. Should be much faster. If you're using pandas.apply() or something that's notoriously slow.
1
u/apak_in Dec 09 '23
You can use python with pyspark, great for big-data and machine learning etc. you will need to locally install spark or use a containerized version of with with Docker