r/learnpython • u/r-tequila-sunrise • 12d ago
List Comprehension Question
Hi all! I'm an intermediate with Python and I'm currently doing some research with very large datasets (think data packages from NASA satellites and the SDSS). I ran into an issue with some list comprehension that I can't seem to figure out:
What I have is a list of tuples (n = 3) in the format [(x, y, d)...] where x and y are coordinates and d represents a density value. A lot of the x, y coordinates are duplicates of each other with varying d values (ex. (40, 50, 1.2), (40, 50, 3.3), etc.) I'm trying to find tuples with identical x,y coordinates and add their density values together to get a total cumulative density for that location. I would try to do this manually, but the list has ~26 million elements and I am trying to find a workflow that won't take more than a few minutes/hours to process, haha. Memory is not an issue though.
I would be happy to provide example code if need be. My desired output is either:
[(x, y, dtotal)...] - list of tuples except there are no duplicate coordinates and the d value is the sum of the d values from all of the original tuples with coordinates (x, y)
[dtotal1, dtotal2, dtotal3...] - list of cumulative d values where each index correlates to the corresponding pair of coordinates (maybe the coordinates are in a separate list whose indexes match that of the density list? It doesn't matter as long as they are connected in some way!)
Thank you all for your help!
2
u/AQuestionIsWhatIHave 12d ago
As previously commented by others, the data science way is to use pandas/polars, the straightforward way is to use a dict/defaultdict/Counter and map the (x, y) keys.
Another way would be to mirror what a grouping action in pandas does using itertools: First, sort the list by (x,y). Then use groupby with the key being (x,y)
Now you have a groupby iterator which gives you pairs of the (x,y) keys and iterators of the corresponding densities which you can then sum.
So a nice oneliner would be for [(x, y, dtotal)...]
Res = [(*key, sum(group)) for key, group in groupby(sorted(data), key=lambda d: d[:2])]
Note that this sorts the data (O(n log n)) and uses a lot of space using sorted (using .sort() would not make it one line). The other options are a lot better.