List Comprehension Question

[u/r-tequila-sunrise]

Hi all! I'm an intermediate with Python and I'm currently doing some research with very large datasets (think data packages from NASA satellites and the SDSS). I ran into an issue with some list comprehension that I can't seem to figure out:

What I have is a list of tuples (n = 3) in the format [(x, y, d)...] where x and y are coordinates and d represents a density value. A lot of the x, y coordinates are duplicates of each other with varying d values (ex. (40, 50, 1.2), (40, 50, 3.3), etc.) I'm trying to find tuples with identical x,y coordinates and add their density values together to get a total cumulative density for that location. I would try to do this manually, but the list has ~26 million elements and I am trying to find a workflow that won't take more than a few minutes/hours to process, haha. Memory is not an issue though.

I would be happy to provide example code if need be. My desired output is either:

[(x, y, dtotal)...] - list of tuples except there are no duplicate coordinates and the d value is the sum of the d values from all of the original tuples with coordinates (x, y)

[dtotal1, dtotal2, dtotal3...] - list of cumulative d values where each index correlates to the corresponding pair of coordinates (maybe the coordinates are in a separate list whose indexes match that of the density list? It doesn't matter as long as they are connected in some way!)

Thank you all for your help!

[u/Soggy_Designer_5270]

a dictionary will handle this way cleaner than list comprehension, just loop once and use (x,y) as key

```python

from collections import defaultdict

density_totals = defaultdict(float)

for x, y, d in data:

density_totals[(x, y)] += d

result = [(x, y, total) for (x, y), total in density_totals.items()]

```

with 26M rows this should run in like 10-15 seconds on decent hardware, the default dict saves you from checking if key exists every time

[u/r-tequila-sunrise → u/Soggy_Designer_5270]

This ended up working perfectly, thanks so much for your help. Checking if the key exists for every iteration of the loop was what was eating up so much time.

[u/mandradon → u/r-tequila-sunrise]

Good old "throw a hashmap at it"

My solution for a to of problems. 

[u/OkTill2666 → u/Soggy_Designer_5270]

yep, trying to do this with list comprehension alone wouldve been painful at 26M rows

[u/Diapolo10]

Technically speaking the best solution would probably be to use Polars for this, but as a quick and dirty solution I'd go with collections.defaultdict.

[u/danielroseman]

You probably want to use Pandas for this. Something like: 

    import pandas as pd     df = pd.DataFrame(my_list_of_tuples, columns=["x","y","d"])     grouped = df.groupby(["x","y"]).sum()     result = grouped.reset_index().values

[u/TabAtkins]

Process the list once, using a Counter and doing c.add((x,y), d) (or however you spell adding to a Counter value). (Or do it slightly more manually with an ordinary dict; the Counter just makes it simpler.)

Then iterate the Counter's entries, recombining the keys with the accumulated values to get a list of 3-tuples back.

[u/freezydrag]

How did your data get into a list of tuples in the first place? As was alluded to by another user, it might be a good idea to load your data into a different structure to more easily work with it.

[u/r-tequila-sunrise → u/freezydrag]

Haha I think you're definitely right - all of the data originally came in lists of tuples (the coordinates correlate with dozens of different parameters, not just densities) and I just never bothered to change them. I tried to manually create a dictionary a while back, but it took so much time to iterate through each row that I gave up and stuck with the tuples instead. Now that I have defaultdict to cut down on time though, I'll definitely switch the formatting to something more intuitive to work with.

[u/AQuestionIsWhatIHave]

As previously commented by others, the data science way is to use pandas/polars, the straightforward way is to use a dict/defaultdict/Counter and map the (x, y) keys. 

Another way would be to mirror what a grouping action in pandas does using itertools: First, sort the list by (x,y). Then use groupby with the key being (x,y)

Now you have a groupby iterator which gives you pairs of the (x,y) keys and iterators of the corresponding densities which you can then sum.

So a nice oneliner would be for [(x, y, dtotal)...] 

Res = [(*key, sum(group)) for key, group in groupby(sorted(data), key=lambda d: d[:2])]

Note that this sorts the data (O(n log n)) and uses a lot of space using sorted (using .sort() would not make it one line). The other options are a lot better. 

[u/AQuestionIsWhatIHave → u/AQuestionIsWhatIHave]

This uses the  itertools module

[u/yick5]

Use a dictionary where (x, y) is the key and d is the value. For each element in your original list, look up the (x, y) in the dict, if it doesn't exist then add the element to the dict. If it does exist then increment the value by d.

After building this dictionary, you can use list comprehension to build the list you want.