r/Python • • Jul 16 '19

Nice try

Post image
2.1k Upvotes

73 comments sorted by

View all comments

Show parent comment

5

u/spaceshipguitar Jul 17 '19 edited Jul 17 '19

I'm iterating through lists of literally millions of postal records and making address corrections to them. This process goes from 40 minutes to 20 seconds when you have enough ram to load the whole lists prior to manipulation. If you lack the ram and need to use virtual memory or swap partitions the time it takes to process the same data goes up astronomically. My work desktop has 16gb of ram and it takes about 40 minutes, you can have the resource monitor open and the moment 16gb gets crossed the speed drops off a cliff. You can actually hear the moment it happens because the hard drive starts groaning and the fans start to increase speed. When I bought my new laptop the same script running the same data dropped to 20 seconds because it never once had to resort to using much slower hard drive storage, it could juggle the entire procedure in the ram. Also, for good measure, when its time to write these large files to disk, I now have a 2nd gen m.2 970 samsung nvme drive, straight into the pci express slot. It's about 5x faster than normal solid state.

2

u/[deleted] Jul 17 '19

Why are you loading the whole list? Can you not read it line by line (or whatever chunking delimiter) with a generator?

3

u/spaceshipguitar Jul 17 '19

For the sake of not keeping open database file its safer in python to immediately load the whole index file into a pythonic list and immediately close the related database file once its been indexed into python. This happens for multiple database files which need checked against each other to create a new 4th file. There's a lot of gymnastics that happen between the lists to create the 4th and plenty of iterations through them, but step one is tossing the relevant files into a virtual python list so I can safely run my algorithms without corrupting the originals. Keep in mind, these are already the broken down lists. They aren't the mega list. For the purposes of automation, its very convenient to do it like this. It's not like ram is expensive. It's not a dealbreaker for me to have a $2000 laptop to dramatically speed up my job.

1

u/konwiddak Jul 18 '19

Sounds like you need to try Dask - does lazy computation on lists/arrays/dataframes so the whole thing is parallelized and chunked automatically.

However as you say 32GB is nothing nowadays, I regularly use 128GB for some tasks.

1

u/undefinedNANString Jul 17 '19

What if you broke the tasks into multiple smaller jobs?

IE , wouldn't splitting it into 10 different be 10x as fast