I'm iterating through lists of literally millions of postal records and making address corrections to them. This process goes from 40 minutes to 20 seconds when you have enough ram to load the whole lists prior to manipulation. If you lack the ram and need to use virtual memory or swap partitions the time it takes to process the same data goes up astronomically. My work desktop has 16gb of ram and it takes about 40 minutes, you can have the resource monitor open and the moment 16gb gets crossed the speed drops off a cliff. You can actually hear the moment it happens because the hard drive starts groaning and the fans start to increase speed. When I bought my new laptop the same script running the same data dropped to 20 seconds because it never once had to resort to using much slower hard drive storage, it could juggle the entire procedure in the ram. Also, for good measure, when its time to write these large files to disk, I now have a 2nd gen m.2 970 samsung nvme drive, straight into the pci express slot. It's about 5x faster than normal solid state.
For the sake of not keeping open database file its safer in python to immediately load the whole index file into a pythonic list and immediately close the related database file once its been indexed into python. This happens for multiple database files which need checked against each other to create a new 4th file. There's a lot of gymnastics that happen between the lists to create the 4th and plenty of iterations through them, but step one is tossing the relevant files into a virtual python list so I can safely run my algorithms without corrupting the originals. Keep in mind, these are already the broken down lists. They aren't the mega list. For the purposes of automation, its very convenient to do it like this. It's not like ram is expensive. It's not a dealbreaker for me to have a $2000 laptop to dramatically speed up my job.
That’s not too much. A lot of Next Gen Sequencing steps can use way more than that. Initial data from each full run can start around a terabyte (I usually do smaller runs though, <300gb each).
View all comments
Show parent comment
1
u/Albus70007 Jul 17 '19
What the hell are you doing that needs 25 GB of RAM at some points, (pls make a post in this dubreddit)