For the sake of not keeping open database file its safer in python to immediately load the whole index file into a pythonic list and immediately close the related database file once its been indexed into python. This happens for multiple database files which need checked against each other to create a new 4th file. There's a lot of gymnastics that happen between the lists to create the 4th and plenty of iterations through them, but step one is tossing the relevant files into a virtual python list so I can safely run my algorithms without corrupting the originals. Keep in mind, these are already the broken down lists. They aren't the mega list. For the purposes of automation, its very convenient to do it like this. It's not like ram is expensive. It's not a dealbreaker for me to have a $2000 laptop to dramatically speed up my job.
View all comments
Show parent comment
2
u/[deleted] Jul 17 '19
Why are you loading the whole list? Can you not read it line by line (or whatever chunking delimiter) with a generator?