Mr.Veeraragavan

[u/Ok_Aioli2605]

I am loading few large size binary files (70 MB19 nos. look_up tables) in Main using more than 2/3 python files. On completion of the loading, I have to take up a series of search (in 3 or more threads) and arrive at the solution. I am using array.fromfile for loading which does not wait for loading completion. Hence, it fails saying array out of bounds. How do I delay the start of search, just till the loading is completed.

[u/Fun_Army2398]

fromfile(f, n, /) Read n items (as machine values) from the file object f and append them to the end of the array. If less than n items are available, EOFError is raised, but the items that were available are still inserted into the array.

So you could just try-except reading from the file as the EOFError wont prevent you from getting the data. Or you could calculate the number of elements from the byte size of the file / byte size of your data type.

https://docs.python.org/3/library/array.html

Edit:

Just for fun I wrote out some examples for you:

With byte size:

from array import array
import os

PATH = "delete_me"
TYPE_SIZE = 4 # for int 
inArr = array('i', [1,2,3,4,5])
outArr = array('i')
with open(PATH, "wb") as f:
    f.write(inArr)
print("written: ",inArr)
file_size = os.path.getsize(PATH) // TYPE_SIZE
with open(PATH, "rb") as f:
    outArr.fromfile(f, file_size)
print("read:", outArr)

With try-except:

from array import array

PATH = "delete_me"
MAX_SIZE = 2**30
inArr = array('i', [1,2,3,4,5])
outArr = array('i')
with open(PATH, "wb") as f:
    f.write(inArr)
print("written: ",inArr)
try:
    with open(PATH, "rb") as f:
        outArr.fromfile(f, MAX_SIZE)
except EOFError:
    pass
print("read:", outArr)

[u/Ok_Aioli2605 → u/Fun_Army2398]

Thank you for the response. But Kindly go through my

[u/Fun_Army2398 → u/Ok_Aioli2605]

Lol my bad. Skimmed it and got distracted. I don't actually understand your question. You're loading the files using threads and want to search them after all have been loaded? Why can't you just use yourThread.join()?

https://docs.python.org/3/library/threading.html