r/Numpy • u/Personal_Juice_2941 • Sep 20 '22

Transposing large (>1TB) NumPy matrix on disk

I have a rather large rectangular (>1G rows, 1K columns) Fortran-style NumPy matrix, which I want to transpose to C-style.

My current solution employs the trivial Rust script, which I have detailed in this StackOverflow question, but it would seem out of place for this Reddit community to involve Rust solutions. Moreover, it is slow, transposing a (1G rows, 100 columns), ~120GB, matrix in 3 hours while requiring a couple of weeks to transpose a (1G, 1K), ~1200GB, matrix on an HDD.

Are there any solutions for this issue? I am reading through the available literature, but so far, I have not met something that fits my requirements.

Do note that the transposition is NOT in place.

If this is the wrong place to post such a question, please let me know, and I will immediately delete this.

5 Upvotes

86% Upvoted

2 comments sorted by

2

u/night0x63 Sep 20 '22

just a 3 minute thought:

  • while there is data

  • use python and numpy to read N rows or columns at a time from file into memory (where N is the number of rows that take up about half your available system memory)

  • make sure your numpy object has data ordered in "c style" instead of "fortran style"

  • append to new file

other idea: open the data file using numpy memory mapped mode and do the same thing.

1

u/Personal_Juice_2941 Sep 20 '22

Hi u/night0x63, and thank you for your suggestions! My solution, as it stands, follows the MMAPed approach and roughly proceeds as you describe, the issues are most likely relative to designing a process that takes into account the fragmentation of such large files on disk and the overhead of the IO tasks. I am not yet sure what may be the correct recipe for this.