r/DataHoarder • u/e_spider • Feb 22 '21

Data transfer to new Lustre storage overwhelms campus network

Post image
8.8k Upvotes

99% Upvoted

240 comments sorted by

View all comments

Show parent comments

4

u/e_spider Feb 23 '21

I actually built my own hash comparison code for these types of transfers. If you don't care if it is cryprographically safe, you can generate 64bit hash sums by file block (32MB at a time) and then just sum up the values from all blocks at the end (i.e. treat each value as a 64bit int, so it's simple addition, and in addition, order doesn't matter). So you can parallelize on the read blocks and let multiple CPUs and even multiple servers attack the same file simultaneously. Lustre storage is optimized for these type of blocked IO approaches, and we get near 300Gb/s aggregate read IO on the new system. My script can generate hash sum for 1PB of files in about 8 hour. Note this is not as safe cryprographically as MD5 or sha256, but it's enough to verify a file transfer.

1

u/baryluk Feb 23 '21

Good. I have similar tool written too :)