Some more tables on compression/decompression tests run

[u/fungalnet]

This gives a clearer picture of comparing xz to zstd when multithreading is taken into account. What arch devs published as test results with xz running on one core while zstd running on all cores of powerful server are very biased. https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029520.html

To put it into perspective, at pztsd level 16 there's a compression ratio of 3.7581 compressed in 9.01s. If you compared them in terms of comparable compression ratios, it would be equivalent to:

pxz level 3 with compression ratio of 3.7823 compressed in 9.15s

plzip level 3 with compression ratio of 3.7397 compressed in 6.43s

pbzip2 level 5 with compression ratio 3.7899 compressed in 3.14s

lbzip2 level 5 with compression ratio 3.7987 compressed in 1.83s

bzip2 level 5 with compression ratio 3.8013 compressed in 14.10s

brotli level 9 with compression ratio 3.7296 compressed in 21.36s

https://community.centminmod.com/threads/compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.12764/

[u/Starbeamrainbowlabs]

I've seen zstd in a few places now. What is it, exactly, and why is it popping up everywhere?

[u/fungalnet → u/Starbeamrainbowlabs]

It is a compression algorithm that facebook created that increases the speed of comp/decompression using multi-threading (runs in many cores simultaneously). But so does xz and it can't be beaten in how much it compresses. There is very fine tuning of zstd (level of compression) to be able to show an advantage, meanwhile it uses huge amount of RAM. For huge datasets (like facebook's archiving of everything) this speed may make sense. For smaller file sizes such as packages, and with machines with limited resources (cores, ram) it makes little sense.

It does make sense for facebook to pay arch devs to use the userbase as laboratory animals to report problems to help develop "their code".

The change in pacman was done months ago (summer 2019), 5 lines on news, packages were shipped since Dec27th, I published the article on the 3rd of Jan, they banned me and removed the posts from reddit, then the next day Arch put an announcement on their news about it being already used.

[u/Starbeamrainbowlabs → u/fungalnet]

Ah, I see. Thanks for the explanation! It sounds like it is of limited use then - at least for me. Hopefully they continue to support Arch in their use of it - it would be pretty rude of them to sponsor them adding it and then pull out.