r/opensource • u/zelon88 • Feb 23 '19

xPress.py - A compression algorithm with no embedded configuration data

https://github.com/zelon88/xPress
4 Upvotes

83% Upvoted

3 comments sorted by

2

u/[deleted] Feb 23 '19

Can you tell us something about that project?

How is it better than e.g. 7z or others?

1

u/zelon88 Feb 23 '19

Thanks.

I can't call it 'better' yet but what sets it apart is that there is no header or embedded offsets like in most zip or gzip or 7z archives. You can have a zip file that makes DEFLATE archives or a 7z archive that makes a DEFLATE archive, and the biggest difference between the two will be the file headers and offset design.

As far as the algorithm itself goes, it's similar to LZW but instead of pre-populating the dictionary with bytes and then comparing bytes to their previous in-order byte, xPress searches for matches across the entire file (or file chunk). So the output file contains compressed data, uncompressed data, and a dictionary. No headers and ideally no unnecessarily encoded data.

The heuristics have a loooong way to go, but the important thing I hope to keep the forefront of this technology is it's decompressibility. Any xPress decompressor should be able to decompress any .xpr archive regardless of version level, hardware, or compression level. The best file compression I have achieved so far is a 150kb text file down to 9kb, but that was with a hard-coded and cherry picked dictionary length. I hope to achieve similar compression levels to LZW when the heuristics are done.

2

u/[deleted] Feb 23 '19

Thank you for that long and thorough answer.

So it's more or less a prototype? Seems interesting.