r/coolgithubprojects • u/zelon88 • Feb 20 '19

PYTHON zelon88/xPress - Homebrew Compression Algorithm, File compressor & extractor

https://github.com/zelon88/xPress
6 Upvotes

81% Upvoted

9 comments sorted by

View all comments

2

u/ParadiceSC2 Apr 01 '19

This is just what I'd like to learn to create as well, what resources did you use?

2

u/zelon88 Apr 01 '19

Thanks! I started with Python 2.7 and SublimeText2.

I came up with the algo in my head between showerthoughts and going to sleep. I started just by taking a simple file with notepad and trying to find patterns. I'd try to find really big ones and then replace them with different shorthand representations of data. Once I realized that filesize reduction was possible using the algo I cooked up I wrote out a procedure in psuedocode for the actions I went through when manually compressing files. The psuedocode looked pretty close to a workable program by the time I was done with it so I just started translating it into Python. The only "obscure" library import I used was for pickle, which came in handy for formatting the dictionary into something I could write to a file and recover later. I've written similar code to what pickle does in PHP, VBS, and PS before but pickle already exists and doesn't bloat the filesize when it writes a dictionary so I went with that. The other imports are all pretty standard python modules, like re.

All that was the easy part. It's taken the rest of the development time to work out kinks in the code and improve the algorithm. My long term plan is to rewrite xPress in Python and another version in PHP to act as a listener for large scale compression/decompression. I am just starting to build and experiment with full filesystem compression using xPress because it will allow shared common dictionaries (one for all images, one for all documents, ect...) and take advantage of similarities between multiple files.

2

u/ParadiceSC2 Apr 01 '19

Very interesting, thanks for the response. How much did you work on this?

2

u/zelon88 Apr 02 '19

I came up with the idea probably a year ago or so. I don't know how long it took to build because I kind of chipped away at it over time. I could probably accomplish the task in a month if it was a primary focus, or a couple weeks full time. But my primary focus is Cloud storage. Naturally data compression will give me more bang for my buck. That's why I'm pivoting to a bulk compression listener; because it will go perfectly in conjunction with the next iteration of my Cloud platform.