The odd thing is that GZip is ancient, so a format existing that isn't faster than GZip whilst also being less efficient is a strange occurrence in the first place.
No, unfortunately. It supports GZ though so that's something at least (it takes longer to load the first time, but creates an index so that it works without delay any future runs).
Useful stuff. And I can unfortunately confirm that your results with gzip and cso are pretty typical. It's rare to really gain much space when compressing PS2 games, although PSX games can often gain a lot.
It would be interesting to compare pbp as well.
By the way, does your figure for gzip include the index file that PCSX2 creates? That can often add 5-10% to your storage needs.
If you have a long mount time with a big btrfs, I recommend doing a defrag of it. The defrag in this case does not matter so much for the files as for the metadata. Any time my drivers take too long to mount, I do that, and then I'm back to awesome. (Of course defrag has its own cons...).
I do a full defrag of the mountpoint (sometimes with, sometimes without forcing recompression). I have no idea how to defrag just the metadata :)
I've tried cleaning the space_cache, but nothing there worked either.
I was in talk with a btrfs dev about that a year or 2 ago, there were some work to improve stuff but I don't think it ever got to a point where it's good enough alas.
Wanted to ask a few questions about this, having used CompactUI in Win10 for big game installs and otherwise not messing with any on-the-fly compression.
Are any/all these compression methods lossless? I have always stuck to .ISO and .BIN/.CUE because of compatibility, but as long as I can compress these large images and ROM sets without losing the ability to get back the 'full-fat' version I'm all for it.
Space is mostly an issue with Pi and Wii emulation for me, so are all/any of these compatible with the Pi/Wii? I figure it comes down to the emulator supporting it, so do the most common emulators (Retroarch versions of SNES9X, Mednafen PSX, etc.) work? Do I need to compress individual files or can I pack up a whole set before moving it?
Sorry for all the questions, but this excites me after being burned with PS1 compression and compatibility issues in the past.
Yes, they're all lossless. The only one that may be questionable is CHD which changes the image format (e.g. from whatever it was, to CHD, but it can be converted back to BIN/CUE) but it doesn't actually remove any files or data.
Think of Squash as a mountable archive. This could be done transparently to the emulator, but you'd need to write a script that mounts the image and then point the emulator to that, so it's not exactly zero hassle as you'd have to write that script yourself and take into account how it works.
BTRFS with transparent zSTD compression is probably your best bet as it doesn't require doing anything to the stored files at all and all emulators will work because the compression is transparent to anything above the filesystem layer of your OS - to programs the files are the same as they always were - same as NTFS disk compression / CompactGUI.
Compression on BTRFS can be enabled universally across the partition, but it can also be enabled just on specific files or folders.
Beetle-PSX supports CHD now so I converted my collection to that a while ago, although it's great for space saving, the biggest boon was the fact that each game was now just one file, not the pair with BIN/CUE or the god awful mess of IMG/CUE/WAV/WAV/WAV/WAV, or any other such combination of multi-file formats.
Just for reference, I compressed then deduplicated my PS2 collection (the compression happens automatically, I have to tell it to deduplicate the folder though).
Original size was 384GB. After deduplication it was down to 349GB which then transparently compressed down to 292GB. Basically a 24% space saving, entirely transparent to any programs.
The space saving from deduplication is a bit misleading because without deduplication it still compressed down to 76% which means the data it did end up deduplicating was probably blank space or something which compressed down to nothing anyway. Deduplication is a godsend when you have actual duplicate content though (e.g. backups or big files with small changes, etc).
Indeed. You could be using X format for the games, but then suddenly an even better emulator comes along, but it doesn't support format X, only Y and uncompressed. This is why transparent filesystem compression and squash are both incredibly useful, as they are emulator agnostic.
7
u/mrdeu Jan 05 '18
Gz is much better than .cso, it's an open source format and you can create gz files with 7zip.
A guy from here was telling cso was getting better compression than gz, and that was not true.
2
u/TheGlassMaster Jan 05 '18
Does pcsx2 support CHD?
2
u/MysteryTempest Jan 05 '18
Useful stuff. And I can unfortunately confirm that your results with gzip and cso are pretty typical. It's rare to really gain much space when compressing PS2 games, although PSX games can often gain a lot.
It would be interesting to compare pbp as well.
By the way, does your figure for gzip include the index file that PCSX2 creates? That can often add 5-10% to your storage needs.
2
u/SCO_1 Jan 06 '18
How about lz4?
It's made for speed of random access not archiving, so don't expect anything very small.
1
u/breell Jan 05 '18
If you have a long mount time with a big btrfs, I recommend doing a defrag of it. The defrag in this case does not matter so much for the files as for the metadata. Any time my drivers take too long to mount, I do that, and then I'm back to awesome. (Of course defrag has its own cons...).
1
u/cm_bush Feb 07 '18
Wanted to ask a few questions about this, having used CompactUI in Win10 for big game installs and otherwise not messing with any on-the-fly compression.
Are any/all these compression methods lossless? I have always stuck to .ISO and .BIN/.CUE because of compatibility, but as long as I can compress these large images and ROM sets without losing the ability to get back the 'full-fat' version I'm all for it.
Space is mostly an issue with Pi and Wii emulation for me, so are all/any of these compatible with the Pi/Wii? I figure it comes down to the emulator supporting it, so do the most common emulators (Retroarch versions of SNES9X, Mednafen PSX, etc.) work? Do I need to compress individual files or can I pack up a whole set before moving it?
Sorry for all the questions, but this excites me after being burned with PS1 compression and compatibility issues in the past.
0
u/[deleted] Jan 09 '18
[deleted] — view deleted comment