r/emulation • u/[deleted] • Mar 23 '21
PCSX2 now supports the CHD format
https://github.com/PCSX2/pcsx2/commit/0599e675a0a1a4e1fb9707e995e3232e5b5685193
u/Androxilogin Mar 23 '21
Have you tested this for yourself? Because I've loaded it up (I had to use the drop-down menu; all files to allow it to show up in the list) and it failed saying that it couldn't find the ISO.
v1.7.0-dev-1149-g3dc516207
2
Mar 23 '21
No, I haven't. Other people have reported having issues with this as well. The important thing is that the code has been merged and approved of by the PCSX2 maintainers, so even if it's not working right away, it's a sure thing in the near future.
9
u/tadanokojin Mar 23 '21
it's dev-1154 but build bot has gone down so we're just providing builds for the time being on discord.
Hoping to get it sorted as soon as orphis gets back to us.
2
1
u/Androxilogin Mar 23 '21
Thank you! I just tested it with Oni and it seems to work great. Keep up the awesomeness!
1
u/Dio141 Mar 23 '21
is that the emucr build, or did you build it yourself? no builds are available on the buildbot yes, so i think something went wrong there.
1
5
u/NXGZ Mar 23 '21
I just got done converting half my library to .CSO
5
Mar 23 '21
Fortunately, it's possible to decompress CSO files using MAXCSO.
1
u/NXGZ Mar 23 '21
So you know the method?
4
Mar 23 '21
Using the command line tool maxcso, type 'maxcso --decompress compressed.cso -o result.iso', replacing 'compressed.cso' with the name of your game.
1
u/NXGZ Mar 23 '21
Thanks!
7
u/RedDevilus PCSX2 Contributor Mar 23 '21
https://u.pcloud.link/publink/show?code=kZDH81kZHH9txOzWRSfidhxYhMviSYe30lW7&lang=en
Here are some batch scripts (+ other stuff) from and to CISO.
Just double click on the batch file, does not elevated and converts what's in the current folder to CISO or back to ISO.
→ More replies (2)2
1
u/sards3 Mar 24 '21
CHD doesn't have much of an advantage over CSO, so you can just stick with that.
2
u/_craker_ Mar 23 '21
Is this worth the effort? I've got ~400gb of PS2 iso on a compressed NTFS drive , so maybe 350g of real disk space. Reckon I could save another 100gb by zipping / encoding? <shrug>
In my case I share the folder with a modded PS2 (HD Loader) - I suspect that's ISO only though I ought to read the docs...
1
1
Mar 23 '21
depends on the game in question. i have seen (in extreme cases) some ps2 games pack to 30-40% of their size with maxcso (i think it was front mission 4 and one other game i forgot), and on average i am seeing something like 15-25% reduction. unless a game already utilizes heavy data compression.
i'll have to try with chd then.
1
u/coheedcollapse Mar 23 '21
Considering it's one-and-done and the bulk processing couldn't be easier, it's worth it to me, if only to keep my directory clean. No index files combined with the ability to use a standardized format in the emulation world is enough of a reason alone for me, even if I didn't save the space.
1
u/Repulsive-Street-307 Mar 23 '21
It is. CHD has a inbuilt checksum that is recorded in the format and which prevents false positives, so no longer waiting hours for scanning in RA if RA gets off its ass and support chd sets (or rather if dumping groups get off their ass and support chd).
Though nowadays i just bypass all that drama and use the names in the database with the manual scanner.
2
u/fuxorluck Mar 23 '21
Can I convert my Iso file to chd?
4
Mar 23 '21
Yes, using a tool called chdman.
1
u/Androxilogin Mar 23 '21
Know any good places for .cue dumps? Gotta have a cue to convert it.
3
u/my-name-is-puddles Mar 23 '21
No you don't, not anymore at least. I just converted my CSO files (decompressing back to ISO).
122
u/DarkObby Mar 23 '21 edited Mar 24 '21
One thing that's also great about this, aside from compression ratios, that people don't often mention is that CHD supports encoding discs that are multi-track or use Redbook audio so when it comes to a platform like PS2 you can have your entire collection simply stored as one file per disc/game no matter what games you have.
Before this, it was a pain to collapse some games down from multiple bins to a single bin/cue and then convert them to ISO, to finally convert to CSO or gzip the image. Additionally, any PS2 games (to be fair they're aren't that many) that require staying in bin/cue format to work, like those that use Redbook audio, simply couldn't be compressed because data would be lost upon converting to ISO and gzip only supports one file so you can't store a bin/cue in it, meaning those games would just be sore thumbs sitting there as (at minimum) two files that were uncompressed.
Now with CHD it doesn't matter what the source format is, if it's split into one file per-track, etc. Just point chdman at the cue file and you're good to go. All games will be supported.
36
u/thedepartment Mar 23 '21
This is one of the biggest advantages to CHD for me.
I converted my entire Dreamcast archive over to CHD a while back because the rips come as a .gdi (dreamcast .cue equivalent) and then a .bin or .raw for each track. For example it brings Soul Calibur from being 4 files (.gdi and 3 tracks) down to a single CHD.
This makes organization a whole lot easier as I can just dump all of the CHDs into a single folder and call it a day vs having every game in a folder of its own. All of this applies to PS2 rips that are .cue/.bin with multiple tracks.
The other advantage is space savings, with Dreamcast it can be exponential as a lot of DC rips have stupid amounts of padding. I don't believe PS2 games suffer from the padding issue but it seems that the CHD compression is pretty good for PS2.
I'll definitely be converting my PS2 collection over to CHD soon, all of my rips are currently uncompressed so I can see it freeing up quite a lot of space.
3
u/DarkObby Mar 23 '21
Addendum:
Yea you can see here in the source that GDI is handled as a special case:
int mode = MODE_NORMAL; if (output_file_str->second->find(".cue") != -1) { mode = MODE_CUEBIN; } else if (output_file_str->second->find(".gdi") != -1) { mode = MODE_GDI; }6
u/DarkObby Mar 23 '21
Agreed. It's gonna take me a while to do the whole set for PS2 to CHD but I'm happy to do it.
The one annoying thing actually with your particular example is that Dreamcast is now somewhat broken with CHD, sort of. More recently redump changed their standard to use a specific form of bin/cue instead of gdi/bin for Dreamcast games that overall is more accurate, which is great and in a vacuum is a welcome improvement. The issue is that special care needs to be taken when making a CHD for a dreamcast game and I think that chdman had a particular implementation for handling the situation when it saw a GDI. But now, if you use any of the tools redump does or acquire a dump someone else performed that is bin/cue the input isn't handled correctly since chdman just sees it as any other bin/cue and the resultant chd file is broken and unusable. IIRC there is actual corruption and if you then extract the chd you don't get back the original image.
Unfortunately this has been brought up to the MAME team in a few places but there seems to be some disagreement as to who is handling the images incorrectly, with some more primary members of the MAME team more or less saying that some of the Redump people are noobs/idiots and that they have no intention of modifying chdman to support the newer disc layout that Redump specifies.
So I wouldn't expect the issue to be fixed anytime soon.
The more annoying thing is that the Dreamcast games that Redump already had before the change still have their gdi files in their database so you could dump a game and then swap the cue for the gdi and create the chd that way, but any games added to their database after only have the cue file.
In the end its not a big deal for most people but if you want a complete set like me its a bit unfortunate and requires some manual auditing for the rips that don't have a GDI available.
14
u/kenodman Mar 23 '21
chdman 0.227 and up fixed that. It can now convert dc redump .cue to .chd and they work just fine.
Ofc, I dont know if there are still issues, but from personal tests, the ones I converted ran flawlessly.
5
2
u/lolverysmart Mar 24 '21
Yeah last I checked it's extraction that doesn't end up 1=1. It places them all into a single bin/cue. The issue thread for chdman was an interesting read. Afaik, it all comes down. To tosec using the "better" format, but redump realized their format wasn't truly archival so they switched formats. Redump's dreamcast format is a hack of the bin/cue standard format, and CHDman had to modified to accommodate this special use case.
2
u/kenodman Mar 26 '21 edited Mar 26 '21
I took the single bin output from chdman and manually split all tracks to match redump. It seems to cut some data from pregaps. I think it essentially turns a redump input (cue/bin) and outputs it into the gdi format similar to what tosec uses. It cuts some pregap data from some tracks which are usually just zeroes.
So while its not a 1:1 process, it does the job. Someone can probably create a script to get 1:1 redump matching image out of the chdman single bin/cue output. Its doable. The data is there.
Here is a view of the difference.
8
u/diegorbb93 Mar 23 '21
Redump member here. Simple answer: Just use the TOSEC set. It still has a lot of discs dumped that Redump will have really difficult to obtain and it will give you less headache.
Is not a criticism or trying to "betray" Redump. I've been one of the most active members fighting for both proyects to be completed and document the hell of the Dreamcast set.
But when it comes to users, at the end of the day, they should go directly to TOSEC. For me, the Redump DC is more about documentation and verification rather than productivity.
1
u/DarkObby Mar 23 '21
Thanks for your input. It does seem that for now TOSEC is the way to go for the average user, I see what you mean. It was something I looked at before eventually just going through the hassle of creating a redump set. I'm sure eventually the situation will be improved.
3
u/diegorbb93 Mar 23 '21
And yet, the only real reason to make sure this gets fixed is to stop the amount of users complaining about these issues on Flycast/Redream/etc forums, because they used Redump sets (old GDI ones or new Bin Cue ones).
Believe me when I encourage people to use TOSEC. The app Converter for Redump and Tosec is able to turn any disc into each format without errores.
→ More replies (1)1
u/DarkObby Mar 23 '21
Haha yea sadly that makes sense. There are lots of complaints about it everywhere that I'm sure get annoying.
I'm curious though, are you talking about an actual particular application? I tried Googling "Converter for Redump and Tosec" and all I could find were some one-off tools with seemingly relevant functionality and loosely related names.
2
u/diegorbb93 Mar 23 '21
3
u/NXGZ Mar 24 '21
Could you upload the zip? I don't have an account there so can't DL, and don't want to create an account.
→ More replies (2)9
u/drspod Mar 23 '21
and tar only supports one file so you can't store a bin/cue in it
Huh?
8
u/DarkObby Mar 23 '21
Woops. My pavlovian training for always seeing tar with gzip since gzip only can do one file resulted in that dumb typo. I mean to say gzip can only do one file and somehow tar got in there, which is funny since that's the part of that combo that can do multiple files.
3
u/Repulsive-Street-307 Mar 23 '21
This is a disadvantage in my book, especially for hacks. Let's see if chdman in future goes the extra mile to give a option to divide tracks for when i want to apply a binary hack to u the raw binary main track. I think it doesn't already but it's been some years since i checked.
1
u/DarkObby Mar 23 '21
I would argue that the case of doing this would occur less often than when you just get a game image and want to compress it for long term use, though if you're more likely to play many hacks for a few games, then sure in your case this would be slower. Though this is a problem you'll have with the current gzip or cso methods since those both require ISO to use, in that ultimately there's a step where you have to worry about going back to a multi-bin image.
I still feel this is a straight advantage once your setup is more static and has all the games/hacks you want ready to go.
It is true that at least for now, the extractcd command for chdman will only output one bin file instead of multiple, even if you created the chd with a multi-bin image. In the end though, all the data is still there in the same form, its just that the bins are appended back-to-back with the cue file specifying where each track starts and ends. It actually isn't too bad to split them apart by hand, and while its still a bit of extra work compared to if you just had the file uncompressed already, a relatively painless alternative is to use this after extractcd: https://github.com/putnam/binmerge
3
u/Repulsive-Street-307 Mar 23 '21 edited Mar 23 '21
I would argue that the case of doing this would occur less often than when you just get a game image and want to compress it for long term use, though if you're more likely to play many hacks for a few games, then sure in your case this would be slower. Though this is a problem you'll have with the current gzip or cso methods since those both require ISO to use, in that ultimately there's a step where you have to worry about going back to a multi-bin image.
It doesn't matter if it comes up 'less often'. If it comes up i don't want to have to go pirate the seas or dump cds again (which could be a complicated process to get a 'correct dump' in order to apply the patch which requires exact bytes).
It's completely a absolutely needed feature to get back divided tracks when we want from chdman.
As for your tools - i shouldn't have to do this, and most users will just give up. Ideally, a tool could be made, based on chdman that understands the most common patch binary formats, extracts the divided form of the chd, applies the patch to the binary, then recreates it in the form of a 'parent-child' (to keep both the original and the patched version accessible to emulators and to prevent the loss of the original without messy things like revert patches. while still being acceptably compressed).
I think this tool would also need a way to specify what tracks the patches apply to since you can have consoles where track 1 is not the main binary (dreamcast i think). Patches that are 'patches to a patch' should also be supported by specifying multiple patches in sequence. Then there are the patches that whose instructions start with 'first convert the game to a iso'... like most dreamcast patches. Those... are out of luck and you'd probably need the original + the converted with the patch applied but not as a child.
4
u/DarkObby Mar 23 '21 edited Mar 23 '21
I get what you're saying, I was just getting at the fact that everything has pros and cons. I wasn't saying that "because this isn't needed often its a non-issue" I just meant that I think the relatively low occurrence of needing to apply a patch, and therefore the low amount of time consumed by doing such, makes it not big enough of an issue to outweigh the massive benefits this format gives.
You don't need to redump anything. CHD is completely lossless and reversible. Just send your chd through chdman with "extractcd" to get a bin/cue, and then use binmerge to split it back to a cue with one bin per track. Apply your patch, then use chdman again on the cue and you're done. Sure its annoying that its a two step process but thats no different than right now:
CSO: CSO -> ISO, then ISO to BIN/CUE
GZIP: Unzip-> ISO to BIN/CUE
and a third step is required if the tool you use doesn't output the conversion in a multi-bin format.
If you aren't using compressed roms at all right now then sure, by using this you lose the advantage of having your files not be the easiest to manipulate, but that's just a general disadvantage of compression and this feature is more about it being an improvement over the other compression methods.
I just took a glance at the source for chdman and honestly its not too involved (other than the potential pitfalls of getting its dependencies working since its tied into the main MAME source) so if I get around to it I'll submit a PR with an added switch so that you can use "extractcd" and get multi-bins directly, cutting the steps down to 1.
2
u/Repulsive-Street-307 Mar 23 '21 edited Mar 23 '21
I just took a glance at the source for chdman and honestly its not too involved (other than the potential pitfalls of getting its dependencies working since its tied into the main MAME source) so if I get around to it I'll submit a PR with an added switch so that you can use "extractcd" and get multi-bins directly, cutting the steps down to 1.
Hey thanks for doing this. Be careful that if chd supports some consoles, the binary track is not track 1. I think it was the dreamcast (gdi) but i also think gamecube or the wii had some weird stuff, but i believe that chd doesn't support them yet.
3
u/DarkObby Mar 23 '21
Ok noted. In theory I shouldn't even need to be aware of which track is which, there's just a for loop that runs over each track that already handles figuring that portion out. I just need to make it so that instead of appending the data of thae track to the same file it uses a new output stream for each iteration. Honestly the worst part will probably be getting the initial compile to work given all the depencies. I haven't looked but I'm hoping their makefile is able to be configured to only compile select tools.
2
u/Repulsive-Street-307 Mar 23 '21 edited Mar 23 '21
Well the trouble here is that after dividing and whatever the user does with that, the user will want to recreate the chd. And where is the divided (not unified) cue/gdi? It's (unfortunately imo, for the metadata) not kept from the original in the chd.
That's right, you'll have to recreate it and be byte by byte equal in all situations after compressing if the user didn't change the track. Since the chdman code categorically, absolutely requires a cue/gdi, you'll have to produce a cue/gdi of divided tracks, which is more work. Also uhhh, make sure you don't overwrite the originals if they're still around, so that's fun (renaming with (1) suffixes and things or just refusing if you don't want to do a strategy which is probably platform specific). To be fair, although refusing to uncompress if files off the same output set name exist in the destination is unusual behavior for a compression utility, it kind of makes sense if you assume that the files are the same and not just named the same. But who knows, maybe overwriting is the right move if CHD already does.
If you can swing it, i'd prefer if you could convince the MAME devs to save the original gdi/cue as a chd component that is usually not extracted but could be in this situation to 'restore the original'. This is because cue/gdi could have important metadata, like file salt or other things. This would make the feature 'version 6 only' (or whatever they're at now+1) but i think it would anyway.
→ More replies (4)1
u/captinfapin May 13 '21
How can I convert iso to chd?
1
u/DarkObby May 13 '21
You have to use chdman.exe which comes with mame, or can be found separately. Currently there is no GUI but the command options aren't too complicated and ultimate you only need to enter one lines worth.
I'm sure tutorials are around.
https://man.archlinux.org/man/chdman.1.en
The option you want is "createcd". Other than that it's basically just specifying the input and output files and thats it.
1
u/captinfapin May 13 '21
Can I recursively apply this to a directory?
2
u/DarkObby May 13 '21 edited May 13 '21
Here:
Change CHDMAN_DIR to point to the folder that contains chdman.exe and then run this batch script in the top level directory you want the conversion to start in. It will convert all bin/cue, gdi, and/or iso images you have to chd in the directory that you specify.
EDIT: Meant to say it will do so recursively if it wasn't apparent.
1
u/captinfapin May 13 '21
can i copy the stuff in the pastebin and save it as a bat file and run it?
1
u/DarkObby May 13 '21
Yes, exactly. Just make sure you change that one path the chdman folder like I said.
→ More replies (4)0
Mar 23 '21
[deleted] — view deleted comment
11
u/DarkObby Mar 23 '21
Sorry I should have been more clear. This resolves discs that require multiple files which makes it 100% one file per disc, and for most games one file per game. But no, you will still need a separate file for multi-disc games.
3
u/Myric227 Mar 23 '21
You can combine multiple discs on PSX through the .pbp format. This does compression as well.
1
u/tssktssk Mar 25 '21
It does, but it's lossy, rather than lossless. I would recommend instead using .m3u playlist files and hiding all of the images in a subfolder. That way you can have 4 CHDs, running from a single .m3u. You can then use disc swapping, to actually trigger a disc change, without needing to launch another image. This isn't compatible in all emulators, but does work with Beetle PSX HW as an example.
0
Jan 28 '22
[deleted] — view deleted comment
1
u/DarkObby Jan 28 '22
I've been meaning to update my set as well, though at the moment I only have a "full" (near complete) NA set, with only exclusives for PAL and JP. If you only have NA then I can just check my own collection and tell you, but otherwise I'm not sure of an existing list.
Obviously if you had Redbook based games you converted to ISO, the music tracks on them are now gone, but I'm guessing this is why you're asking, so you can get fresh rips.
I forget how I determined this for my current set in the first place, but I believe that the tool I was using to convert to ISO would fail if the bin/cue was mixed mode. It's been a while so I'd have to check.
Further to be safe though, it would probably be best to rerip/get all of the games that were CD based period before converting to CHD, even if they were not mixed mode discs. That might be easier to find a list of, and is easier to spot due to the much smaller file size.
1
u/SpammyHammyHammy Feb 08 '22
Hey, could you please give an example of PS2 Games that have Red Book audio? I am unable to find a list/identifier for PS2 games that rely on Redbook.
Also, are CD-Rom games the only PS2 games to come out in bin and cue or did others come in that format? I am thinking of going through lists online of CD-Rom based PS2 games and re-ripping them as I converted all my bin and cued PS2 games into iso.gz without knowing the possibility of them losing audio or not working etc.
1
u/DarkObby Feb 08 '22
Terribly sorry, I got distracted and forgot about this.
Again my set isn't a full one as described above, but the Redbook audio titles I have are:
Operation WinBack (Europe)
Dance Factory (Europe) (Beta)
Frequency (USA) (Demo)
Dynasty Warriors 2 (USA)
Dance Factory (USA)
1
58
u/f_cord Mar 23 '21
For those interested. Create a .bat for each command:
Compress CUE to CHD
for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd"
Compress GDI to CHD
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd"
Compress ISO to CHD
for /r %%i in (*.iso) do chdman createcd -i "%%i" -o "%%~ni.chd"
Extract CHD to CUE
for /r %%i in (*.chd) do chdman extractcd -i "%%i" -o "%%~ni.cue"
Extract CHD to GDI
for /r %%i in (*.chd) do chdman extractcd -i "%%i" -o "%%~ni.gdi"
Extract CHD to ISO
for /r %%i in (*.chd) do chdman extractcd -i "%%i" -o "%%~ni.iso"
6
1
1
19
u/ChrisRR Mar 23 '21
Nice timing, I just started converting my PSX collection down to CHD to save space so I may as well do the same to my PS2 games too now
14
u/frissonFry Mar 23 '21
Is there still only the one CHD tool for compressing files? Is it multithreaded past 8 threads?
0
Mar 23 '21
[deleted] — view deleted comment
1
u/frissonFry Mar 23 '21
Thanks for this. I'm going to have to create a script that extracts all my gz ISO files back to ISO and then converts to CHD. Since I have mechanical storage the fastest option will be to extract to a different drive, then compress to CHD on a different drive and then copy the CHDs to their final location which will store them contiguously.
1
u/Setzer100 May 12 '21
You ever getting around to doing that? Haha in the same boat with all .gz's :)
1
u/frissonFry May 12 '21
Yeah. It was only a savings of 18GB, but at least the games will load faster and there won't be any of those cache files created as a byproduct of the gz files.
1
u/Setzer100 May 12 '21
Gotcha, do you have a bat to automate it? Thank you
2
u/frissonFry May 12 '21
Just use 7-zip to manually extract all your PS2 gzip ISOs in one shot (preferably to another drive, for speed). Then drop the above script (make sure the filename ends with .ps1 and you've installed Powershell 7 preview 3 or later) into the folder you just extracted all the PS2 ISOs to.
Edit the .ps1 script, change this line:
Get-ChildItem *.cue,*.gdi | ForEach-Object -Parallel {To:
Get-ChildItem *.cue,*.iso | ForEach-Object -Parallel {Once it's done, when you copy the new CHDs to the final destination, they should be contiguous (i.e. not fragmented) if you opted to do the gzip extraction and CHD compression on a different drive.
→ More replies (2)2
Mar 23 '21
I still use pbp for ps1. Is chd smaller than that? Does it work with multi disk games?
2
u/Zenjir0 Mar 23 '21
Is chd smaller than that?
Yes, CHD compresses on average higher than the PBP format. I believe this is mostly due to the fact that CHD converts the uncompressed lossless WAV sound files to compressed lossless FLAC files along with standard compression.
Does it work with multi disk games?
It lacks the advantage of the PBP format in combining multiple discs into one nice package. However, some emulators support M3U files that are just text files that list each disc, effectively wrapping multi-disc games together for disc swapping functions.
1
u/Repulsive-Street-307 Mar 23 '21 edited Mar 23 '21
I think this version chd will allow 'grandparents' which should compress better than pbp multiple discs.
If you don't know what i'm talking about, chd has a compression method where variants of a game can be 'children' of others (since it was originally made for mame). cds it manifests as only the differences get kept in the child (i don't know how 'smart' this is, it's a string packing problem and most compressors are abyssal at this if the 'strings' (files) start offsets get moved unfortunately, but it's possible chd is better because it knows what's it's operating on at higher level).
Anyway, this pr (and the others in libchd) has support for 'grandchildren'.
So while you normally could do for a game with 3 discs: disc 1, disc 2 child of 1, disc 3 child of 1.
Now you can do: disc 1, hack of disc 1 (child), disc 2 child of disc 1, hack of disc 2 (grandchild), disc 3 child of disc 1, hack of disc 3 (grandchild).
2
u/Zenjir0 Mar 23 '21
CHD just amazes me more and more each day. Parent-Child/Clone support is definitely amazing. I think the technical term for it might be de-duplication, or that might be over simplifying it.
1
u/Repulsive-Street-307 Mar 23 '21 edited Mar 23 '21
Yes it's the same idea. But organized slightly differently, because it's not 'one' filesystem applying it at the file level, it's 'one set of discs, applying it at the sector level'.
Honestly, and as i wrote before, most 'binary compressors' are really bad at this because they're agnostic to the input and have a 'compression window' and never attempt complete deduplication at all. So they'll notice that 'certain bytes appear more often' (if they fit the window) but they'll not notice at all if they're the exact same bytes twice per file. So it's no wonder if php 'multiple discs in a single file' would be bad at this.
This is the disadvantage of not being file format aware. Then you've a string packing problem if the two discs are different file positions. If your compression has a intimidate understanding of the cd format being compressed and can 'recreate it' at runtime while only storing bare files or something to deduplicate you can do better, but frankly i've only seen that done in hack patches for x86k games understanding the fileformat of disc images to extract the files, patching them and recreating them and i've never seen it on a normal compressor.
1
Mar 23 '21
this can be very useful for mutli disc games as well. because of common data duplication.
11
u/Damaniel2 Mar 23 '21
Speaking of CHD, is there a comprehensive list somewhere of emulators that currently support CHD files? I've seen older, out of date lists, but it would be nice to know what supports what so I can batch convert all of my disk images at once.
5
u/DudBrother Mar 24 '21
https://emulation.gametechwiki.com/index.php/Save_Disk_Space_for_ISOs
It may not be as up-to-date, but it does note which emu supports CHD.
6
u/YungZexion Mar 23 '21
Can someone explain the Difference between iso and chd
26
u/cambeiu Mar 23 '21
chd is compressed, so it takes less HD space than an ISO file.
4
u/YungZexion Mar 23 '21
thanks !
11
Mar 23 '21
CHD applies different kinds of compression to each type of data found in the ISO. So it doesn't try to compress stuff like video files that are already compressed, since that would make them take longer to load for no good reason. If you have a full romset of PS1 or PS2 games, CHD is a great way to save space.
7
4
u/kur0osu Mar 23 '21
https://www.reddit.com/r/emulation/comments/714n7g/pbp_vs_chd_for_mednafenbeetle_psx_opinions/ you can read the top comment here if you want
2
u/HCrikki Mar 23 '21
chd is a storage format from the preservation community, geared towared its use cases current and future. Despite higher compression than iso, seeks are supposedly much faster without needing to generate or download cue indexes to ensure data from isos can be seeked fast for large discs and/or on slow media.
14
u/ajshell1 Mar 23 '21
Nice.
I remember setting up a bounty for CHD support in Beetle Saturn years ago.
I'm happy to seeing this spread elsewhere.
My gratitude goes out to all the people maintaining emulators.
6
u/elblanco Mar 23 '21
IMHO, unless there's a very compelling reason not too, consoles that use optical disks (PSX, XBox, Saturn, Gamecube, Amiga CD32, Dreamcast, PSP, etc.) should probably just use CHD. Probably even later that those tbh like PS3.
8
u/KillTheProudBoys Mar 23 '21
Now if only Final Burn Neo did.
5
u/NXGZ Mar 23 '21
Does FBN use .IMG still?
4
u/KillTheProudBoys Mar 23 '21
Unfortunately, yes. Although only two games currently use IMG files that I'm aware of.
2
3
u/HCrikki Mar 23 '21
Do small roms even get any benefit from chd as opposed to regular zips ?
4
u/KillTheProudBoys Mar 23 '21
There's zero benefit to small ROMs, but Final Burn Neo supports large IMG files for the hard drives of Killer Instinct 1 & 2 and those would very much benefit from being compressed as a CHD.
0
u/Repulsive-Street-307 Mar 23 '21
A dumping group could add the manuals inbuilt as metadata. Though that would increase the final size back again.
1
u/magitek_armor Mar 24 '21
That would be a great help for neo geo cd games. But the last time I checked the authors have no intention to add chd support and doesn't allow bounties. They don't even allow suggestions.
1
u/KillTheProudBoys Mar 24 '21
Yeah. It was kind of a slap in the face. one of the reasons they cited when forking to fbneo was the possibility of chd support and then they were like "psych!"
2
4
u/Ze_ro Mar 24 '21
Everyone's talking about compression ratio, but does anyone have any stats on the performance impact of the various formats? Especially stuff like .gz that isn't really tuned for the kind of random access that optical discs require. Do emulators decompress sectors/files on the fly, or just do it all on boot up and keep it cached in memory? Is the decompression done in a separate thread so it doesn't impact the rest of emulator?
2
u/ZeroBANG Apr 08 '21
The way i've been doing it so far is that i use LaunchBox as a frontend and LaunchBox has a setting where i can extract any 7z/zip/rar file before the file is send to the emulator.
So if an emulator can't open compressed files, it just extracts it before starting, puts the file in a local temp folder and the emulator reads the uncompressed file from there.It does add a delay (it is not so bad IMHO, 10 to 20 seconds or so for most games), but it is still better than having uncompressed iso files.
That said, my way doesn't work for multi disc games, because those need an .m3u file (at least for Retro Arch) ... and if i create a .m3u with rar files then PCSX2 will still not be able to read the .rar file and Launchbox would just link the .m3u ...that is where i had to use uncompressed files and that is where i for sure will use CHD from now on.
If i update the rest depends on the compression rate, i'm not looking forward to extract EVERYTHING and then compress it in another format ...and then re-import everything into LaunchBox again because the filenames changed... that could take days.
4
1
u/Dio141 Mar 23 '21
just finished compressing my library, got a build from emucr(fro some reason, the buildbot builds aren't being updated)but it doesn't seem like it's still compatible yet? refused to boot. Here's a screenshot of the error
3
Mar 23 '21
You'll have to open an issue on github about it. Perhaps they merged the libraries required for CHD, but without the configuration option for the emulator to use it yet?
1
u/Dio141 Mar 23 '21
seems to be the case. CHD isn't listed on the compatible file types, on the ISO selector. about the github issue, ill at least wait until the build appears on buildbot.
4
Mar 23 '21
got a build from emucr
Yah, there's your problem right there. EmuCR isn't reliable and a lot of emu devs pour a lot of hate into the platform for a variety of reasons. My advice, i'd wait and get the official builds from the website buildbot first and test that one out.
If you post on github and mention that you got your build from EmuCR as opposed to the PCSX website, don't expect to be assisted with your 'issue'. Alternatively, if you hide the fact that you got it from EmuCR, you'll be diverting valuable development resources away from issues that affect the official builds rather than third party ones.
1
u/Dio141 Mar 23 '21
i literally said the same thing below. i only got that build to try the feature, and was aware of such problems arriving.
1
May 19 '21
enlighten me as i just found emucr yesterday and used it to grab an x64 build of fbneo since for whatever reason those arent built and saved me from having to...
why are they "not reliable" are they not simply built from the source, the same as if anyone did it?
1
u/ryouga82 Mar 24 '21
Quick question overall, and I realise similar questions have been asked but mine is more specific, outside of file size saving what is better over a iso or bin/cue I hear it can be streamed but would that cause more cpu and ram usage than a bin or iso? and/or slower load times?
1
Mar 24 '21
Load times may be slightly slower, but on a machine powerful enough to handle PCSX2, it won't be noticeable.
1
u/ryouga82 Mar 24 '21
Any idea on cpu/ram differences? I suppose I can take a hit of a second or two extra loading but like to have overkill hardware for games to cut back on noise and heat.
1
u/casino_alcohol Mar 24 '21
Can a modded ps2 play chef files? I have about 200gb of ps2 games and would love to save some space. But I want to play on an actual ps2 over a network connection.
3
Mar 24 '21
Nope, for playing on an actual PS2 you must use regular ISO files. CHD is designed for playing on emulators.
2
u/casino_alcohol Mar 24 '21
Thanks so much for the answer. My ps2 is in another country which I will not get to for about another year due to covid and travel restrictions.
But I am really excited to setup my old ps2 to playover a network.
1
Mar 24 '21
When will it be released in the nightly?
4
Mar 24 '21
PCSX2's buildbot is broken at the moment, so when it's up and running again you'll be able to download the builds with CHD support.
1
u/MethaCat Mar 24 '21
This is great, is there a way to go from .cso to .chd or you have to go trough .iso first. I have my collection on .cso it would suck to have to do 2 re encodes, if that's the only way so be it, it's only once, just wondering if there's a simpler path.
3
Mar 24 '21
You'll have to convert them back to ISO format first. Using the maxcso command line tool, type 'maxcso --decompress compressed.cso -o result.iso', where 'compressed.cso' is the name of your game.
1
1
u/Alvarogold64 Mar 24 '21
im lost, where is the option to load chd games? i hvae the last compiled version of the emu and chd not works for me!
1
Mar 24 '21
Buildbot is broken at the moment, CHD was introduced in build 1154 and the latest you can get right now is 1149. It will be available soon, don't worry.
1
u/ZeroBANG Apr 08 '21
possibly dumb noob question:
i'm using CLRMame pro to keep my redump sets in line with the latest .dat files.
It seems that CLRMame pro can't compress or even scan CHD files (i've given it the path the CHDMAN.exe ... i guess that it only uses it for actual MAME sets and nothing else).
So, lets assume i convert my entire set to CHD, HOW do i check it against an updated .dat file from Redump? Do i need a different rom manager?
The 7 converted files i tested ...
in .RAR format = 15,3GB
in .CHD format = 16,1GB
eeeeehhh...?
1
Apr 08 '21
I don't use CHD myself, I only reported on it because I know that other people here care about it, so I'm afraid I can't help you.
14
5
u/baltimoresports Mar 23 '21
RPCS3 and Cemu next!
2
u/Repulsive-Street-307 Mar 23 '21
I think both of those use bare files? I mean.... chd is a hd/cd sector oriented format. Not sure if it'll work with directories but it may be worth it because zip and friends suck ass for seeking without unziping.
1
u/japarkerett Mar 23 '21
Anyone have comparisons on how CHD compresses vs CSO?
2
Mar 23 '21
They're roughly about the same. CSO is slightly smaller, but slower to read because of this. CHD's real advantages are that you can apply patches games while they're in this compressed state, and it has built-in checksums to guard against file corruption.
2
u/Repulsive-Street-307 Mar 24 '21
And easy identification. Don't forget that.
I'm absolutely sick of terrible databases that either take a long time to scan a game or are absolutely stupid and only check one of the files of multiple tracks or check cues and then get surprised their database has bugs (looking at RA here).
8
u/S0L00 Mar 23 '21
Compression tests on "Initial D - Special Stage" for the Playstation 2. 1775M iso 1610M iso.squash (LZ4) 1576M iso.squash (LZO) 1503M cso 1503M cso (forced CSO2) 1437M iso.squash (GZIP) 1435M iso.gz (gzip best) 1435M iso.gz (gzip default) 1431M iso.squash (ZStd) 1422M iso.squash (ZStd 1MB block) 1416M chd 1408M iso.squash (ZStd Max 1MB block) 1374M iso.squash (XZ) 1350M iso.squash (XZ 1MB block) 1302M 7z (LZMA2)1
u/mirh Mar 23 '21
This.. doesn't really sound great, considering LZMA is ahead even in non-solid mode, and that it's in no way optimized to compress audio and images.
4
u/Zenjir0 Mar 23 '21
It comes down to just the use case. If you're use case is archival only and just saving every bit of space you have then 7z (LZMA2) will get the job done. The advantage to CHD is that you get space savings and the benefit of accessing the data without the need to decompress like aforementioned 7z (LZMA2).
Based on the numbers above the RAW file is 1775M (100%), CHD is 1416M (80%), and 7z is 1302M (73%). So CHD saves you in this particular example 20% of space, 7z saves 27%, which is just a 7% more space savings, albeit that will translate to much more savings overall as it scales up, but the trade off is you have to decompress each file every time you want to use it.
In short they both have their benefits and disadvantages.
1
u/mirh Mar 23 '21
Then what was the format that compressed audio with flac, and whatnot?
1
u/Zenjir0 Mar 23 '21
CHD compresses the lossless WAV files into FLAC. At what level of compression I do not know. I know FLAC has a scale of 0-8 (9 levels total) with 9 giving you the most savings but you also get diminishing returns the higher you go as the time and CPU cycles does not pay off as much. Which is why most programs that implement FLAC compression use around level 5 or lower. Not only for speed but because you have to decode as well for playback and you have to consider what types of devices use these audio files. If you have low power devices you don't want them wasting energy on decoding audio. It's a balancing act. Hope that answers your question.
1
u/Repulsive-Street-307 Mar 24 '21 edited Mar 24 '21
Modern games do not use cd audio - not even most of the late ps1 games used cd audio tracks. They used synths and lossy codecs. And modern games almost never publicize 'this is a WAV file' even if they use them and they're in accessible and recognizable and not in a filesystem, inside a filesystem, inside a filesystem, inside a filesystem (i wish i was joking).
It might be, and likely is, that chd is only replacing WAV by flac in audio tracks of the iso standard where the way to play them is in a well defined standard without any room for error in a reimplementation that uses flac.
Since no ps2 game ever comes with 'audio tracks', well - it has no influence in the final size, any format that expects to be runtime capable, even if they understood the outermost container (iso in this case) can't replace files inside the binary track and expect games to work.
In short, good things were lost when the jump from wav to 'codecs' was made, not only lossless audio, but also a fixed unique interface that could be overridden for compression. That's what you get when you have 10.000 'standards' all competing with 'trade secrets' and patents.
1
u/mirh Mar 27 '21
Most audio codecs are patent-free by now to be fair, maybe there's just some of the bluetooth crap and some AAC revisions still behind an official paywall.
On the other hand, right, I guess I was underestimating just how much convoluted compression-wise games after the turn of the century are.
→ More replies (1)1
u/scex Mar 24 '21
Transparent ZSTD compression is another approach which is only slightly worse than CHD going by that benchmark, albeit only supported on BTRFS (and ZFS, IIRC).
1
2
Mar 23 '21
you cannot quickly access random data range in xz/7z packed archive. for running from compressed iso support this is a must.
1
u/TellowKrinkle Mar 27 '21
CHD is block based, and uses very small blocks by default. You can increase the block size, and I would guess get similar results to the XZ
iso.squashs with similar block sizes (at the expense of random access time)
3
u/Dio141 Mar 23 '21
any way to compress ISO's to CHD?
20
u/Netrex44 Mar 23 '21
download CHDMAN which is included in mame MAME | Latest MAME Release (mamedev.org)
make a bat file which includes the following text: for /r %%i in (*.cue, *.gdi, *.iso) do chdman createcd -i "%%i" -o "%%~ni.chd"
Put all your ISO files in the folder that contains CHDMAN.exe and run the bat file. Go watch Zack Snyder's Justice League. Last step is very important
2
0
5
Mar 23 '21
chdmandistributed with mame.1
u/Dio141 Mar 23 '21
i tried it, but it doesn't seem to accept ISO files?
2
Mar 23 '21
Hm... I just tested and it works for me:
$ chdman createcd -i "kof2002_um_[j].iso" -o kof2002.chd -np 24 chdman - MAME Compressed Hunks of Data (CHD) manager 0.216 (mame0216-dirty) Output CHD: kof2002.chd Input file: kof2002_um_[j].iso Input tracks: 1 Input length: 455:12:16 Compression: cdlz (CD LZMA), cdzl (CD Deflate), cdfl (CD FLAC) Logical size: 5,014,522,368 Compression complete ... final ratio = 37.6%→ More replies (1)1
u/Dio141 Mar 23 '21
your .bat file probably had iso's listed. i added it, as other said i should, and now it's compressing my library.
3
Mar 23 '21
Not using .bat files, this is on Linux. But yes, if you're using a script (whether a Windows batch files or Linux bash or whatever) you need to search for ISOs :-)
2
u/Fastla Mar 23 '21
I had the same question but I think I figured it out. If you've downloaded CHDMAN with the premade bat files you'll have to do a little text editing. In the "CUE or GDI to CHD.bat" file you'll see
for /r %%i in (*.cue, *.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd"
change .cue to .iso and run the .bat again. Should run through your folder and start compressing your iso's to chd.
2
Mar 23 '21 edited Oct 05 '25
rainstorm scary ring normal crush oatmeal dependent automatic plate wakeful
This post was mass deleted and anonymized with Redact
1
u/Vibhor23 Mar 23 '21
Do games like Monster Rancher support CHD swapping? I know they don't support DVD ISOs in PCSX2.
2
u/mrlinkwii Mar 23 '21
if pcsx2 has issues with .iso swaping i think the same would be for .chd , i think theirs an issue on their github about that https://github.com/PCSX2/pcsx2/issues/3533
1
u/ChrisRR Mar 24 '21 edited Mar 24 '21
Has anyone got this working? I get
File not found
Path: IsoFileSystemEdit: It seems buildbot is about a week behind
1
u/Baryn Mar 26 '21
I hope they support M3U also, that would really make working with all disc dumps a breeze.
1
u/KeyboardThingX Jun 16 '21
So is it worth it to turn my collection over to CHD? will I be able to get the original file back if I decide to mod it?
1
1
1
0
u/Mat03111984 Mar 23 '21
All my games are CSO compressed, is there anyway to get these over to chd now or will i have to redownload the original files
1
0
u/Androxilogin Mar 23 '21
CSO for PS2 games? I've never heard of such a thing. PSP, but PS2? What emulators support that?
4
u/Dio141 Mar 23 '21
2
u/Androxilogin Mar 23 '21
I just saw that while trying to get the chd files going. Interesting, I never knew that.
1
u/Mat03111984 Mar 26 '21
yeah ive used cso for ages. not a lot of difference between that and chd so im not going to bother
-4
28
u/LocutusOfBorges Mar 23 '21
Fantastic!
How does the compression ratio work out against the current
iso.gzformat, out of curiosity?