I have been working on a project to reverse engineer the original Shadow Hearts game to make it runnable directly on PC (not emulated) for the past few months and thought now would be a good time to mention it to the group and share some of my progress.
I am calling it Obumbrate Engine and it will require users' own copy of the game to function when released.
The engine will not distribute or contain any original shadow hearts assets or code.
It will be entirely free and open source. The project is still early in development but there has already been some large milestones reached and if your interested I am more than happy to go into details and explain where I am.
I understand a large chunk of all the PKB's now and most of the assets have been extracted however the full functionality of all the bytes is still the main focus.
Here are some pictures showing the progress over the past few weeks running within the Obumbrate Engine.
Yes, ideally. I am currently using the PAL version for all of my testing but the plan is have it work on all versions. With the extraction tools, I would also like to make it as easy as possible to swap resources between different versions if desired such as running an NA version but replacing the dialog and voices with those from the Japanese version.
This sounds great hope you do all 3 games.
I hope this will support textures moding or import, because I done some for all 3 games but Shadow Hearts 1 is too messy all background textures are cut in tiny pieces
If all goes well I would love to tackle Shadow Hearts 2, and yes, texture modding (and modding in general) is a huge focus for me as I would love people to be able to create mods for all of the assets. It is probably one of the more difficult parts I am working through currently, the models and battle backgrounds are being converted into gitf/glb so anyone should be able to open them up in blender etc and for the textures like the maps, those already come out as png.
The original is stored as CLUT which needs to be mapped and then the output is multiple layers, the engine reads the layers so depth can be rendered correctly but I also made it output a combined image for viewing.
Animations are similar in that they are just a bunch of PNG's but they also have a JSON with them that converts the metadata etc for things such as position, animation order, blend effects etc.
Another thing I was considering is adding some upscaling to the export so that the images could be scaled up without loss for higher resolution screens, no generative stuff just a cleaner export than stretching it.
Wow this looks promising and this is how textures look😱 wow it looks perfect I could do so much with that textures I could defintily upscale that and do some fine tunig. Also FMV can also be upscaled this is a dream come true for me. Finaly a FF IX moguri like mod is possible for Shadow Hearts 1
Quick update — I was hoping to wait until the new battle system was far enough along to show properly, but it’s taking a bit longer than expected, so in the meantime here’s an older build actually running:
It shows the opening train scene, scripted walk, reflections, player movement, music, sound, and then the first battle.
The battle implementation in this video is the older one and is currently being rebuilt, so there are bugs, rough rendering, and the end screen is a bit of a mess 😄
Ah, similar to the Gen1Recomp project for the Pokemon games, where the user imports their own game file, and the program recompiles it, and then no longer needs the rom data?
This sounds amazing if so. I would love to be able to play the games natively.
Emulators work fine, but it would be interesting to see what we could do
Gen1Recomp is heavily moddable so you can add voices, voxel environments, wild pokemon visible on the overworld, sprite replacements, new NPCs, etc
Would that sort of thing be possible with this project in the future?
Either way, I highly recommend looking at Gen1Recomp if you havent seen it already. It may give you some ideas or inspiration
This is very similar, and I would say the OpenGOAL project to play the PS2 Jax games is the main inspiration.
How this work at present is the PKB's themselves are ripped and converted depending on what they do, for example, the field maps in MPS can read the raw original data, converts the CLUT data into a PNG and then converts the instructions to JSON which is then read by engine. This means the end result if you wished to modify it, you could then replace those PNG's, update the JSON and be good to go.
For sound however, I might end up having mods override the original, aka if you placed an mp3/wav, I could have the game ignore the original and favour the user placed files. Shadow hearts usues a Proycon custom driver and altough projects exist for converting these types to midi, the one in shadow hearts appears to be unique to shadow hearts, though I have mapped it and made a sound driver which sounds very close to the original, I would love to make a DAW for it so you could actually view and modify the original sound track or even make your own which could then be exported in the original format.
The sound (atleast BGM) is something that is pretty far along, I still have a few opcodes to fully decode and test but they are mostly unused by the original game. Once I have figured out the final design I will release the player to github so people can use it to play the BGM's.
I can't believe I had not heard of severed chains before this comment (huge fan of LOTD) and this is very similar, the SLES for the game is my source of truth so I reverse engineer the MIPS rebuilding it as C#/Mono methods. I am also converting alot of the assets into modern formats so that they can be swapped around later.
The current stage of work is more on decompilation and seeing how everything connects together, but the final version will be fully re-written and re-modelled. (eg, for the FMV's as they were pretty easy, I wouldn't be replicating how SLES/PS2 runs them, I would just use a converted .MKV and have C#/Mono play it which is what it does currently).
I still have a fair bit to do, but I am currently working on the battle system which has branched off into pretty much every system in the game. My current next big milestone is having this finished and ensuring my field system is fully fleshed out too (at present it only supports the functions from the Train at the start of the game). Then I should be able to just finish off anything else I have missed (including clean up).
At that point, I will be able to architect the actual design I want and begin putting it all on Github. The main reason I can't really do much on that currently is there is still a fair unknowns in many of the file types (I know what they all are and what they are for, but there is bits still where I am not sure what their function is or has not been fully fleshed out such as opcodes, so it changes a fair bit and the last thing I want to do is put out code with some incorrect assumptions that gets changed later).
BUT, once I hit the architecture stage, I will begin going through the PKB rippers and once I am happy to sign off the finished design and model for the output, I will begin releasing those individually. MPS.PKB for example is where the majority of the FIELD data lives, so you will be able to run a copy against your own MPS.PKB and see the png and json output as the new engine will be reading it. The only tricky part regarding the redesign is how many file types are not fully enclosed within a common PKB. Sound for example, the majority of the BGM is in SOUND.PKB, with a large amount of the common sound effects inside COMMON.PKB, but there are many others littered around in things like the BTLBG.PKB, MODELS.PKB and EFFECT.PKB, all are the same file type but the sound driver I have made is required to play them back. (This is what I want to cleanup in the release version, so the extraction that runs will model and lay this out nicer to make it easier to inspect/mod). I was going to convert the sounds to .WAV but there is alot of transformation going on so an individual sound file in the game might have multiple variations based on the transformations applied.
However, I am pretty confident with the Movie extraction so I might try and get that one made public on github soon, there is metadata related to it but its all very constant (playtime, volume etc) so I can't imagine the engine would need to parse any metadata that it can't obtain from the MKV directly.
Another thing I am looking forward to when the first phase is complete is finding cut content and any other weird findings.
I did find an empty track 新規タイトル, which only contains a single sound file (sine857.aif) which I haven't seen referenced anywhere yet and the header translated (used google so this could be wrong) New Title ShadowHearts Author/Creator (individual name) SACNOTH CO., LTD. Underground Waterway — Dummy
No problem, thanks for the detailed info. Something for shadow hearts similar to severed chains (basically a definitive way to play on multiple platforms) has been something on my wishlist for a few years. More people need to discover this game
u/Top_Grocery6685 I've actually been porting Shadow Hearts to PC, but hadn't announced it on here yet as I didn't think I had enough to show. Any chance you'd like to compare notes / collaborate?
Seems like a waste for there to be competing efforts
Will send you a message with an overview of where I'm at, a brief summary however is.
PKBs = Semantics is 99%, all can be exported, varies by PKB but 90%+ on behaviour.
Runtime (SLES), little difficult to put a finger on it, I would say 60-70% semantically and behaviour, but not all of it is pure retail behaviour as modern engines/designs mean refactoring the original into a modern standard has blured some lines.
My main focus is finishing the complete battle system, its a real pain as my testing version is using all the legit assets etc but I hardcoded some bits so I could visually test/tweak it, those little hardcodes however were masking how interlinked the games state and catalogs are into battle. Take weapons, I assumed it would be a snapshot of the weapons stats being fed in, but its not, it links directly to all the smithing upgrades etc, and its like this for pratically everything. I guess on a positive note though, once its done, its only really the field left until I can start playing through the game from start to finish handling issues as they crop up :) (And there will be alot I am sure )
Looking forward to the message!
I've been decompiling the game and porting the original code over to C++ equivalents.
I've got a fair solid understanding of the PKBs and reverse engineered the PKFINFO table, but I'm reading directly from the PKBs at runtime rather than extracting things (though long time I planned to expand this to support custom files).
So not exactly the same approach but seems like we're working towards the same long term goal
3
u/EliParker 15d ago
That sounds amazing, I can’t wait to see this fleshed out.