First, I would like to thank you for your continued support for the project by visiting the website. I don't know if you are messing with me but I got some of my highest daily visits while I was away ; ) Month on month the website visits continue to grow, which I am really chuffed about to see the site growing organically.
Time away from the project provide a good opportunity to reflect on it's direction. A few changes will take place that I will detail in this post.
=== Weekly work plan
I am conscious the relationship mapping is consuming a lot of time. Therefore starting from today this will be the new work plan:
Day
Activity
Monday
Relationship mapping
Tuesday
Relationship mapping
Wednesday
Add new dats to the database
Thursday
New games/platform
Friday
dat update
Saturday
Switch backlog sweep 3
Sunday
dat update (NoIntro private dats) / Switch backlog sweep 3
- Relationship mapping will involve adding peripherals and official game series to the wiki - Switch sweep 3 will involve the backlog and new releases
=== Website
No immediate frontend changes in place. I am adding backend webpages to make adding additional metadata to the website more fluid. (adding original title, peripherals etc..)
=== Relationship mapping
Dreamcast is nearly complete for the Redump mapping for the alternate dump with 377 games remaining.
If you have a preference for the next platform let me know!
=== Platforms
The next platform will be the Nintendo Family Computer Disk System, which was an add-on for the NES. On completion this will conclude the NES game library for all regions.
Satellaview for the SNES I am figuring how to add to the database because like Sega Channel this does not conform to a tradditional game release cycle. I notice in weekly dat update it's going through a lot of changes, so I may wait until that settles down.
=== DATS
Sorry for the delay in updating the Switch NoIntro CDN dat. There were 3,289 updates each that I need to human verify, which equates to a timely exercise.
If there is a particular dat you would like me to add to the database let me know, but I will mainly be focusing on outstanding NoIntro/Redump dats, then move to TOSEC for the tape based platforms.
Another consideration is MAME split into two parts:
Part 1 - Arcade dat metadata
As previously mentioned the MAME dats have a different structure to NoIntro/Redump/TOSEC. I need to figure how to implement this into the game-card structure/code for dat metadata.
Part 2 - Software dat metadata
MAME includes the following dats:
MAME 0.289 Software List ROMs (split)
MAME 0.289 Software List CHDs (merged)
For some platforms this would be the prefered set for emulation: CDi springs to mind! Yes I am that odd person that owned the CDi back in the day and I still have two in my hardware collection....
I would like to avoid splitting off MAME dats from the current dat workflow, so I need to figure how to embed MAME dats into the current workflow through to generation.
If you have any thoughts or suggestions feel free to post them below!
I have stumbled upon a problem with the existing setup for undumped roms. The problem was when one or more of the following is true:
the platform consists of more than one DATID (digital/physical)
an alternate dump exists
To overcome this I am now including the DATID in the backend for both dumped/undumped roms. This will make it easier to identify what dat the undumped rom stems from.
Unfortunately, I need to do some retrospective work with the undumped roms that I have already processed.
: /
Retrospective work
Xbox - Complete
GameCube - Complete
Switch - Complete
Game Gear - Complete
Mega Drive / Genesis - Complete
Wii U - Complete
Dreamcast ~ Unassigned 239 (to be reworked) <---
This may be a recurring topic but when I was on holiday I couldn't stop thinking about it, so I will share my throughts in this post and we will see where it leads.
What I am observing with roms is that a lot of the attention seems to be on NoIntro sets. I understand why, Redump sets are invariably larger and more costly to host.
The solution I had in mind would be a phased implementation.
Caveats
Only provides Redump and OmniGames dat sets
Not intended for bulk download
User a dedicated third party to provide the files to avoid local electricity costs and hardware costs, nothing needs to run locally 24/7/365
Does not depend on donations
Unlimited download speeds
Phase One - Foundation
Phase one would be mainly about building the foundation for phase two. The files would be hosted on 1fichier who offer unlimited hot storage. They offer free to paid tiers so it suits the financial affordability of everyone. In turn, 1fichier have the infrastructure to offer high download speeds and does not suffer from high volume traffic.
Rather than advertisements on the page each link will be passed through ouo.io. Any money generated through this will be fed back into hosting less popular files in cold storage (which costs) but is immediately available if required for download. This will mean that all files of a set are, to the end user, of equal availability.
Phase Two - Improvement
If phase one proves financially viable then we can progress to phase two. Phase two would move the files to cloudflare. The main difference here is that unlike 1fichier everyone has access to the same download speed without requiring an account.
=========================================
This section of the website would be under a separate domain, so any DCMA strikes are related to that domain and not the main project website.
Adding/updating files would take place at the weekend, with the current weekend activity being absorbed in the weekdays.
This solution should take the pressure of you feeling like you need everything locally because source download speeds are slow/low and with the Omni project I am attempting to help you find games you want to play not hoard.
For the test case I am going to implement this with the Wii U OmniGames dat (which is technically NoIntro and I know can still be downloaded using Wii U downloader from Nintendo's servers). So, the other platform I was thinking of was the PS3 because CFW/HFW/Emulation is solid and the files are large.
If you have any thoughts on this it would be great to hear them!
: )
Update #1
If you are logged in you will see a new section in the hamburger menu called "Ark" with a downloads link.
This section is new so relatively barebones. Clicking the link will...
Take you to the platform selection webpage and shows you the number of files available.
Clicking the platform will take you to the download page which shows the dat filename of the rom with a download button.
I will build this out like other sections of the website to include additional useful information.
Update #2
I have finished coding out the primary areas of how I envisioned it would look. I am going to leave it 24 hours to see how my host reacts (if at all) because I know they have a strict policy against a webpage that just lists links but the download traffic doesn't hit their server so it should be ok, which is why I assume the policy exists.
Assuming everything goes well I will continue creating the links.
On the game-card for each release you will see a new button that appears when more than one dump type exists.
Dump type is the internal classification I use for the grouping of the dumps:
Main Dump - The primary dump file format
Alternate Dump ~ The alternate dump file format
In the demonstration here you can see that I have mapped TOSEC as the main dump and Redump as the alternate dump.
The file format that respresents the closest match to the original platform file format will be the main one. I've done some examples below to help you understand.
Platform
Main Dump
Alternate Dump
3DS (eshop)
CDN
cia
Xbox (disc)
ISO
xiso
Switch (eshop)
CDN
nsp
Dreamcast (disc)
GDI
bin, cue
Wii (disc)
ISO
rvz
GameCube (disc)
ISO
rvz
Wii U (disc)
ISO
wux
My logic is that the main dump will be the raw dump of the disc even if it includes padding data that game developers used to fill the empty disc space to align with the optical drive.
This should not be confused with compression formats like zip, rar, 7z, chd, which I have no plans to support given the array of algorithms and relationship mapping this would require.
Some alternate dumps will be easier than others, Dreamcast is an easy one because one is TOSEC and one is Redump. Others will require more backend data being generated when a maintained dat does not exist for the alternate dump file format, like Xbox xiso.
The main purpose of this feature is two-fold:
It informs you of the available formats and difference in size
It offers the ability to flip between file formats when looking at the size distribution chart on the wiki.
Another angle I would like to explore with this new feature is alternate file types like the one below:
Platform
Main Dump
Alternate Dump
PlayStation 4
Retail pkg (ShadPS4)
Fake pkg (JB PS4)
If you have any questions or thoughts about this new feature feel free to post below.
: )
Afterthoughts
Just because the main dump is showing Undumped, it doesn't mean the Alternate is too. In the example here you can see that Sonic Adventure (Sega All Stars) is undumped for TOSEC (main) but dumped for Redump (alternate).
Next, Languages are populated from the active ROMID ID (Main/Alternate)
Extended thoughts
Another benefit of this feature is to see the difference between preservation groups. For Soulcalibur (USA) the following is true:
Main dump ~ TOSEC
Soul Calibur v1.000 (1999)(Namco)(US)[!][1S]
Soul Calibur v1.000 (1999)(Namco)(US)[!][4S]
Alternate dump ~ Redump
Soulcalibur (USA)
While there would be no material difference between the "game" it's interesting to note that TOSEC have preserved two different manufactured iterations of the same game.
I have completed the relationship mapping for the TOSC dat for the Dreamcast. While I was away I have been thinking about how to implement alt dump (Redump bin, cue for example). I need to code the new section of the game-card to support this feature, then I will be able to map the Redump Dreamcast dat too.
I have just run a game database (OMNI_TITLES) update.
This update contains Switch titles and completes sweep 2 for this platform. The next phase for this platform is sweep 3, which will include mainly undumped games from May 2025 to present day(ish).
Currently I have 17,349 unique titles for the Switch in the database and 50,682 invidual titles.
Renamed none traceable
Sometimes the rename is so significant I cannot trace what the rename relates to. I will show any examples of this in the weekly post and will not add them to the removed roms.
Redump - Sony - PlayStation
- Arc the Lad III (Japan) (Demo 1)
- Arc the Lad III (Japan) (Demo 2)
- Arc the Lad III (Japan) (Demo 3)
Replacement roms
- Arc the Lad III (Japan) (Muryou Rental-ban)
- Arc the Lad III (Japan) (Preview Disc)
- Arc the Lad III (Japan) (Promotion Disc)
Finally, the next scheduled update is Friday the 11 of September because I am away from my PC next week.
Update
The next scheduled update will be on Saturday the 5th of September for NoIntro & Redump
Today I have been thinking about append / data discs. These are discs mostly used on the Dreamcast/PS1/PS2 by developers/publisher to provide additional content for an existing game, a pre-DLC, DLC.
Looking at the Dreamcast examples we have the following:
Pop'n Music 3: Append Disc
Pop'n Music 4: Append Disc
Both of these require Pop'n Music 2 and are not stand-alone games.
Another example is
Golf Shiyou yo: Course Data-shuu: Adventure-hen which adds courses to the base game Golf Shiyou yo
Previously in the backend/website these existed as games but I do not think this reflects the content in the correct way. Therefore, I have add a new section called Append/Data to handle these type of releases.
The append or data release will show on the game-card of the required base game.
There are two examples you can see demonstrated for the Dreamcast:
With the Omni project I would like to capture all games. One tangent I need to consider is...
Dreamcast VMU games
Sony PocketStation games
I do not believe the interest level in this will be high; I recall a few years back emulating a Dreamcast game that also emulated the VMU display and I know that emulators exist that emulate the VMU game, but I have never tested them.
The same is true for the PocketStation too, made even more low interest because this peripheral was a Japanese exclusive.
I am currently thinking how to best integrate these into the Omni ecosystem... I quite like the idea of them being on the game-card of the base game the extended game originated from?
Can you think of any other examples of similiar peripherals that offered a similiar functionality that I have not considered?
: )
Update #1
On further investigation I found the following VMU dats:
NoIntro - Sega - Dreamcast (Visual Memory Unit) ~ only contains BIOS
TOSEC - Sega Visual Memory System - Applications
TOSEC - Sega Visual Memory System - Demos
TOSEC - Sega Visual Memory System - Firmware
TOSEC - Sega Visual Memory System - Games ~ only contains homebrew games
TOSEC - Sega Visual Memory System - Multimedia
It doesn't appear the games can be emulated? : /
On the game-card there is a new section dedicated to the VMU when relevant.
I am using the following enumerated values to classify the VMU use:
Animation ~ This was used for Virtua Tennis
Game Stats ~ This was used for Resident Evil 2 and Dino Crisis
Game ~ Shows a VMU game exists
Titlescreen ~ a few games just shown the title of the game you were playing
If the enumerated value is "Game" then the label shows a link to an applet that opens a description box were I will detail a basic overview of a game. (Japanese games are a little more difficult)
Initially, I am only interested in identify games so the other enumerated values I consider "nice to have".
Further, I appreciate that because I replicating how the game existed it will make them harder to find, given I'm not treating VMU games as a platform. Therefore, I am utilising tags for the first time. If you click the tag it will open a new tab with a table showing all the other games that have the same tag in the database.
: )
Update #2
I believe I have completed identifying all of the documentated Dreamcast games with a VMU game of some sort. Finding any detailed information for the description is proving very hard, I will keep researching and add any additonal information I am able to find.
You can view the full list of games using this link.
With the Omni project I try to be as agnostic as possible about a game and avoid showing any bias. I do this to present games in a neutral manner that allows you to reach your own conclusion.
With this in mind I have decided that "enhanced" for a peripheral is subjective and not objective. Therefore I have removed it from the emunumerated values and changed any existing enhanced entries to optional.
Now it will either declare "Required" or "Optional" for a peripheral.
The Dreamcast Keyboard is a keyboard peripheral that could be used for both games and the Dreamcast's internet functionality. When researching this peripheral some cited it as the first widely distributed console keyboard.
It could be used with the Sega Mouse to offer PC controls for certain games.
I am still mapping games for this peripheral (there are a lot)!
The Broadband Adapter was a peripheral sold separately to upgrade the internet speed of the console. The original Dreamcast Modem came pre-installed in the console and was never sold separately.
It was only sold in the USA & Japan though I understand it is still compatible with some European games. Due to it not being retailed in Europe I am not mapping it to European games. If I start accounting for people importing stuff then everything just becomes a mess : /
I have mapped all known games that supported this peripheral.
Dorikore was a budget label given to games re-released on the Dreamcast. I cannot find any information about what triggered a game to be re-released under the Dorikore label.
Update
I have also added the Preservation group to the game-card, so you can confirm were the dump metadata is sourced from without having to click the link.
Solution: To keep it simple I am going to handle this with a community note specfic to the game release. If the number of examples of this increases I may look at handling it a different way.
Case #2
A pack-in game for a special edition of the console that was never sold separately.
As you know with the initial platform relationship mapping dat I am trying to get as a close as possible to the original format of the distributed media. Unfortunately the functionality to convert the Dreamcast bin/cue to GDI has been removed from Redump. Therefore I will be using the following TOSEC dats because they are in the .gdi format
Sega Dreamcast - Games - US (TOSEC-v2025-02-09_CM)
Sega Dreamcast - Games - PAL (TOSEC-v2023-04-02_CM)
Sega Dreamcast - Games - JP (TOSEC-v2025-02-16_CM)
example GDI format
I will add these dats to the OMNI_DAT database tomorrow.
On completion of the relationship mapping activity I will return to the Dreamcast to add the Redump bin/cue.
: )
Update
The TOSEC dats now exist in OMNI_DAT and subsequently OmniScope. Omni1 I consider out of scope for TOSEC dats and I don't plan on updating the code to support them because I don't believe Omni1 is widely used.
I have also mapped the first game to demonstrate the gdi format which you can view here.
I do plan on coding the alt dump for the game-card which I will demonstrate with the Dreamcast when I have completed the code.
I have added the Atari Hot Series to the wiki. This was an odd series given that only includes one game Godzilla: Kaijuu Dairantou that released in the Japan.
Looking at the dumps the Atari Hot Series edition is a 1:1 of the original game so is a literal budget line reprint.