• From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

Recently, I've been working on improving Dolphin's ability to tell users whether their dumps are good or not. (https://github.com/dolphin-emu/dolphin/pull/7922, if you're interested.) What really would be nice to have is the ability to check whether the user's hash matches the hash in the Redump database, but I'm not sure what the best way to do that would be.

What would be ideal is if Dolphin had a way to map game ID+revision+disc number to a hash, either using a local copy of a database or by querying redump.org through some kind of API. (In case you're not familiar with GameCube/Wii games, the game ID can be obtained from the Serial field in the Redump database.) The intended way of doing something like this seems to be to use the downloadable dat files, but they don't contain game IDs, so all Dolphin would be able to do is check whether the user's hash exists in the database. What this means is that if the user's hash isn't in the database, Dolphin can't tell whether this is because the user has a bad dump or because the user has a dump of a disc that isn't in the database yet. So I would like to avoid using a database that doesn't contain game IDs if possible.

The only currently available alternative I can see that would be able to do what I want is sending HTTP requests to redump.org like a normal user browsing the website and then parsing the results. This works, but having to parse the HTML is annoying and can lead to things breaking if the website changes the structure of its HTML. Does this sound like a good or bad idea, both in terms of whether the HTML structure can be expected to stay the same and whether redump.org is fine with the extra load? (If this is implemented, Dolphin would only connect to redump.org if the user specifically presses the verify button for a game, so I hope the load won't be too high.)

Or is there a possibility of Redump releasing dat files with additional fields (serial, version, etc...) or making a web API available?

  • Registered: 2015-05-23
  • Posts: 390

A problem with this - the serial field is for the disc label serial, not the disc image header serial. It usually matches, but doesn't always. Maybe redump should start cataloging the header serials?

  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

Hiccup wrote:

A problem with this - the serial field is for the disc label serial, not the disc image header serial. It usually matches, but doesn't always. Maybe redump should start cataloging the header serials?

I'm not aware of any discs like that other than http://redump.org/disc/10000/ (which seems like a somewhat suspicious dump since everything beyond the first four bytes apparently is identical to http://redump.org/disc/7249/). Do you have any examples, other than that one?

  • Registered: 2017-09-29
  • Posts: 163

A redump.org API in general would be absolutely great for emulators, ROM Managers, and many other uses. Please implement one!

PX-4824TA (offset +98), PX-755SA (offset +30), ASUS BW-16D1HT (offset +6)

5 Reply by Landcross (edited by Landcross 2019-03-29 12:37:00)

  • Registered: 2018-04-20
  • Posts: 177

I agree that any form of API would be a great, great addition to Redump.

Though, I can imagine there are also drawbacks to a Redump API. It would probably increase the server load (imagine all thousands of Dolphin users querying the API), especially when more rom managers/emulators/launchers will use it. And it would increase the 'risk' of someone scraping Redump and 'stealing' the data. Though, it's of course a point of discussion whether the data is really a property of Redump tongue

Or, maybe not an online API, but an 'offline' one that generates a json/xml file for offline use. Where a user would select which data (maybe even from a limited selection, e.g. only hashes, titles, editions and serials) he/she wants and then Redump generates a file. It would at the very least be much less of a strain on the server.

  • iR0b0t
  • iR0b0t
  • Administrator
  • Offline
  • From: Research equipment
  • Registered: 2008-02-21
  • Posts: 6,703

JosJuice wrote:

is there a possibility of Redump releasing dat files with additional fields (serial, version)

If you want to have it, I can make it happen.

What will you do about discs which don't have a serial or version, though?
http://redump.org/discs/system/gc/sort/serial/?page=1

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)
  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

iR0b0t wrote:
JosJuice wrote:

is there a possibility of Redump releasing dat files with additional fields (serial, version)

If you want to have it, I can make it happen.

That would be very nice. For Dolphin's purposes, the serial and version are the only things that really are needed (in addition to the hashes, of course). The disc number would also be convenient to have, but it can be determined based on the serial.

iR0b0t wrote:

What will you do about discs which don't have a serial or version, though?
http://redump.org/discs/system/gc/sort/serial/?page=1

I don't think there's anything we can do with those unlicensed discs other than checking whether the hash is in the database or not. A lot of those discs even share the name that is displayed in the GC IPL, so there really isn't anything good to go on other than hashes.

  • iR0b0t
  • iR0b0t
  • Administrator
  • Offline
  • From: Research equipment
  • Registered: 2008-02-21
  • Posts: 6,703

Apparently we don't store the version of gamecube and wii discs anymore. I am kind of "delaying" the deletion task for that matter because i don't know yet if we still need it or not.

@JosJuice
Whould the serial tag still be enough for the integration purpose?

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)

9 Reply by JosJuice (edited by JosJuice 2019-07-03 07:53:07)

  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

iR0b0t wrote:

Apparently we don't store the version of gamecube and wii discs anymore. I am kind of "delaying" the deletion task for that matter because i don't know yet if we still need it or not.

The version is being stored, just in a different format. Old 1.01 is equivalent to new Rev 1, old 1.02 is equivalent to new Rev 2, and so on. (But it seems like 1.00 is being replaced with just nothing? Not sure why, but it's not a problem for me, since I can simply treat an empty field as meaning Rev 0.) I'll just need to add an exception for Korean GameCube games with the country code E, since it seems like they now start counting the revision from 0 rather than 48.

iR0b0t wrote:

@JosJuice
Whould the serial tag still be enough for the integration purpose?

Serial + revision would be enough.

EDIT: Actually, a small correction: Due to the exception with Korean GameCube games mentioned above, I would also need information about whether games are Korean or not. But this information is already present in the current datfile if you check whether the game name has (Korea) at the end, so it shouldn't be a problem. It would have been convenient if those revisions still were being counted from 48, though.

10 Reply by johnsanc (edited by johnsanc 2019-07-04 17:57:19)

  • Registered: 2011-03-07
  • Posts: 174

Why not just store the header information as a new field and parse the first few bytes to put in a new field in the dat? That should contain all the info Jos needs AND it adds additional info the database that isnt there for Nintendo systems that many other systems do have.

This can be scraped easily and provided in bulk.

There is A LOT of good parseable information in nintendo disc headers that could be useful:

GameCube: https://www.gc-forever.com/yagcd/chap13.html#sec13.1
Wii: https://wiibrew.org/wiki/Wii_Disc

I believe Jos would only need the highlighted text portion in a new field in the GC/Wii dats:
https://cdn.discordapp.com/attachments/435946976140460037/596390339485499393/unknown.png

  • iR0b0t
  • iR0b0t
  • Administrator
  • Offline
  • From: Research equipment
  • Registered: 2008-02-21
  • Posts: 6,703

http://redump.org/datfile/ixl/serial,version

supported options = serial and/or version

separated by comma, order does not matter.

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)
  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

Thank you!

For integrating this into Dolphin, would you prefer that we ship a copy of the dat file with every release of Dolphin, or would you prefer that Dolphin downloads the dat file from redump.org on behalf of the user? The former would place less of a load on the server, but the latter would enable people to get the hashes for new dumps without having to update Dolphin. (There aren't all that many undumped discs now, though... At least if you don't count demos and all the Korean Wii games.) If I do the latter, I'll make it download only if the user is trying to verify a disc (and with local caching), so the server isn't going to be getting a request for every person who uses Dolphin or anything like that.

Also, if we ship a copy of the dat file with Dolphin, is it fine to ship the Wii dat file too even though it's not publicly available on redump.org?

  • iR0b0t
  • iR0b0t
  • Administrator
  • Offline
  • From: Research equipment
  • Registered: 2008-02-21
  • Posts: 6,703

Yes, you can do both, gamecube & wii, and auto-download whenever needed. Server is powerful enough to handle it.

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)
  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

I've created the Dolphin pull request: https://github.com/dolphin-emu/dolphin/pull/8330

Feel free to unsticky this thread now, since there isn't much left to discuss.

  • Registered: 2009-03-14
  • Posts: 2,621

Jos, anyway dolphin could blast a note about the Wii/gc missings in conjunction with this feature add? We're pretty much at a brick wall for finding rare stuff - such as euro Wii demos, etc.

All my posts and submission data are released into Public Domain / CC0.
  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

I suppose I could add a notice if you try to verify a disc that isn't in redump, but I don't know of a reliable way to distinguish between legit GameCube discs that should be in redump and custom homebrew GameCube discs that shouldn't be in redump. (For Wii discs I can check the signing and hashes.) Do you think I should add it despite that?

  • Registered: 2009-03-14
  • Posts: 2,621

Sure, may as well. I mean when in doubt they could always provide a pic for proof of ownership.

Just blast out the miss lists

GC
Asia - http://wiki.redump.org/index.php?title= … ia_Missing
PAL - http://wiki.redump.org/index.php?title= … AL_Missing
USA - http://wiki.redump.org/index.php?title= … SA_Missing

Wii
Asia - http://wiki.redump.org/index.php?title= … ia_Missing
PAL - http://wiki.redump.org/index.php?title= … AL_Missing
USA - http://wiki.redump.org/index.php?title= … SA_Missing

All my posts and submission data are released into Public Domain / CC0.

18 Reply by JosJuice (edited by JosJuice 2019-08-24 15:15:26)

  • From: Sweden
  • Registered: 2017-10-30
  • Posts: 45

Ah, I didn't think about embedding the miss lists into Dolphin. But some of the games there don't even have known game IDs (mainly the monthly Japanese GC discs), so I don't know if I can use those lists...

Or maybe you just meant I could include links to the miss lists for the user to check?

  • Registered: 2009-03-14
  • Posts: 2,621

Yeah links. But ofc if you can look for missing versions somehow for games that would be cool too.

All my posts and submission data are released into Public Domain / CC0.