Re: Relation to tosec.org

I think the most important reasons for dumping data tracks in RAW 2352 are:

- no Frankenstein dumps like gigadeath said.. the full main channel is dumped.
- you can easily combine the tracks of a dump that were done using the Redump method into a single image using copy /b. Then you have a proper image of the complete cd with correct gaps and offset. This wouldn't be possible with 2048 tracks without converting them to 2352 first.
- the extra error correction data can be used to verify that the track data is correct (without having to dump the track twice).

I really don't know why 2048 would be better than 2352 (it's smaller yes, but what else?). Maybe you could include both 2352 and 2048 checksums in your database, but this would only take you more time.

Re: Relation to tosec.org

Vigi wrote:

I really don't know why 2048 would be better than 2352 (it's smaller yes, but what else?). Maybe you could include both 2352 and 2048 checksums in your database, but this would only take you more time.

That would be a feature request to ClrMame then, wouldn't it...

Re: Relation to tosec.org

Eidolon wrote:
gigadeath wrote:

-to achieve consistency between systems

You consistently dump RAW data tracks for every system, but you do not consistently dump just the user data for every system.

redump.org doesn't "just dump the user data" for any system, so it's a null point. If they did, it will have to be looked on a case-by-case basis on weather a system and/or game needs the extra data. Dumping the raw sectors for anything and everything is simple; the rules don't change for different systems or games.

Eidolon wrote:
gigadeath wrote:

-because if you really hate raw dumps you can convert them later but you can't do the other way around with 100% reliability

I don't "really hate" them, I just question their usefulness for systems which do not require the EDC/ECC data as user data. For these systems, it just produces data overhead and increases the chance for getting bad dumps as explained in my previous post.

I have no idea where you got the idea that it increases the chance of bad dumps, this doesn't make sense. Either the CD drive is just giving you the raw data (redump.org) for you to store, or it's just sending back the user data, but it's still reading it all to do internal checking/correction.

Eidolon wrote:
gigadeath wrote:

-because dumping raw takes only 10 seconds more and not 10 hours

See above, usefulness and data overhead.

The increased time is purely a limitation of your hard disc. The CD drive spins the disc at exactly the same speed (as said above, it's reading the same amount of data) weather you're dumping just user data or all data.

29 (edited by Eidolon 2008-01-22 00:17:23)

Re: Relation to tosec.org

chungy wrote:

redump.org doesn't "just dump the user data" for any system, so it's a null point. If they did, it will have to be looked on a case-by-case basis on weather a system and/or game needs the extra data. Dumping the raw sectors for anything and everything is simple; the rules don't change for different systems or games.

That is an acceptable answer. You should mention it in the mission statement of your website. From redump.org multi-system point of view, this makes sense. I'm just curious though; which systems have MODE1 data tracks where the game actually makes use of the additional EDC/ECC data in that MODE1 track?

chungy wrote:

I have no idea where you got the idea that it increases the chance of bad dumps, this doesn't make sense. Either the CD drive is just giving you the raw data (redump.org) for you to store, or it's just sending back the user data, but it's still reading it all to do internal checking/correction.

Oh, it does make sense. It is very simple when you look at the different layers of error correction.
- for audio tracks as well as MODE2 data tracks (user data = 2352 bytes), there is the CD drive's internal C1/C2 correction mechanisms.
- for MODE1 data tracks (user data = 2048 bytes), there is the drive's internal C1/C2 correction mechanisms PLUS the 304 bytes of EDC/ECC data contained in each RAW sector of 2352 bytes.

So, the more error correction you have available for your chunk of user data, the less chance there is that the user data is wrong.

Of course, the discussion is a bit theoretical:
- If you have a good condition CD, you get reliable RAW data.
- If a drive's internal C1/C2 error correction cannot compensate for read errors in the RAW data, the resulting dump would not be usable for the redump database anyway.

Re: Relation to tosec.org

If you have a horrible drive/destroyed CD, you're going to have a bad result even dumping, let's say, 32 bytes large sectors.

If your drive/disc are good enough to dump 2048 sectors correctly, then they're good enough to dump 2352 sectors correctly too. A reliable drive does the same multipass checking on raw sector as it does on cooked sectors.


As analogy, if you're good at making paper planes you'll do them good either making 23 paper planes or 25 paper planes. If you're horrible at making paper planes, you'll screw up even if you make a single one.

Re: Relation to tosec.org

Eidolon wrote:

'm just curious though; which systems have MODE1 data tracks where the game actually makes use of the additional EDC/ECC data in that MODE1 track?

PlayStation, for one smile Some PC games also have falsified EDC/ECC data for rudimentary copy protection (granted this does protect against the ripping-only-user-data method as was common for bypassing copy protections).

Oh, it does make sense. It is very simple when you look at the different layers of error correction.
- for audio tracks as well as MODE2 data tracks (user data = 2352 bytes), there is the CD drive's internal C1/C2 correction mechanisms.
- for MODE1 data tracks (user data = 2048 bytes), there is the drive's internal C1/C2 correction mechanisms PLUS the 304 bytes of EDC/ECC data contained in each RAW sector of 2352 bytes.

So, the more error correction you have available for your chunk of user data, the less chance there is that the user data is wrong.

I think our problem is a language problem tbh... earlier you were saying it increased the chance of the data being wrong, and now you're saying it decreases it? Other than that, I see no issues with this quote smile

Of course, the discussion is a bit theoretical:
- If you have a good condition CD, you get reliable RAW data.
- If a drive's internal C1/C2 error correction cannot compensate for read errors in the RAW data, the resulting dump would not be usable for the redump database anyway.

Which is why redump.org encourages people to dump CDs of their own even if it's already in the database. You might verify a previous dump, you might find it to be wrong, maybe both are wrong (though if two different CDs produce identical data, it's extremely unlikely both of them are wrong). You might even find out that you have a CD from a completely different manufacture!

Re: Relation to tosec.org

Heya Snake.. Welcome to the forums.. I have to say you defended yourself quite well there. Here's my feedback tongue

Snake wrote:

Again, this doesn't apply to SegaCD. The first audio track pregap will contain silence.

Theoretically yes, but there's a chance that data was moved into the pregap when mastering, so if you don't correct the write offset, there's a chance that data is cut off if the pregap isn't included. (We've seen data in the pregap before write offset correction several times by now, just so you know)

Snake wrote:

Out of interest - is there any reason why you guys are NOT doing this?

Separate tracks have proven to be more convenient - at least for our project. There are a lot of games that have the same audio tracks across different regions and versions, and this wouldn't be visible if we would only list combined images. We've considered adding the combined checksums a couple of times, but we never got around doing that.

Anyway, you got our reasons for dumping 2352 - full main channel, same size for all sector types, easier to combine.. You're right that CDMage should be used to check RAW tracks for errors (except maybe when the checksum matches after dumping twice).

Re: Relation to tosec.org

I still don't see how raw dumping has to contain errors. Eidolon today matched a dump of mine the first time he dumped it. And he probably never used EAC. Dumps I made for both SegaCD and PC matched no problem, even if the discs and the drives are from totally different parts of the world.

Of course there's a probability to get errors, but if the probability to get errors with cooked sectors is 2%, and the probability to get errors with raw sectors is 3%, it doesn't change much. If you compare, cart reading failure is around 15%, yet that didn't stop us (No-Intro) in our cart dumping process.

Note that:
- every dump will need a successive verification anyway, which would sweep away the probability of error; the verification process has to happen no matter the method, even if we'd dump using carrots as disc readers
- as I said cart reading has a much bigger rate of failure than any CD reading method; note that the principle of dumping carts is to read them at least 3 times each, which is more than we require for CDs; CD dumping is relatively safe, and as much safe in raw mode as it is in cooked mode

Re: Relation to tosec.org

Vigi wrote:

Separate tracks have proven to be more convenient - at least for our project. There are a lot of games that have the same audio tracks across different regions and versions, and this wouldn't be visible if we would only list combined images

This I think will bring to parent/clone relationship structure, which is another fair reason to get tracks separated.

Re: Relation to tosec.org

Snake wrote:

The dumps Themabus and I verified together had no such problems at all, and they're full raw dumps. Our dumps matched perfectly

Yes, but what he's saying is that there is a higher chance of errors, and this is absolutely correct. By reading the data as 2048 bytes/sector it allows your drive to use the extra data to actually verify that the user data is correct and/or fix it on the fly. So if the drive reads it and does not report an error, you can guarantee that your data track is correct. By reading as 2352 bytes/sector this is not the case. Running the resulting image through CDmage will inform you of such errors and allow you to attempt to fix them, but why not let your drive do that for you? Your drive can also attempt to re-read the sector if needed - CDmage can't do that.

Does IsoBuster not already re-read problem sectors? I haven't used it on discs with problem sectors yet, but I know that cdrdao in raw-reading mode (btw it can read the subchannel data, if it's of any interest) *will* re-read problem sectors multiple times until it gets a good copy (or correctable, either way...).

Re: Relation to tosec.org

chungy wrote:

Does IsoBuster not already re-read problem sectors? I haven't used it on discs with problem sectors yet, but I know that cdrdao in raw-reading mode (btw it can read the subchannel data, if it's of any interest) *will* re-read problem sectors multiple times until it gets a good copy (or correctable, either way...).

It appears that some drives do this and some don't. I know that my Plextor drive does it.

37 (edited by gigadeath 2008-01-22 01:24:03)

Re: Relation to tosec.org

Vigi wrote:

It appears that some drives do this and some don't. I know that my Plextor drive does it.

Which return us to the point that a good CD-rom drive is needed. A good drive is a prerequisite of proper dumping. A good drive has (and not "should have") no problem reading data in raw mode. No problem at all. If not your drive is not good for the task.

38 (edited by chungy 2008-01-22 01:25:26)

Re: Relation to tosec.org

You've never had to re-insert a cartridge before a game system recognised it? I'd find that hard to believe tongue (this message was to Snake...)

Re: Relation to tosec.org

Snake wrote:

Really, the only thing we don't agree on is whether the audio offset is important. *Provided all the data is there*, I don't think it is - because the time between issuing the 'read' command and the drive actually playing the audio is going to vary WAY more than any audio offset anyway. Plus, if all the data is there, it can be corrected any way you like afterwards.

If I'm not mistaken, the issue is that the method we dislike returns 1/75 of second wrong. You say this data is so little it doesn't matter, we say that this data matters, and that a proper dump should include it.

If it all reduces to that I can safely says that Redump.org will continue the dumping the way it does now.

we want full proper dump -> we care about offsets -> we use EAC... it's pretty simple, really

I understand what you say, but there's A DIFFERENCE IN MENTALITY AT THE START OF THE DISCUSSION. That can't be worked out. Redump.org cares about things you don't care about. It's fine but it doesn't change what we think of our method of dumping.


And BTW, in cart dumping you have to take into account dirty contacts. Rust is a bitch wink

Re: Relation to tosec.org

Snake wrote:
chungy wrote:

You've never had to re-insert a cartridge before a game system recognised it? I'd find that hard to believe tongue (this message was to Snake...)

Not 15% of the time, no tongue

You're good at keeping your carts clean then. 10 years of foggy basement have wrecked mine. Not to the point they're unreadable toe tongue

41 (edited by themabus 2008-01-22 02:14:52)

Re: Relation to tosec.org

Snake wrote:

Really, the only thing we don't agree on is whether the audio offset is important. *Provided all the data is there*, I don't think it is - because the time between issuing the 'read' command and the drive actually playing the audio is going to vary WAY more than any audio offset anyway. Plus, if all the data is there, it can be corrected any way you like afterwards.

but, Snake, it's not just that. be it even 1 byte - you'll get different crc. and besides all cds have already an factory offset, sum that with drive offset, multiply with 4 and you will get the amount of bytes you'll miss. and the thing with Sega cds is that this factory offset can be huge so it's rather an exception if you get to align audio right and can get crcs to mach on some drives - but for this to happen on many drives it would have to be like a ...miracle smile
here, i did some more math:
name|offset(bytes)|silence(b)|difference(b) -you'll miss that on 0 offset drive
nostalgia|7800|1864|~6k
sangokushi 3|3880|1344|~2.5k
jangou w c|19|lucky
cosmic fantasy|7208|676|a lot
gambler j.|1823*4|1760|4x
...i just walked up directories sorted by serial skipping winning post... only 1 cd from 5 has little offset and a lot of silence, but if you don't belive you can teset by yourself - make the images with cdrwin and see if last byte in image is still data, it will often be.

Re: Relation to tosec.org

I know this has nothing to do with Sega CD games but it does have some relivence to this topic. I understand what you are trying to say Snake but there are games like Tomb Raider for PSX whre the last track (track 57) is nothing but silence except for one byte of data near the end of the last sector that can't be read in drives that can't overread into the lead out and that are not reading based on corrected offsets. This one byte also makes a completly different checksum compared to if you just read that track as is and assumed that because you see no more data (or no data at all in this case) that there really is no more data to read.

Re: Relation to tosec.org

In the end it all seems to come down to keeping the first and last audio samples, which is what CDRWin appears to fail at (because there's no offset correction).

@Snake.. I understand from your posts on Eidolon's Inn how offsets aren't really important for your projects, but you have to understand why they are important to us.

First of all it's for documentation. We like the idea of having the audio tracks put back into the position they were before mastering. Even though in the end it only comes down to a few shifted bytes, if you split dumps into separate tracks and store them this way, there can be great benefits to it. For instance for 600mb games with 550mb of identical audio tracks, you can just get the data track and import the audio from another file, without even having to touch a patch tool.

If we would be using TOSEC's method we wouldn't be able to know if a 600 mb game has 550 mb of identical audio in the PAL version compared to the NTSC version (example: http://redump.org/disc/469/ + http://redump.org/disc/475/). Of course you could try to make a patch, but you loose some insight on how these dumps actually differ.

You could just assume that all dumps are different if the disc isn't exactly the same, but at the end that won't help you (at least not if you want to store the cd-images of the dumps somewhere), and here's why: There are plenty of games that only have one difference: the offset. If you would dump these games using TOSEC's method, you would have 2 separate dumps and need 2 dump entries. Using our method there's just 1 dump and 1 entry: http://redump.org/disc/447/ http://redump.org/disc/1777/

If you're planning to include 'intelligent checksums' for both data and audio in your database then I guess you will have the same advantages as our method for games where all audio tracks are identical, but there are also a lot of games where 1 or 2 audio don't match and all the others do. This is why we prefer offset correction and separate tracks. If you don't care about seeing the similarities between two dumps, you can just go ahead with your proposed method, but if you're planning on keeping the cd-images or spreading them around OR if you decide to care about showing people the similarities between dumps, you now know the advantages of our method.

It's not about good or bad. If you actually pay attention to the offsets instead of ignoring them, you are able to remove all the irrelevant differences between a dump and get an ideal dump. Let the offsets help you by separating them from the dump and by storing them as values, not by ignoring that they are there! tongue (the factory write offset is a relevant piece of information about the CD that should be stored in the db).

All tracks have a factory write offset. The only difference is that you don't see them on data tracks, because data isn't read in audio mode (you can still do that and detect the write offset for information purposes using this method: http://forum.redump.org/viewtopic.php?id=2057). The point I'm trying to make here is that what we are really doing here is treating audio the same way the drives are treating data (by removing the offsets that aren't affecting the data track).

Re: Relation to tosec.org

themabus wrote:

here, i did some more math:
name|offset(bytes)|silence(b)|difference(b) -you'll miss that on 0 offset drive
nostalgia|7800|1864|~6k
sangokushi 3|3880|1344|~2.5k
jangou w c|19|lucky
cosmic fantasy|7208|676|a lot
gambler j.|1823*4|1760|4x
...i just walked up directories sorted by serial skipping winning post...

Too bad, I don't have any of these games to test them with CDRWIN. Would be nice if one of you could check and see if CDRWIN really cuts of the last audio data bytes, or simply reads a few sectors into the postgap until there are only zeros.

Re: Relation to tosec.org

Eidolon, the discussion seems to me a bit like a dog trying to bite his own tail now.

We gave you PLENTY of reasons why Redump.org method is preferable to others, in IRC, Redump forums and your own forums. Now that you know these reasons, you only have to decide. You're in the dumpers category now, if you want to submit dumps here I'll be the first being happy, if you choose another way fine, I wish you good luck, but we won't support what we consider an inferior method. If settling to an inferior method was fine to us, we would have settled to dumping for TOSEC ages ago. At the time I didn't consider TOSEC to be a good way and I waited 'till Redump.org gave me the opportunity to do thing the better way. I had to choose too.

As I told you in IRC, for the love of God I only hope all of this isn't an emu politics issue. Maybe you want your own project to attract people to your community, I'm cool with that, but even if you decide to keep the dump info on your site only, I think it would be better to follow Redump.org method. We won't "steal" anything. Using CDRWin only to have it "your own way" sure isn't beneficial to the community.

Re: Relation to tosec.org

No, it's got nothing to do with community politics, and I don't think I'm going in circles.

Like in every discussion, when I get new information (like the comparison I did yesterday between Isobuster+EAC and CDRWIN methods), I have to question again the premise of the REDUMP guide.

For the new questions I have posed now (especially pregap detection), I still did not get sufficient answers, but hope to get them.

I have understood your goals now (or I think I do)
- treat all CDs equal, i.e. RAW mode dumps not caring wether its MODE1/MODE2/CDI/whatever, to have the guide as simple as possible
- normalize audio data by adjusting for the read/write offsets and shifting them in the track dump accordingly
- store data and audio tracks in seperate files and checksum them seperately to provide comparability between different game releases
- anything else I forgot?

If I find out that there is an easier way to achieve the same goals, and it can be proven that it works just as reliably, and it even provides the same results and checksums as the REDUMP method (and thus could be used to submit data to both redump, tosec and whatever else) why not pursue it?

Please understand I am not trying to "rip apart" (no pun intended, hehe) the redump.org project, I am merely joining the discussion.

It's not a matter of choice, it's a matter of how to collaborate.

Re: Relation to tosec.org

Eidolon wrote:

If I find out that there is an easier way to achieve the same goals, and it can be proven that it works just as reliably, and it even provides the same results and checksums as the REDUMP method (and thus could be used to submit data to both redump, tosec and whatever else) why not pursue it?

TOSEC won't change their dumping method. Why would you care about submitting data to them if almost all of their dumps are flawed? (isn't that what we agreed about?)

Of course it would be nice to have a faster and more reliable dumping method, but I don't see how an obsolete tool like cdrwin (can it even detect proper gaps?) can improve things. Even though I like the results of your test, I would still like to see a time comparison that includes splitting the tracks and creating a proper cuesheet etc. until the output it identical to Redump's. Besides, you will always need a drive that supports certain features (overreading, etc) if you want to be sure that the output is correct. I'm still waiting for the PerfectRip successor myself. In the meanwhile, I'll keep my eyes open for an improved method from you and Snake.

Re: Relation to tosec.org

Vigi wrote:

Of course it would be nice to have a faster and more reliable dumping method, but I don't see how an obsolete tool like cdrwin (can it even detect proper gaps?) can improve things.

I wouldn't call CDRWIN obsolete. It is an actively maintained program just as Isobuster and EAC are. And my comparison proves that in theory it is easy to convert a CDRWIN dump to the REDUMP format (and vice versa), because the offset value you get from your Isobuster-clicking-around-between-tracks-1-and-2 is easily calculable with the CDRWIN dump. From that, you can derive the write offset.
It seems noone here in this community has bothered to actually check out CDRWIN properly (maybe my statement is a little premature, let's give your other community members some time to answer...).
And hey I'm NOT working for Goldenhawk technology, before anyone asks...

Vigi wrote:

Even though I like the results of your test, I would still like to see a time comparison that includes splitting the tracks and creating a proper cuesheet etc. until the output it identical to Redump's.

How much time you save in the end depends on the availability and intelligence of the analysis tool I outlined. Of course, the redump disc submission process may have to be adapted as well in order to parse different files.

Vigi wrote:

Besides, you will always need a drive that supports certain features (overreading, C2, etc) if you want to be sure that the output is correct.

Correct.

Vigi wrote:

I'm still waiting for the PerfectRip successor myself.

Depending on how my research goes, the first step towards that that could be a one-click CDRWIN image dump, plus running it once through that "analysis tool".

Of course, one single tool would be better.

Vigi wrote:

In the meanwhile, I'll have my eyes open for the stuff that you and Snake come up with.

Yeah, let's all keep open minds on the matter.

We disagree on the importance of RAW data for systems which don't need it, and the practical importance of normalizing the audio data offset using measured/calculated read/write offset values.

But we agree on the importance of not losing data, have proper pregap detection (ensures proper start of audio tracks) and of individual track checksums to be able to make comparisons.

That's enough common ground, I say.

Re: Relation to tosec.org

I do like CDRWin, I've first used it maybe 6 or 7 years ago (if not longer) and I also used it to dump the gdrom discs that are in the database. However, I don't see how CDRWin is any better than other cue/bin tools like IsoBuster, Fireburner, etc., because they all seem to create the same output.. they're all pretty basic and don't have the features that EAC has when it comes to dealing with errors. (Your Burai disc for instance. It doesn't have 100% track quality everywhere, does it still give the same 'intelligent' audio checksum as EAC?)

Re: Relation to tosec.org

Vigi wrote:

(Your Burai disc for instance. It doesn't have 100% track quality everywhere, does it still give the same 'intelligent' audio checksum as EAC?)

That is a good question. I will check that.