This is horrible disc
We will need a dump with more leadout sectors to get remaining sectors. Either ASUS with Rib's firmware or trap disc.
You are not logged in. Please login or register.
Redump Forum → Posts by superg
This is horrible disc
We will need a dump with more leadout sectors to get remaining sectors. Either ASUS with Rib's firmware or trap disc.
It is nonsense to change behavior depending on CD-i or not.
Good luck and all the best!
I don't know if such rule (valid for CDi) should be applicable to all systems (especially IBM PC)...
This rule is not affecting IBM PC at all.
Small suggestion: is it possible to emit some metadata describing mastering error of a sector - some info describing what's faulty and how it was corrected - for the purpose of being able to reverse repair operation (to have a possibility of creating unmodified dump)?
It's possible but I don't think it's useful for general public. Descrambling is lossless operation and you can always restore any sector how you want, especially if descrambling algorithm is consistent and easy to understand.
If you really care about having unmodified dump, preserve .scram, that will never change.
To:superg
Thanks for the answer. I still have a question.
Redumper unscrambles the sector if sync is broken but msf and mode are normal. Why? Optical drive treats the broken sync sector as error, do not unscramble.
Yes, this again fixes some bad CDi masterings because they actually store data sectors without sync headers.
redumper build_490 fixes the incorrect split for your Heroes 2, feel free to redump and it should match DIC.
01_invalid_sync.0.pass
10_invalid_mode_zeroed_intermediate.0.pass
11_invalid_mode_non_zeroed_intermediate.0.pass
12_invalid_mode_non_zeroed_intermediate_last_byte.0.passCan you explain from the CD specifications why this sector is unscrambled?
The only thing I know of that comes from the CD specification is that if there is sync header and track is data, it needs descramble. Most of these test cases came from bad mastering cases such as early masterings and Philips CDi.
03_mode0_zeroed_data.0.pass
04_mode0_non_zeroed_data.0.pass
05_mode0_non_zeroed_data_last_byte.0.passUser area of the mode 0 sector is all zero. It's CD specifications. But I confirmed that other software also treats as no error sector. I'll fix this case.
Same is about mode0 sectors. Specs say that those should be empty, but in reality I see discs where
16_msf_mismatch_invalid_sync.42.fail
:
:
45_no_msf_invalid_mode_no_intermediate_max.null.failWhat is the "msf_mismatch" and what is the "no_msf"? It seems msf of test data is normal (00:02:00).
Each filename has a number suffix that indicates the expected LBA. When redumper descrambles, it knows the position of of the sector and if it matches, it's the strong check for the successful descramble.
So .42 means that expected LBA of the sector contents is 42, on the other hand .null means that there is no positional information.
For the details see https://github.com/superg/redumper/blob … ts.cc#L187
The bottom line is, DIC never followed CD specs in the first place (always descramble if there is sync header and it's data track). Long time ago I've been checking your descramble code and you have a lot of logic that checks if padding is empty, if mode is correct and some other things. So the current status quo is to have clean dumps without garbage and redumper is doing exactly that. The test cases were created solely to make sure that any subsequent redumper code change is not affecting whole redump.org database.
So, sectors in the range 299-1498 are zeroes when scrambled?
That would explain C2 errors in that range as drive is constantly losing traction reading monotone data sequences like that. Actually that's why scrambling was invented in the first place - to shuffle data around.
I've seen similar issues when trying to read leadout of session 1 of Dreamcast discs. Leadout contains unscrambled data that is printing sega text art on the disc.
C2's is an outcome of a drive not being able to read it.
Is that sector range used in any way by the game?
If not, we could prefill the whole range with 0x55 like we do with pretty much any other case.
To keep things consistent for the time being, let's not change our pattern for a particular protection because it will makes things more complicated. We will have a conversation on what and how we fill in the future.
That disc is shifted 2 seconds into the lead-out, that's 150 sectors.
PLEXTOR can read only 100 lead-out sectors. ASUS with patched Rib's firmware can read unlimited lead-out sectors.
You have to either use audio trap disc with PLEXTOR to trick the drive into reading more or ASUS with patched Rib's firmware and redumper option `--overread-leadout`
PAL PlayStation boots the disc just fine (and it's playable).
JP PlayStation doesn't boot the disc.
Ok, couple more findings:
Indeed, Gekido has a backup libcrypt key in audio subchannel, http://redump.org/disc/833/ updated.
Net Yaroze disc doesn't have mirror or backup keys, just 6 bits in 6 first libcrypt sectors, http://redump.org/disc/45854/ updated, .SBI or .LSD can be downloaded from there.
I've also updated my redumper libcrypt detection code so it should be a little bit better although theoretically there might be false positives. nocash - thanks for all the hints!
Lastly I have yet to find my PAL PSX to check if the disc boots, chances are low but I'll try to finalize it this week.
Just the heads up, retail US console (not modded) doesn't boot it, tells to insert PlayStation format disc.
Yes! Something like a CloneCD .SUB file should do it. Or better, if you can filter out the intact sectors: Something like an .SBI or .LSD file, or some .TXT file like the list with the "Sectors with LibCrypt protection" on the redump pages (but with ALL scratched or modified sectors, for the whle disc from minute 0 and up).
PM with link sent, sorry I didn't notice you asked to filter out good sectors, if that's show stopper let me know.
I don't know who has invented that .LSD format, but it is already on redump. For example: on http://redump.org/disc/592/ at top of the page it does show "Download: SHA1 • MD5 • SFV • Cuesheet • SBI" and at the bottom it shows "Sectors with LibCrypt protection" as so "03:08:05 41 01 01 07 06 05 00 23 08 05 ff b8" (which includes the CRC16 bytes "FF B8", which are only found in LSD files, not in SBI files).
Ha, don't know how could I miss that download link from redump for all these years . Yeah I think it's generated from the libcrypt text data that we paste into the database (same as .SBI).
Well, get rid of the _LIBCRYPT_SECTORS_MEDIEVIL table, and always use _LIBCRYPT_SECTORS_BASE for everything including MediEvil.
Then there's that "exactly 8 of 16 sectors are changed" convention, and the detection could rely on that (or maybe the LIBCRYPT_SECTORS_COUNT code already does something alike, I understand that part of the code).
If the backups exist, check that each group of 2 or 4 sectors has the same state (all changed or all not changed), for example, look at the table here: http://problemkaputt.de/psxspx-cdrom-pr … bcrypt.htm the 4 sectors for "bit15" should be all having the same state.
If you want to support scratched discs, release the backup rules a bit.
A "1" bit must have errors in all 4 sectors.
A "0" bit shouldn't have errors on any of the 4 sectors.
A "0" bit could have some errors on scratched discs, you could filter them out, show warnings, and raise the warning level depending on the nunber of scratches.
Yeah, I could do something like that, hopefully this weekend.
I am wondering if they really don't have Libcrypt backups on minute 9, or if the dumping hardware/software is just unable to find them.
I can check if Gekido has libcrypt backup in audio tracks, have the disc here. Also, just for science, we have a modified ASUS drive firmware that can read all the way into leadout, if there will be some super small Track 1 PAL game, leadout subchannel can be verified as well.
Yes! Something like a CloneCD .SUB file should do it. Or better, if you can filter out the intact sectors: Something like an .SBI or .LSD file, or some .TXT file like the list with the "Sectors with LibCrypt protection" on the redump pages (but with ALL scratched or modified sectors, for the whle disc from minute 0 and up).
Will prepare the file over the weekend.
There's that myth(?) that the Yaroze Demo disc didn't work on retail consoles. If you have a PSX retail console around, could you give it a try? And mention if it was a EU/US/JP console, with/without modchip.
I also know this and I've seen references to SCEW wobble in some of the PsNee ports, but long time ago rama said that it was irrelevant. At some point I had this idea just to mod PSX with a chip that will sniff on a bus where wobble is reported and decode what disc sends, that would greatly help identifying region locks for cases where we don't really know.
Will also check the disc over the weekend as I have to dig out my PAL PSX from some storage boxes.
NB. I've just noticed that there are also .LSD files that do actually contain the whole 12-byte subchannel data (unlike SBI which had only 10-byte and did lack the CRC16). Nice. Where does .LSD format come from, also from redump?
Hmm, I'm not familiar with .LSD, probably not from redump. I can easily generate anything from redumper output, .SBI is lame, but that's the only way we can have it right now. Without going too much into the details, we currently have no way of improving the website because the communication with our keyholder is literally non-existent for years.
search *.bin -hex="03 08 02 05 03 08 01 04 03 09 53 56 03 09 52 55"
I could do something like this easily on my romset, this is a good idea just to identify potential missing candidates.
LibCrypt does usually read 16 key bits. The MediEvil detection does merely use a hardcoded table with only 8 sectors (plus unused 8 backup sectors). That's not very good.
It won't work with any yet undiscovered discs that use the same LibCrypt version, but with different key bits.
In fact, there are at least three known MediEvil (EUR) versions (english, german, spanish) that do use different keys - but the detection would only work for one of them.
I am well aware of that, all my Medievil EU retail copies have only one encoding scheme, that is hardcoded based on the only discs I have here. I'm very open for a better detection method, please suggest.
As of newer libcrypt protections, as you've already figured out, I have a list of all possible sector numbers (64 different sectors, 32 at minute 3 and backup 32 at minute 9) which might potentially be used by libcrypt and a very strong check that two invalid Q frames 5 sectors apart (anti scratch). The only thing I could potentially change is to account that minute 9 backup might be truncated but I remember I was checking Track 1 sizes for the whole library and I haven't found any candidate for that.
Also, subcode sucks. For minty PS1 discs, I get hundreds of Q crc errors and this is really the best situation because people dump all kind of discs in different conditions. That said, in Net Yaroze case I cannot really rely on 6 other Q sectors to be intact. Let me think on this a bit.
By the way, this is the list of all known libcrypt protected discs: http://redump.org/discs/libcrypt/2
P.S. Let me know if you want/need RAW or processed subchannel for the Net Yaroze disc, I can share.
Hmm, something doesn't add up here, for Medievil (http://redump.org/disc/592/) I have the following sectors with subchannel errors:
MSF: 03:08:05
MSF: 03:18:49
MSF: 03:20:56
MSF: 03:21:55
MSF: 03:23:17
MSF: 03:25:03
MSF: 03:32:19
MSF: 03:34:51
MSF: 09:20:45
MSF: 09:30:63
MSF: 09:33:37
MSF: 09:35:52
MSF: 09:37:14
MSF: 09:38:58
MSF: 09:46:13
MSF: 09:48:59
They don't match your first table.
However Gekido (http://redump.org/disc/833/) matches your first table:
MSF: 03:08:05
MSF: 03:09:56
MSF: 03:13:10
MSF: 03:14:29
MSF: 03:15:24
MSF: 03:18:49
MSF: 03:20:56
MSF: 03:21:55
MSF: 03:23:17
MSF: 03:24:12
MSF: 03:25:03
MSF: 03:28:28
MSF: 03:32:19
MSF: 03:33:56
MSF: 03:34:51
MSF: 03:35:42
From preservation perspective, we only want to maintain a minimal set of modified subcode Q sectors that are needed to run the game. As most of the Q data is standard and can be easily generated.
We keep this modified data in form of .SBI files that are available for download.
The detection you describe is already implemented in redumper: https://github.com/superg/redumper/blob … x.ixx#L241
Internally I maintain a list of all known libcrypt sectors, two schemes so far:
1. Medievil: 16 sectors with no backup
2. Other games: 16 (Gekido) or 32 sectors with each having a backup 5 sectors apart
For this Yaroze disc I'll likely add a 3rd check.
As of detecting EXE patterns, it's out of our preservation scope.
Thanks for the technical info nocash, very interesting!
From redump perspective I would like to detect this scheme when dumping and preserve subchannel of the altered sectors so we could potentially detect it on the other discs (if any).
I have that disc here.
Just for the reference, if that's MCN/ISRC between tracks case, it's already fixed in redumper, I am using P channel too.
I will check that later.
So this is a little bit more complicated for Dreamcast because of SD/HD area. Gluing SD and HD area makes a little sense for Dream but we follow the same rules as we set for other systems. Ideally we would want to preserve everything that comes between SD/HD area but there is a current technical limitation in doing that. For instance, SD area leadout contains unscrambled logo data (Sega CD art you can see on data side), I would even want to preserve it as well. We slowly evolve our methods but everything takes time.
The padding you refer to is 150 sectors at the start of Track 2.
At redump when we perform the track split (not only Dreamcast but all systems) we preserve gaps (index 0 entries) for each track. There are multiple reasons for that:
1. sometimes gaps aren't empty and have meaningful data (audio cd for example)
2. this allows for an easy merge of all tracks into one if needed (CloneCD .img) without losing any data.
In general, redump.org strives to be the most precise 1:1 optical media preservation and I don't see why would we favor lossy formats.
Hello,
I'm trying to create a dump for a Playstation 1 game with a PX-712A and the process seems to hit these kinds of errors:[INFO] This drive has 295 offset in the c2. Changed to /s 2. This drive supports [OpCode: 0xd8, SubCode: 0] This drive supports [OpCode: 0xd8, SubCode: 1] This drive supports [OpCode: 0xd8, SubCode: 2] This drive supports [OpCode: 0xd8, SubCode: 8] Checking reading lead-in -> OK Checking SubQ adr (Track) 1/ 1 Checking SubRtoW (Track) 1/ 1 Checking Pregap sync, msf, mode (LBA) -2311 Scanning sector for anti-mod string (LBA) 72425/233476[F:ReadCDForScanningPsxAntiMod][L:2620] GetLastError: 121, Das Zeitlimit f³r die Semaphore wurde erreicht. Please wait for 25000 milliseconds until the device is returned lpCmd: a8, 00, 00, 01, 1a, ea, 00, 00, 00, 02, 00, 00 dwBufSize: 4096 [F:ReadVolumeDescriptor][L:669] GetLastError: 55, Die angegebene Netzwerkressource bzw. das angegebene Gerõt ist nicht mehr verf³gbar. Please wait for 25000 milliseconds until the device is returned lpCmd: a8, 00, 00, 00, 00, 10, 00, 00, 00, 01, 00, 00 dwBufSize: 2048 [F:ReadCDForFileSystem][L:833] GetLastError: 55, Die angegebene Netzwerkressource bzw. das angegebene Gerõt ist nicht mehr verf³gbar. Please wait for 25000 milliseconds until the device is returned
Am I doing something obviously wrong here?
Edit: Tried it with another PS1 game and it fails in a similar fashion.
Use redumper
"Write offset" is set to +670? I think the db needs to be fixed by your hash. Anyway, please contact reentrant.
I totally agree. Fixed dump page (e.g. http://redump.org/disc/99290/) don't need and I want to delete it.
I just want to make sure we are on the same page here. "Fixed Dump" concept is different, it's not "fixed descrambling", it's split according to the offset shift. Check the offset table I posted for http://redump.org/disc/74810. If you split according to these offsets, you will have a fully functional image. Otherwise if you split using only offset from the first track, the image will not work on emulators and when burned in any mode.
(3) - no sync, I'd say it's audio. But I'd add one exception to that rule.
Then, sector with damaged sync is "audio" ok?
Yes, I would agree to that.
Tell me the url of the database of this site.
Sure, I'll check my test dumps and share it.
I think so. Some Sega Saturn and CD-i ready disc (and etc.) have it. (e.g. http://redump.org/disc/58172/, http://redump.org/disc/35804/ )
Yeah, also a couple PSX discs have this data spillover to audio.
You say, "subchannel based split we should use only data/flags from subchannel". Redump.org adopts "subchannel based split" except for TOC vs. Subs desync disc. Then (7),(8) should be descrambled in accordance with subchannel and (4),(5) should not be descrambled in accordance with subchannel.
No I don't think we do what you describe. Maybe it simply wasn't discussed before in detail and we just follow how it was implemented in DIC. TOC/subchannel mismatch is very confusing for everybody and we absolutely have to clarify and formalize it.
Here's what data we collect for cue sheet / track split and where it's available:
1. Count of sessions (available in TOC but can be derived from subchannel too if other session lead-in is included)
2. Count of tracks (available in both TOC/subchannel)
3. Data track flags: data/audio, 4ch, dcp, pre (available in both TOC/subchannel)
4. Track index 01 (available in both TOC/subchannel)
5. Other indices: index 00, index 02+ (available only in subchannel)
6. MCN/ISRC (available in both TOC/subchannel)
7. other CD-TEXT (available only in TOC)
As you can see from this list, TOC and subchannel share almost everything
TOC: (1)(2)(3)(4)(6)(7)
subchannel: (1)(2)(3)(4)(5)(6)
For TOC based split, the primary source of truth is data from TOC.
On the other hand, for the subchannel based split, the source of truth will be data from subchannel.
It's only logical to follow this rule for every data type that we extract as it removes the confusion and separates the concepts.
At redump.org saying that we prefer TOC basically means that all data from TOC should have highest priority.
I agree if admin and other mods agree. I think TOC and SubQ and sync should be checked.
1. Data track on TOC, Data sector on SubQ and sync is valid --- it's apparently "data" and there is no room for discussion.
2. Audio track on TOC, Audio sector on SubQ and no sync --- it's apparently "audio" and there is no room for discussion.
3. Data track on TOC, Data sector on SubQ, but there is not a sync (or sync is damaged) --- It's "data" or "audio"?
4. Data track on TOC, Audio sector on SubQ, there is not a sync (or sync is damaged) --- It's "data" or "audio"?
5. Data track on TOC, Audio sector on SubQ, there is a sync --- It's "data" or "audio"?
6. Audio track on TOC, Audio sector on SubQ, but there is a sync --- It's "data" or "audio"?
7. Audio track on TOC, Data sector on SubQ, there is not a sync (or sync is damaged) --- It's "data" or "audio"?
8. Audio track on TOC, Data sector on SubQ, there is a sync --- It's "data" or "audio"?
Some general things first. I think we should totally separate TOC and subchannel things. Some data tracks are marked audio in TOC and data in subchannel in Photo CD. Some CD-I have hidden track in subchannel that is not listed in TOC. Jaguar often has different track flags in TOC vs subchannel.
As we have two concepts here, primary TOC based split and secondary subchannel based split (Subs Indices).
For TOC based split we should use only data/flags from TOC, for subchannel based split we should use only data/flags from subchannel. The only data we want to use from subchannel for a TOC based split is when we're finding track split points as this data is not present in TOC.
(1),(2) - strongly agree
(3) - no sync, I'd say it's audio. But I'd add one exception to that rule. There are some CD-I discs where whole sync is zeroed but data is scrambled, it's more than one disc, so ideally:
if(standard_sync || zeroed_sync && expected_scrambled_msf) it's data. In general I find MSF is a very handy and strong check for scrambled.
(6) - definitely audio
(4),(5),(7),(8) - this is a source of discrepancies because of TOC subchannel properties mix.
- http://redump.org/disc/74810/ - Does the original disc play on a CD-i? What about a backup of the unfixed dump?
Original definitely works on CD-i. As of unfixed backup - it boils down to whether scrambled image can be burned as is, I'm not too familiar with writing limitatio ns. If you can't write scrambled, burned unfixed dump will not work as it will try to rescramble data track and it will be garbage.
- http://redump.org/disc/99290/ - This has 8.848 errors despite being fixed? What's going on?
All these 8848 sectors are zeroed even after shift correction, this is normal.
There are some clear cases mentioned in the topic where bad mastering is causing dumps to descramble incorrectly and creating tons of erroneous data sectors, because there's some samples missing or added at random positions in a data track. That's the main focus of this topic, right?
Yes, this is correct.
I don't remember if this also makes the original discs non-functional or if the drive performs some sort of on-the-fly correction to output a correct sector?
Yes I think the same. Drive seeks for sync frame, if it's not finding it I think it tries to reposition so these discs work on players and PC. Actually a good experiment would be to dump such disc using BE opcode as data and see what drive returns - will do that.
And as F1ReB4LL was pointing out also on Discord, there seem to be many cases of discs with scrambled data in the Track02 pregap after offset correction, for example: http://redump.org/disc/1770/ + http://redump.org/disc/1716/ + http://redump.org/disc/7986/
And if I remember correctly, this disc http://redump.org/disc/5479/ also has garbage at the start of the audio. If you remove the bytes, the track matches the PS1 track, so it seems to have been ripped from the PS1 version. And IIRC the same was true for Fighting Force PC vs. PS1. But I'm not sure anymore, as it's 13-15 years since those were first dumped, time sure flies
It's unclear whether this is caused by for example the gold master disc being a CD-R that was burned with track-at-once or something, but the most logical explanation is that an audio track was copied with offset garbage and then burned again. But this is a different issue then that we don't have to discuss here?
Exactly, this is known issue to me and it even happens to some official PSX discs. These we shouldn't touch anyways as it's part of the audio.
IIRC Truong / Ripper theorized that erroneous sectors with garbage bytes at the end of a data track were the result of a "split sector" or "half sector" or whatever they called it, that is part data / part audio
If you check the scrambled output, is it data and zeroes interleaved or does the data stop at some position and is it only zeroes after that?
So yes it does look like it's a transition from data to audio e.g. from scrambled to unscrambled, but there are some byte artefacts, I will make some hex screen captures later.
But errors at the end of the data track also seem to be a different issue and since the remainder of the disc is audio tracks, performing offset shift corrections for such discs does not improve the dump in any meaningful way?
So this effect happens also between data tracks (no audio tracks), http://redump.org/disc/74810/ - this disc has only data tracks, second track is CDXA video or something.
There were some examples recently where DIC was leaving sectors scrambled inside a data track with correct sync/header and mostly correct data, resulting in different dumps than before. So the default descrambling behavior must have been changed by sarami at some point or it's a bug. If a sector is inside a data track and the vast majority of it is data, IMO there's no sense in leaving it scrambled and the descrambled data is indeed more meaningful.
Yes I saw that, at some point I think sarami changed something in DIC. This is the most important thing I'm trying to "fix" here. Regardless of applying shift correction, or not - I think we should not rely on sector content (or rely less) when deciding whether we should unscramble sector or not. It's impossible to come up with good decision algo if sector is partially damaged as pretty much any byte can be damaged and this shift issue here clearly demonstrate that.
Redump Forum → Posts by superg
Powered by PunBB 1.4.4, supported by Informer Technologies, Inc.
Currently installed 6 official extensions. Copyright © 2003–2009 PunBB.