1 Topic by nocash (edited by nocash 2024-05-08 15:39:40)

  • Registered: 2012-12-14
  • Posts: 17

I've already posted this in http://www.psxdev.net/forum/viewtopic.p … mp;p=23041 but I thought it might be interesting here, too.

Until today, the oldest known LibCrypt'ed CDROM appears to have been MediEvil. But there's another older title...
http://redump.org/disc/45854/ SLED-01340 (EXE date 03 Jun 1998) Net Yaroze Demo Disc (EUR)
http://redump.org/disc/592/ SCES-00311 (EXE date 05 Aug 1998) MediEvil (EUR)
The Yaroze demo disc is clearly reading from the usual LibCrypt sectors, but not exactly as how one might think...

The standard LibCrypt 16bit keys are read using the following sector list:

                 libcrypt_msf_list:   ;standard libcrypt list
                 ;(seek addr at MM:SS:FF-3, and verify addr at MM:SS:FF)
                 ;        MM, SS ,FF-3,FF
800C124C 05020803 db      03h,08h,02h,05h  ;bit15
800C1250 56530903 db      03h,09h,53h,56h  ;bit14
800C1254 10071303 db      03h,13h,07h,10h  ;bit13
800C1258 29261403 db      03h,14h,26h,29h  ;bit12
800C125C 24211503 db      03h,15h,21h,24h  ;bit11
800C1260 49461803 db      03h,18h,46h,49h  ;bit10
800C1264 56532003 db      03h,20h,53h,56h  ;bit9
800C1268 55522103 db      03h,21h,52h,55h  ;bit8
800C126C 17142303 db      03h,23h,14h,17h  ;bit7
800C1270 12092403 db      03h,24h,09h,12h  ;bit6
800C1274 03002503 db      03h,25h,00h,03h  ;bit5
800C1278 28252803 db      03h,28h,25h,28h  ;bit4
800C127C 19163203 db      03h,32h,16h,19h  ;bit3
800C1280 56533303 db      03h,33h,53h,56h  ;bit2
800C1284 51483403 db      03h,34h,48h,51h  ;bit1
800C1288 42393503 db      03h,35h,39h,42h  ;bit0

The Yaroze LibCrypt 16bit key is read using this unusual sector list:

                 libcrypt_msf_list:  ;special net yaroze list
                 ;(seek addr at MM:SS:FF-3, and verify addr at MM:SS:FF)
                 ;        MM, SS ,FF-3,FF
80015D94 05020803 db      03h,08h,02h,05h  ;bit15 (would be usually bit15)
80015D98 04010803 db      03h,08h,01h,04h  ;bit14 (would be usually N/A)
80015D9C 56530903 db      03h,09h,53h,56h  ;bit13 (would be usually bit14)
80015DA0 55520903 db      03h,09h,52h,55h  ;bit12 (would be usually N/A)
80015DA4 10071303 db      03h,13h,07h,10h  ;bit11 (would be usually bit13)
80015DA8 09061303 db      03h,13h,06h,09h  ;bit10 (would be usually N/A)
80015DAC 29261403 db      03h,14h,26h,29h  ;bit9  (would be usually bit12)
80015DB0 28251403 db      03h,14h,25h,28h  ;bit8  (would be usually N/A)
80015DB4 24211503 db      03h,15h,21h,24h  ;bit7  (would be usually bit11)
80015DB8 23201503 db      03h,15h,20h,23h  ;bit6  (would be usually N/A)
80015DBC 49461803 db      03h,18h,46h,49h  ;bit5  (would be usually bit10)
80015DC0 48451803 db      03h,18h,45h,48h  ;bit4  (would be usually N/A)
80015DC4 10071303 db      03h,13h,07h,10h  ;bit3  (would be usually bit13) ;\
80015DC8 09061303 db      03h,13h,06h,09h  ;bit2  (would be usually N/A)   ; again, same as above
80015DCC 29261403 db      03h,14h,26h,29h  ;bit1  (would be usually bit12) ;
80015DD0 28251403 db      03h,14h,25h,28h  ;bit0  (would be usually N/A)   ;/

The key seems to be then used to decrypt another EXE file on the Yaroze disc. To get the correct key, it wants subchannel errors on all six sectors that "would be usually bit15..bit12", and intact subchannel data on all six other sectors. Four sectors are checked twice, giving a total of 16 bits.

Within the yaroze code the 16bit key is simply AAAAh. Or, in stanard LibCrypt notation it would be FC00h (or anything in range FC00h..FFFFh if it should contain further read errors in the "usually bit11..bit0" range).

Alongsides it's storing the 4th letter of the SCEX region code in bit23-16 of the key (but it doesn't actually seem to use those bits for anything, other than displaying them in the "Final key" TTY message).

Running that disc on retail consoles
The Net Yaroze Demo Disc is reportedly working only on Net Yaroze consoles. But I think that's a misconception.
The real problem is that it won't work when burned on CDRs (without taking care of the LibCrypt stuff).

Another problem might be the SCEX region code, if it's the wrong region then it won't work (without modchip).
I don't known which region code is used on the Yaroze Demo disc. The Yaroze console would accept the usual three regions SCEi, SCEA, SCEE, and the special Yaroze-only region code SCEW, in the latter case it would be really not working on any retail consoles (but again, a modchip should fix that problem).

PS.
If there are any further "mysteriously not working" discs from 1998 then it might worth checking if they do also contain similar LibCrypt sectors.

  • Registered: 2019-05-25
  • Posts: 1,221

Thanks for the technical info nocash, very interesting!
From redump perspective I would like to detect this scheme when dumping and preserve subchannel of the altered sectors so we could potentially detect it on the other discs (if any).
I have that disc here.

3 Reply by nocash (edited by nocash 2024-05-08 15:38:58)

  • Registered: 2012-12-14
  • Posts: 17

Cool. I don't know what dumping tools you have, but most should work as-is, or with minor modifications.

1) Dump the sub-channel data from the whole disc, and create a list that shows which subchannel bytes are wrong on which sectors. You are probably already having hardware & software tools that can do that? I don't have an exact sub-channel dump with all data and CRCs for that disc yet.

2) Detecting subchannel patterns. Yeah, something might become visible after dumping.
There are probably only 6 changed sectors on minute 03 (or maybe twice as many if it contains backup copies 5 sectors apart, but even the newer MediEvil didn't have that), and probably no changes on minute 09 (or it would be difficult to dump them, as that'd be ways in the Lead-Out of the yaroze disc). And there might be some other patterns like CRC's being XORed with something other than 0080h.
But if there are other similar old discs, it's hard to predict if they follow the same pattern.

3) Detecting EXE patterns. A simple way would be to scan the boot EXE (or also other EXEs or the whole Track 1.BIN) for values from the old and new "libcrypt_msf_list". Like searching these values in hex editor:

NewLibCrypt:  03 08 02 05 03 09 53 56 03 13 07 10 03 14 26 29 03 15 21 24 03 18 46 49 03 20 53 56 03 21 52 55 03 23 14 17 03 24 09 12 03 25 00 03 03 28 25 28 03 32 16 19 03 33 53 56 03 34 48 51 03 35 39 42
or
OldLibCrypt:  03 08 02 05 03 08 01 04 03 09 53 56 03 09 52 55 03 13 07 10 03 13 06 09 03 14 26 29 03 14 25 28 03 15 21 24 03 15 20 23 03 18 46 49 03 18 45 48 03 13 07 10 03 13 06 09 03 14 26 29 03 14 25 28

That should work in most cases (unless the EXE is compressed, or the bytes are crossing a sector boundary in the BIN file).

4 Reply by superg (edited by superg 2024-05-08 17:33:46)

  • Registered: 2019-05-25
  • Posts: 1,221

Hmm, something doesn't add up here, for Medievil (http://redump.org/disc/592/) I have the following sectors with subchannel errors:

MSF: 03:08:05
MSF: 03:18:49
MSF: 03:20:56
MSF: 03:21:55
MSF: 03:23:17
MSF: 03:25:03
MSF: 03:32:19
MSF: 03:34:51
MSF: 09:20:45
MSF: 09:30:63
MSF: 09:33:37
MSF: 09:35:52
MSF: 09:37:14
MSF: 09:38:58
MSF: 09:46:13
MSF: 09:48:59

They don't match your first table.


However Gekido (http://redump.org/disc/833/) matches your first table:

MSF: 03:08:05
MSF: 03:09:56
MSF: 03:13:10
MSF: 03:14:29
MSF: 03:15:24
MSF: 03:18:49
MSF: 03:20:56
MSF: 03:21:55
MSF: 03:23:17
MSF: 03:24:12
MSF: 03:25:03
MSF: 03:28:28
MSF: 03:32:19
MSF: 03:33:56
MSF: 03:34:51
MSF: 03:35:42

From preservation perspective, we only want to maintain a minimal set of modified subcode Q sectors that are needed to run the game. As most of the Q data is standard and can be easily generated.
We keep this modified data in form of .SBI files that are available for download.

The detection you describe is already implemented in redumper: https://github.com/superg/redumper/blob … x.ixx#L241
Internally I maintain a list of all known libcrypt sectors, two schemes so far:
1. Medievil: 16 sectors with no backup
2. Other games: 16 (Gekido) or 32 sectors with each having a backup 5 sectors apart

For this Yaroze disc I'll likely add a 3rd check.

As of detecting EXE patterns, it's out of our preservation scope.

5 Reply by nocash (edited by nocash 2024-05-08 22:56:25)

  • Registered: 2012-12-14
  • Posts: 17

superg wrote:

Hmm, something doesn't add up here, for Medievil (http://redump.org/disc/592/) I have the following sectors with subchannel errors: ... They don't match your first table.

What do you mean? It looks okay to me.
Mind that the table contains a list of all 16 sectors that could be changed in order to form the 16bit key, but conventionally only 8 sectors are actually changed.

superg wrote:

We keep this modified data in form of .SBI files that are available for download.

I know that SBI format, and it should work fine once when we know which sectors are changed on the yaroze disc.
As long as we don't know that, there's no way around dumping the whole subchannel data in a huge .SUB file - and then we could figure out which bits are changed, unchanged, or just scratched.

NB. I've just noticed that there are also .LSD files that do actually contain the whole 12-byte subchannel data (unlike SBI which had only 10-byte and did lack the CRC16). Nice. Where does .LSD format come from, also from redump?

superg wrote:

As of detecting EXE patterns, it's out of our preservation scope.

It's out of my scope, too. I don't have a large cdrom-image collection, nor a PC with fast harddisk for scanning loads of data.
Anyways, as a starting point, the EXE scanning would be an alternate, easier, and more reliable way to find out which discs need subchannel preservation.
I mean, there are probably some people with super-computers and command line tools, who could just type something like:

search *.bin -hex="03 08 02 05 03 08 01 04 03 09 53 56 03 09 52 55"

And then check if their cdrom collection shows additional matches apart from the yaroze disc.

superg wrote:

The detection you describe is already implemented in redumper: https://github.com/superg/redumper/blob … x.ixx#L241
Internally I maintain a list of all known libcrypt sectors, two schemes so far:
1. Medievil: 16 sectors with no backup
2. Other games: 16 (Gekido) or 32 sectors with each having a backup 5 sectors apart
For this Yaroze disc I'll likely add a 3rd check.

Raw link for older browsers: https://raw.githubusercontent.com/super … x.ixx#L241

I have some comments there : )

Detecting the yaroze disc would work, sure. But detecting any possibly existing similar discs isn't possible without knowing how they might differ exactly. Do they also have 6 changed sectors within the checked 12 sectors? Or maybe less than 6, or more than 6 changed sectors?

LibCrypt does usually read 16 key bits. The MediEvil detection does merely use a hardcoded table with only 8 sectors (plus unused 8 backup sectors). That's not very good.
It won't work with any yet undiscovered discs that use the same LibCrypt version, but with different key bits.
In fact, there are at least three known MediEvil (EUR) versions (english, german, spanish) that do use different keys - but the detection would only work for one of them.

Btw. The figure with "16 or 32" changed sectors doesn't exactly match up with how LibCrypt works:
It's using a 16bit key, which does conventionally have eight "1" bits, which are encoded in 8 changed sectors.
And then there two different kinds of backups:
-- The anti-scratch backup 5 sectors apart (on newer discs).
-- The unused extra backup on minute 09 (on discs with Track 1 larger than 9 minutes).
Depending on which of those backup(s) do exist, LibCrypt should typically have "8 or 16 or 32" changed sectors (in 16 or 32 or 64 possible locations).

Or who-knows-what if Track 1 should happen to end in the middle of the minute 09 backup region. And except for the Yaroze disc, which is apparently having only 6 changed sectors (in 12 possible locations), or maybe more if backups exists somewhere.

6 Reply by superg (edited by superg 2024-05-09 03:01:40)

  • Registered: 2019-05-25
  • Posts: 1,221

NB. I've just noticed that there are also .LSD files that do actually contain the whole 12-byte subchannel data (unlike SBI which had only 10-byte and did lack the CRC16). Nice. Where does .LSD format come from, also from redump?

Hmm, I'm not familiar with .LSD, probably not from redump. I can easily generate anything from redumper output, .SBI is lame, but that's the only way we can have it right now. Without going too much into the details, we currently have no way of improving the website because the communication with our keyholder is literally non-existent for years.

search *.bin -hex="03 08 02 05 03 08 01 04 03 09 53 56 03 09 52 55"

I could do something like this easily on my romset, this is a good idea just to identify potential missing candidates.

LibCrypt does usually read 16 key bits. The MediEvil detection does merely use a hardcoded table with only 8 sectors (plus unused 8 backup sectors). That's not very good.
It won't work with any yet undiscovered discs that use the same LibCrypt version, but with different key bits.
In fact, there are at least three known MediEvil (EUR) versions (english, german, spanish) that do use different keys - but the detection would only work for one of them.

I am well aware of that, all my Medievil EU retail copies have only one encoding scheme, that is hardcoded based on the only discs I have here. I'm very open for a better detection method, please suggest.

As of newer libcrypt protections, as you've already figured out, I have a list of all possible sector numbers (64 different sectors, 32 at minute 3 and backup 32 at minute 9) which might potentially be used by libcrypt and a very strong check that two invalid Q frames 5 sectors apart (anti scratch). The only thing I could potentially change is to account that minute 9 backup might be truncated but I remember I was checking Track 1 sizes for the whole library and I haven't found any candidate for that.

Also, subcode sucks. For minty PS1 discs, I get hundreds of Q crc errors and this is really the best situation because people dump all kind of discs in different conditions. That said, in Net Yaroze case I cannot really rely on 6 other Q sectors to be intact. Let me think on this a bit.

By the way, this is the list of all known libcrypt protected discs: http://redump.org/discs/libcrypt/2

P.S. Let me know if you want/need RAW or processed subchannel for the Net Yaroze disc, I can share.

7 Reply by nocash (edited by nocash 2024-05-09 15:04:24)

  • Registered: 2012-12-14
  • Posts: 17

superg wrote:

Hmm, I'm not familiar with .LSD, probably not from redump. I can easily generate anything from redumper output, .SBI is lame, but that's the only way we can have it right now.

I don't know who has invented that .LSD format, but it is already on redump. For example: on http://redump.org/disc/592/ at top of the page it does show "Download: SHA1 • MD5 • SFV • Cuesheet • SBI" and at the bottom it shows "Sectors with LibCrypt protection" as so "03:08:05    41 01 01 07 06 05 00 23 08 05 ff b8" (which includes the CRC16 bytes "FF B8", which are only found in LSD files, not in SBI files).

superg wrote:

I am well aware of that, all my Medievil EU retail copies have only one encoding scheme, that is hardcoded based on the only discs I have here. I'm very open for a better detection method, please suggest.

Well, get rid of the _LIBCRYPT_SECTORS_MEDIEVIL table, and always use _LIBCRYPT_SECTORS_BASE for everything including MediEvil.
Then there's that "exactly 8 of 16 sectors are changed" convention, and the detection could rely on that (or maybe the LIBCRYPT_SECTORS_COUNT code already does something alike, I understand that part of the code).
If the backups exist, check that each group of 2 or 4 sectors has the same state (all changed or all not changed), for example, look at the table here: http://problemkaputt.de/psxspx-cdrom-pr … bcrypt.htm the 4 sectors for "bit15" should be all having the same state.
If you want to support scratched discs, release the backup rules a bit.
A "1" bit must have errors in all 4 sectors.
A "0" bit shouldn't have errors on any of the 4 sectors.
A "0" bit could have some errors on scratched discs, you could filter them out, show warnings, and raise the warning level depending on the nunber of scratches.

superg wrote:

I remember I was checking Track 1 sizes for the whole library and I haven't found any candidate for that.

Good to know. The two cases with short Track 1 are
-- Minute 9 is in Lead-Out (Net Yaroze), or
-- Minute 9 is in the Audio-Track region (Gekido)
I am wondering if they really don't have Libcrypt backups on minute 9, or if the dumping hardware/software is just unable to find them.

superg wrote:

P.S. Let me know if you want/need RAW or processed subchannel for the Net Yaroze disc, I can share.

Yes! Something like a CloneCD .SUB file should do it. Or better, if you can filter out the intact sectors: Something like an .SBI or .LSD file, or some .TXT file like the list with the "Sectors with LibCrypt protection" on the redump pages (but with ALL scratched or modified sectors, for the whle disc from minute 0 and up).

PS.
There's that myth(?) that the Yaroze Demo disc didn't work on retail consoles. If you have a PSX retail console around, could you give it a try? And mention if it was a EU/US/JP console, with/without modchip.

  • Registered: 2019-05-25
  • Posts: 1,221

I don't know who has invented that .LSD format, but it is already on redump. For example: on http://redump.org/disc/592/ at top of the page it does show "Download: SHA1 • MD5 • SFV • Cuesheet • SBI" and at the bottom it shows "Sectors with LibCrypt protection" as so "03:08:05    41 01 01 07 06 05 00 23 08 05 ff b8" (which includes the CRC16 bytes "FF B8", which are only found in LSD files, not in SBI files).

Ha, don't know how could I miss that download link from redump for all these years wink. Yeah I think it's generated from the libcrypt text data that we paste into the database (same as .SBI).

Well, get rid of the _LIBCRYPT_SECTORS_MEDIEVIL table, and always use _LIBCRYPT_SECTORS_BASE for everything including MediEvil.
Then there's that "exactly 8 of 16 sectors are changed" convention, and the detection could rely on that (or maybe the LIBCRYPT_SECTORS_COUNT code already does something alike, I understand that part of the code).
If the backups exist, check that each group of 2 or 4 sectors has the same state (all changed or all not changed), for example, look at the table here: http://problemkaputt.de/psxspx-cdrom-pr … bcrypt.htm the 4 sectors for "bit15" should be all having the same state.
If you want to support scratched discs, release the backup rules a bit.
A "1" bit must have errors in all 4 sectors.
A "0" bit shouldn't have errors on any of the 4 sectors.
A "0" bit could have some errors on scratched discs, you could filter them out, show warnings, and raise the warning level depending on the nunber of scratches.

Yeah, I could do something like that, hopefully this weekend.

I am wondering if they really don't have Libcrypt backups on minute 9, or if the dumping hardware/software is just unable to find them.

I can check if Gekido has libcrypt backup in audio tracks, have the disc here. Also, just for science, we have a modified ASUS drive firmware that can read all the way into leadout, if there will be some super small Track 1 PAL game, leadout subchannel can be verified as well.

Yes! Something like a CloneCD .SUB file should do it. Or better, if you can filter out the intact sectors: Something like an .SBI or .LSD file, or some .TXT file like the list with the "Sectors with LibCrypt protection" on the redump pages (but with ALL scratched or modified sectors, for the whle disc from minute 0 and up).

Will prepare the file over the weekend.

There's that myth(?) that the Yaroze Demo disc didn't work on retail consoles. If you have a PSX retail console around, could you give it a try? And mention if it was a EU/US/JP console, with/without modchip.

I also know this and I've seen references to SCEW wobble in some of the PsNee ports, but long time ago rama said that it was irrelevant. At some point I had this idea just to mod PSX with a chip that will sniff on a bus where wobble is reported and decode what disc sends, that would greatly help identifying region locks for cases where we don't really know.
Will also check the disc over the weekend as I have to dig out my PAL PSX from some storage boxes.

  • Registered: 2019-05-25
  • Posts: 1,221

nocash wrote:

Yes! Something like a CloneCD .SUB file should do it. Or better, if you can filter out the intact sectors: Something like an .SBI or .LSD file, or some .TXT file like the list with the "Sectors with LibCrypt protection" on the redump pages (but with ALL scratched or modified sectors, for the whle disc from minute 0 and up).

PM with link sent, sorry I didn't notice you asked to filter out good sectors, if that's show stopper let me know.

  • Registered: 2019-05-25
  • Posts: 1,221

Just the heads up, retail US console (not modded) doesn't boot it, tells to insert PlayStation format disc.

  • Registered: 2019-05-25
  • Posts: 1,221

Ok, couple more findings:
Indeed, Gekido has a backup libcrypt key in audio subchannel, http://redump.org/disc/833/ updated.
Net Yaroze disc doesn't have mirror or backup keys, just 6 bits in 6 first libcrypt sectors, http://redump.org/disc/45854/ updated, .SBI or .LSD can be downloaded from there.

I've also updated my redumper libcrypt detection code so it should be a little bit better although theoretically there might be false positives. nocash - thanks for all the hints!

Lastly I have yet to find my PAL PSX to check if the disc boots, chances are low but I'll try to finalize it this week.

12 Reply by superg (edited by superg 2024-06-02 19:30:27)

  • Registered: 2019-05-25
  • Posts: 1,221

PAL PlayStation boots the disc just fine (and it's playable).

JP PlayStation doesn't boot the disc.