1 Topic by LedZeppelin68 (edited by LedZeppelin68 2016-04-05 14:10:58)

  • Registered: 2006-12-22
  • Posts: 105

Hello, let me introduce the tool - GCM Lossless Encoder, now it is NASOS (Not Another Shrinker Or Scrubber)
the purpose is to "optimize" GameCube and Wii images for better archiving
it doesn't delete uncompressable "junk" data, but uses algorithm to reconstruct it
that's why I call it lossless

*this is not shrinker or scrubber*

beta 1:
added batch processing

alpha 4:
fixed wii9 decode (forgot to uncomment in alpha 3)
fixed 2nd GC disc issue without changing format, old .dec files are compatible

alpha 3:
changed the name of utility to "NASOS"
support for Wii dvd images
this is the last alpha, use it only for testing

alpha 2:
works much faster
some statistics added
batch encoding

alpha 1:
first release
support for gamecube dvd images

GameCube TESTs

Animal Crossing (GAFP)
1.34 Gb - RAW image in RAR
26 Mb - optimized image in 7z

Donkey Konga (GKGE)
1.34 Gb - RAW image in RAR
235 Mb - optimized image in 7z

The Legend of Zelda - Four Sword (G4SE)
1.3 Gb - RAW image in RAR
159 Mb - optimized image in 7z

Wii TESTs (All Wii images are around 4.3 GiB in rar)

Resident Evil Archives: Resident Evil Zero (RBHP)
optimized: 2.8 GiB in 7z

Angry Birds Trilogy (SAWE)
optimized: 929 MiB in 7z

Animal Crossing: City Folk (RUUE)
optimized: 370 MiB in 7z

the tool have drag and drop interface
just drop the image or optimized image on it and wait

.NET framework 4.5 needed

Post's attachments

GCMLENCa1.rar 7.68 kb, 44 downloads since 2015-10-26 

GCMLENCa2.rar 8.39 kb, 44 downloads since 2015-10-28 

NASOSalpha3.rar 11.77 kb, 66 downloads since 2015-11-02 

NASOSalpha4.rar 11.41 kb, 34 downloads since 2016-04-04 

NASOSbeta1.rar 12.91 kb, 103 downloads since 2016-04-05 

You don't have the permssions to download the attachments of this post.
  • Registered: 2015-06-30
  • Posts: 47

This is pretty nice for saving space, but it leaves a lot to be desired if there isn't more you can do with an image after it's optimized.

When an image is optimized, it outputs the file with an iso.dec extension. Is this normal, and can it be reversed with anything so it's usable again?

DRD-8120B (+594), LTR-48246S (+6/-6), GE20LU10 (+667/0), GS23N (+667/0), UH12NS30 (+6/?)

3 Reply by LedZeppelin68 (edited by LedZeppelin68 2015-11-07 13:19:03)

  • Registered: 2006-12-22
  • Posts: 105

Oddbrother wrote:

Is this normal, and can it be reversed with anything so it's usable again?

Yes, of course, drop the .DEC file back to utility and you'll get the original one


the utility produces RAW to DEC conversion and DEC to RAW

4 Reply by do_0m (edited by do_0m 2015-11-16 04:43:53)

  • Registered: 2010-04-05
  • Posts: 50

No ****ing way. How did you determine the algorithm? Do you plan on releasing the source code?

My go-to for testing was going to be Animal Crossing, but it seems like I don't have any untouched images lying around. I've already used them for various tests.

So I dug up this and went to work.

Here's the SHA256 for good measure. fa4362802e1ea0b891ce90377e0d122dcb8feae22463409e37488cf72335f9a5

When in the "dec" format the program creates:
Baten Kaitos - Eternal Wings and the Lost Ocean - Special Experience Disc (Japan).iso.dec checksums:
MD5: 623de456a34c1ffeae743fe93d668502
SHA1: e6dd27a614e58bbe46a35d35f8b464c2d0f26858
SHA256: 13f0323a71ba3d4c580af277562b296603b825298e268a5dbc5c7d89675d868d

Now I turned off my PC and left it powered-off for 5 minutes, to ensure the padding / garbage data wasn't residing in memory somewhere. I verified the SHA256 hash once again with Hashcalc. It was still 13f0323a71ba3d4c580af277562b296603b825298e268a5dbc5c7d89675d868d.

Then I re-built the original ISO with the program. By the program's timer, it took 28 seconds. Sounds about right, but I wasn't keeping track.

Every hash matches. Even the SHA256. Unless you can generate SHA256 collisions, or the program is sneakily storing the padding data somewhere else on the PC, it seems like the real deal. (I'll of course test rebuilding the ISO on another PC.) smile

I'm dying to know how this works. One thing to know is that GPack (maybe) and WIT (definitely) have a neat trick to squeeze another ~1-5% out of a Wii disc. As I recall, there are built-in checksums on Wii discs (and maybe GameCube discs too). The checksum algorithm is known, so you can throw them out and simply re-calculate them when rebuilding the original ISO.

5 Reply by Jackal (edited by Jackal 2015-11-16 19:26:52)

  • Jackal
  • Moderator (Retired)
  • Offline
  • Registered: 2006-12-23
  • Posts: 4,132

Wow, so someone figured out the "random" padding structure? Maybe the same can be done for Xbox? tongue

6 Reply by reentrant (edited by reentrant 2016-03-20 12:54:28)

  • Registered: 2014-10-06
  • Posts: 704

The source code is already released. If you're curious how it's done download .net reflector and load the assemblies into it. The code is not obfuscated and it's clearly visible how it's implemented. Having looked at it I'm pretty sure this project is based on a different project. There are two similarly looking function names: a100026e0 and a10002710. Naming convention is probably taken from some disassembler like IDA and the code is probably taken from HexRays and adapted to .net.

So I believe something happened at Nintendo and there was a leak of some x86 binary. The algorithm there is impossible to reverse engineer.

Code from GCMLENCa1.exe:

private void a10002710(regs reg, BinaryReader br, BinaryWriter bw, uint[] stack)
{
    reg.EAX = stack[(int) ((IntPtr) ((reg.ESP + 4) / 4))];
    reg.ECX = 0;
    reg.ESI = 0xadb5c;
    do
    {
        reg.EDX = 0;
        while (reg.EDX < 0x20)
        {
            reg.EAX *= 0x5d588b65;
            reg.EAX++;
            reg.ECX = (reg.ECX >> 1) | (reg.EAX & 0x80000000);
            reg.EDX++;
        }
        bw.BaseStream.Position = reg.ESI;
        bw.Write(reg.ECX);
        reg.ESI += 4;
    }
    while (reg.ESI != 0xadba0);
    br.BaseStream.Position = 0xadb9cL;
    reg.EAX = br.ReadUInt32();
    br.BaseStream.Position = 0xadb5cL;
    reg.ECX = br.ReadUInt32();
    reg.EDX = reg.EAX;
    reg.ECX = reg.ECX >> 9;
    reg.EDX = reg.EDX << 0x17;
    reg.ECX ^= reg.EDX;
    reg.EAX ^= reg.ECX;
    bw.BaseStream.Position = 0xadb9cL;
    bw.Write(reg.EAX);
    reg.EAX = 0xadb60;
    do
    {
        br.BaseStream.Position = reg.EAX - 4;
        reg.ECX = br.ReadUInt32();
        br.BaseStream.Position = reg.EAX;
        reg.EDX = br.ReadUInt32();
        br.BaseStream.Position = reg.EAX + 60;
        reg.ESI = br.ReadUInt32();
        reg.ECX = reg.ECX << 0x17;
        reg.EDX = reg.EDX >> 9;
        reg.ECX ^= reg.EDX;
        reg.ECX ^= reg.ESI;
        bw.BaseStream.Position = reg.EAX + 0x40;
        bw.Write(reg.ECX);
        reg.EAX += 4;
    }
    while (reg.EAX != 0xae340);
    this.a100026e0(ref reg, br, bw);
    this.a100026e0(ref reg, br, bw);
    this.a100026e0(ref reg, br, bw);
    stack[(int) ((IntPtr) (reg.ESP / 4))] = reg.EDI;
    bw.BaseStream.Position = 0xae384L;
    bw.Write(520);
    reg.ESP -= 4;
    stack[(int) ((IntPtr) (reg.ESP / 4))] = reg.ESI;
    reg.ESP += 4;
}

Still it's very nice quality work smile

  • Registered: 2008-09-29
  • Posts: 2,193

I look forward to running some tests smile Awesome

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.

8 Reply by tossEAC (edited by tossEAC 2016-03-22 03:33:04)

  • Registered: 2008-09-29
  • Posts: 2,193

It would be nice if you could add an output directory so the isos can be on one drive and the output files end up on another drive, it would help speed up the converting, as reading from one drive and writing to another gives better speed than reading and writing to the same drive.

Thanks.

Had a re-think, what would be best is drag-n-drop, exactly as it is, nothing changed, but have an option that allows you to set an input and an output folder, for both dec and enc, were it processes all files in the input folder and outputs all files to the output folder, that way you could leave it processing hundreds of files without having to do each file separately, that would speed things up, and would mean you could leave it running while you got some sleep, or watched TV, etc... etc.

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.
  • Registered: 2006-12-22
  • Posts: 105

there is a tool in gc sdk for mastering dvd images. I was surprised, when discovered that it produces this "garbage".
after that, it was only a matter of time to dissassemble it.

a100026e0 and a10002710 - yes, IDA and x64dbg helped

since sdk is available for ages, it's a pity, that there no such tool since

and scene produces so many shrinked, scrubbed, relocated crap.


about XBOX and XBOX360 images, there is also tools in their sdks, but they produces "standard" fatx isos, like those in original xbox scene releases. My thoughts, this iso-container then transfered to MS for final mastering (security sectors, "garbage" padding).

10 Reply by Jackal (edited by Jackal 2016-03-22 07:06:57)

  • Jackal
  • Moderator (Retired)
  • Offline
  • Registered: 2006-12-23
  • Posts: 4,132

LedZeppelin68 wrote:

about XBOX and XBOX360 images, there is also tools in their sdks, but they produces "standard" fatx isos, like those in original xbox scene releases. My thoughts, this iso-container then transfered to MS for final mastering (security sectors, "garbage" padding).

There is also a tool in the SDK for creating a double layer disc layout (it also shows the position of the security placeholders) and iirc it can also generate output images with padding? Although I'm not sure if this is the final padding.

  • Registered: 2006-12-22
  • Posts: 105

this is more recent and cleaned C# code from my new utility

uint blockcount - the full gc image is 1459978240 bytes long, it is divided in 262144 [0x40000] byte blocks
so first block is 0x00, second is 0x01 etc... until the end.

byte[] ID - 4-byte array,  first 4-bytes from image, gamecode

byte DiscNumber - 0x06 byte from image (Disc 1 == 0x00, Disc 2 == 0x01, etc)

        private byte[] GetJunkBlock(uint blockcount, byte[] ID, byte DiscNumber)
        {
            uint[] buffer = new uint[0x824];
            byte[] GarbageBlock = new byte[0x40000];
            int i = 0, j = 0;
            uint sample = 0;

            blockcount = blockcount * 8 * 0x1ef29123;
            while (i != 0x40000)
            {
                if ((i & 0x00007fff) == 0)
                {
                    sample = (((((uint)ID[2] << 0x8) | ID[1]) << 0x10) | ((uint)(ID[3] + ID[2]) << 0x8)) | (uint)(ID[0] + ID[1]);
                    sample = ((sample ^ DiscNumber) * 0x260bcd5) ^ blockcount;
                    a10002710(sample, ref buffer);
                    j = 0x208;
                    blockcount += 0x1ef29123;
                }
                j++;
                if (j == 0x209)
                {
                    a100026e0(ref buffer);
                    j = 0;
                }

                GarbageBlock[i] = (byte)(buffer[j] >> 0x18);
                GarbageBlock[i + 1] = (byte)(buffer[j] >> 0x12);
                GarbageBlock[i + 2] = (byte)(buffer[j] >> 0x8);
                GarbageBlock[i + 3] = (byte)buffer[j];
                i += 4;
            }
            return GarbageBlock;
        }

        private void a10002710(uint sample, ref uint[] buffer)
        {
            uint temp = 0;
            int i = 0;
            while (i != 0x11)
            {
                for (int j = 0; j < 0x20; j++)
                {
                    sample *= 0x5d588b65;
                    temp = (temp >> 1) | (sample++ & 0x80000000);
                }
                buffer[i] = temp;
                i++;
            }

            buffer[0x10] ^= (buffer[0] >> 0x9) ^ (buffer[0x10] << 0x17);

            i = 1;
            while (i != 0x1f9)
            {
                buffer[i + 0x10] = ((buffer[i - 1] << 0x17) ^ (buffer[i] >> 0x9)) ^ buffer[i + 0xf];
                i++;
            }
            for (i = 0; i < 3; i++)
            {
                a100026e0(ref buffer);
            }
        }

        private void a100026e0(ref uint[] buffer)
        {
            int i = 0;
            while (i != 0x20)
            {
                buffer[i] ^= buffer[i + 0x1e9];
                i++;
            }
            while (i != 0x209)
            {
                buffer[i] ^= buffer[i - 0x20];
                i++;
            }
        }
  • Registered: 2006-12-22
  • Posts: 105

Jackal wrote:

There is also a tool in the SDK for creating a double layer disc layout (it also shows the position of the security placeholders) and iirc it can also generate output images with padding?

padding with zeroes

  • Registered: 2008-09-29
  • Posts: 2,193

Just thought I would let you know, Dead Space - Extraction (Europe) (En,Fr,De,It).iso.dec came out a few bytes bigger than as a raw iso...

4.37 GB (4,701,637,632 bytes) from 4.37 GB (4,699,979,776 bytes)

apart from that one, the tool works perfectly, and based on what I have converted, stripping the junk, and then torrent7zing the dec.iso makes savings of around 60% and this was over about 200 isos that I have.

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.

14 Reply by Pikmin (edited by Pikmin 2016-03-30 09:48:23)

  • Registered: 2009-10-21
  • Posts: 477

This is pretty amazing, now the fun of using this on 200+ isos begins, any chance there will be a Linux version?

15 Reply by do_0m (edited by do_0m 2016-04-03 21:41:38)

  • Registered: 2010-04-05
  • Posts: 50

Great tool. If there was a way to put a batch of ISOs through it, and after "compressing" each one, put each .dec file back through the program (in RAM) and verify the reformed ISO matches a certain (Redump) hash, it would be almost perfect. It doesn't have to follow that exact process, but I do want to be certain that the program doesn't distort any discs. There are enough strange GC and Wii ISOs that I can't be sure every single one of them complied with Nintendo's padding standard.

I'm not sure what causes it, but I can't seem to drag+drop on Windows 10 from the file explorer. I have to find an ISO with Everything search (third party program) and drag from there to NASOS.

16 Reply by LedZeppelin68 (edited by LedZeppelin68 2016-04-04 21:12:50)

  • Registered: 2006-12-22
  • Posts: 105

I have updated NASOS to alpha 4

If there was a way to put a batch of ISOs through it

NASOS already support it, just drag several ISOs

the only verification NASOS do is storing md5 of ISO in .dec file, and comparing it against md5 of decoded image

I wanna show you another tool, successor of NASOS, GameCube ISO Merger (do not works with Wii by now)
do_0m, it does all that you want.

the only difference, that besides removing garbage padding, it also removes duplicate data (like Walrus do)
and like Walrus, it merges all region clones of one game in one container

Merger - do batch merges (you can specify output folder, from one hdd to other, speed increase)
Archiver - 7z batch archiver, works with any files
Tester - unmerges in memory and verifies, also use it as batch unmerger
Log - shows info about gcmerge file

I have processed with it ~800 GC isos and ongoing.

the statistic is: 3 isos cannot be decoded properly

Sega Soccer Slam (Japan) - has error in file system
Doshin the Giant and Kyojin no Doshin have errors in garbage, unknown reason

Post's attachments

GameCubeISOMergerMarkI.rar 732.35 kb, 27 downloads since 2016-04-04 

You don't have the permssions to download the attachments of this post.
  • Registered: 2008-09-29
  • Posts: 2,193

Can Nasos have this to... (you can specify output folder, from one hdd to other, speed increase)

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.

18 Reply by Jackal (edited by Jackal 2016-04-05 06:10:42)

  • Jackal
  • Moderator (Retired)
  • Offline
  • Registered: 2006-12-23
  • Posts: 4,132

LedZeppelin68 wrote:

I have processed with it ~800 GC isos and ongoing.

the statistic is: 3 isos cannot be decoded properly

Sega Soccer Slam (Japan) - has error in file system
Doshin the Giant and Kyojin no Doshin have errors in garbage, unknown reason

Maybe the first one is a bad dump? Is there a scene release to compare with?

  • Registered: 2006-12-22
  • Posts: 105

Jackal wrote:
LedZeppelin68 wrote:

I have processed with it ~800 GC isos and ongoing.

the statistic is: 3 isos cannot be decoded properly

Sega Soccer Slam (Japan) - has error in file system
Doshin the Giant and Kyojin no Doshin have errors in garbage, unknown reason

Maybe the first one is a bad dump? Is there a scene release to compare with?

it is not bad dump for sure

I'll show you
GameCube has its own filesystem - FST.

every file in FST have 12-byte record, divided in three 32bit big-endian values:

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00278030                                      00 00 09 70              ...p
00278040  29 01 D0 48 00 27 90 E1                          ).РH.'ђб

0x00000970 0x2901D048 0x002790E1

0x00000970 - first 0x00 means it is a file, (0x01 means a directory)
0x970 - is offset of file name (by the way it's name "soccerslam.dsf")

0x2901D048 - offset in image, where file starts
0x002790E1 - size of file

very simple

0x2901D048 + 0x002790E1 = 0x29296129 must be the end of file, but take a look, there is extra data after it
https://2.downloader.disk.yandex.ru/preview/1cc5599a668560199dbeed72bb440596ff683beea62109db5a38108ccdd7338d/inf/_Bsx5avbMGPoK4jU3Ktl903fJIKVb1wlwnluvaJ7OAZurqjM2kcOVRVFAgKw5EZSdTAPLyvywReV8GWldR4-2w%3D%3D?uid=2070458&amp;filename=sss.png&amp;disposition=inline&amp;hash=&amp;limit=0&amp;content_type=image%2Fpng&amp;tknv=v2&amp;size=1163x755

maybe bad dump, but there is one thing, lets look inside the image, and there is interesting founding

soccerslam.ddf
soccerslam.dif
soccerslam.dlf
soccerslam.dol
soccerslam.dsf

this files used by official makegcm tool to build iso, look inside dif file, it is text file

0x000000c5=0,0x2901d048,0x002790e1,"soccerslam.dsf"
very similiar values

dif file used to make FST, but complete iso made using dsf, which does not have size values, only offset, where file belong
so editing dsf without updating dlf, brings this mastering error

  • Registered: 2006-12-22
  • Posts: 105

by the way, i've updated NASOS, let's call it beta 1, now with batch processing

21 Reply by tossEAC (edited by tossEAC 2016-04-09 23:55:54)

  • Registered: 2008-09-29
  • Posts: 2,193

Is their a small chance you could invent a tool to fix scrubbed isos, that way it would be easier collecting missing Asian dumps in particular, as a lot of the scene releases got scrubbed, rendering them useless to redump collectors, but a scrubbed iso fixing tool, I am guessing would go down a treat smile

Thanks for beta 1, much better to be able to have isos on one drive and the created dec.iso on another, doesn't seem to have any speed increase though, which is strange, but still a good option.

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.
  • Registered: 2008-09-29
  • Posts: 2,193

FOXHOUND Would like to ask you something, about some sort of romulus integration, but he has no way of contacting you. Would you like me to find a way you can get hold of him, I'm sure he has an email address or something.

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.
  • Registered: 2006-12-22
  • Posts: 105

tossEAC wrote:

Is their a small chance you could invent a tool to fix scrubbed isos, that way it would be easier collecting missing Asian dumps in particular, as a lot of the scene releases got scrubbed, rendering them useless to redump collectors, but a scrubbed iso fixing tool, I am guessing would go down a treat

it's possible, but with very few scene isos sad

the kinds of scene isos:

1) 1:1 clean isos
2) scrubbed, simply junk removed (GCtool from paradox) (possible to restore, since FST is untouched)
3) shrinked, junk removed + rellocated files to make smaller iso (not possible, FST is rebuilded)
4) scrubbed and relocated files (starcube group release) (inpossible to recover, FST is rebuilded)

I have already such tool, but there is a very small chance of success
tossEAC, which Asian dumps are missing?

FOXHOUND Would like to ask you something, about some sort of romulus integration, but he has no way of contacting you. Would you like me to find a way you can get hold of him, I'm sure he has an email address or something.

Sure, how can I contact?

  • Registered: 2008-09-29
  • Posts: 2,193

I actually meant Asia Wii games.

He who controls the SPICE... controls the UNIVERSE!
The SPICE must flow.
  • Registered: 2006-12-22
  • Posts: 105

tossEAC wrote:

I actually meant Asia Wii games.

there is Wii scrubbed releases too? OMG, I thought GC was a good lesson...

so, which one to test?