Merge program for CD and DVD images (Page 1 of 2)

The tools are jar files, or jar files wrapped in exes. Both require java to be installed.[/strikethrough]

In my opinion having .net dependencies or similar is much more annoying. Java is near ubiquitous. What is it you don't like about java?

I am not understanding what this is for, a better description please?

New version: v0.2.10

Differences from v0.2.1:

• Better memory usage

• Basic support to extract from zip and 7z files when unmerging (does not handle compressed files containing directories)

• Partial sector matching option without having to do multiple passes (not used by default)

• Bug fixes

• Better description

Note: Default behaviour has changed. There is now only one pass by default instead of three. If you want partial sector matching, it is recommended to use partial matching without multiple passes.

CDGroup v0.2.10

Resident Evil 4

---------------------------------------------------------------------

8 ngc raw isos = 10.8 GB

8 ngc pakkiso'd = 6.38 GB = 4.42 GB saved over raw iso

8 ngc raw CDGroup'd = 4.28 GB = 2.1 GB saved over pakkiso

8 ngc compress'd CDGroup'd = 3.07 GB = 1.21 GB saved over raw CDGroup

---------------------------------------------------------------------

Resident Evil 4 (Europe) (En,Fr,De,Es,It) (Disc 1).iso:1459978240:22DB91D40629DF75E63A019E0F9D72F5

Resident Evil 4 (USA) (Disc 2).iso:1459978240:2381ACD2199D6E7566932DF86901903D

Resident Evil 4 (Germany) (En,Fr,De,Es,It) (Disc 1).iso:1459978240:290D0B60E8B4D53A05C478D91A3CCDA2

Resident Evil 4 (Europe) (En,Fr,De,Es,It) (Disc 2).iso:1459978240:41A0F650DD5A80DC8D0C268037FF2320

Biohazard 4 (Japan) (Disc 2).iso:1459978240:499CBEC401C8D80D35B521B215E8D039

Biohazard 4 (Japan) (Disc 1).iso:1459978240:6B57D0AE0B872A50457276FB84D58CF2

Resident Evil 4 (Germany) (En,Fr,De,Es,It) (Disc 2).iso:1459978240:7B6C469694E9BB14EFFEC5D0017E08DD

Resident Evil 4 (USA) (Disc 1).iso:1459978240:CA749757E3B9D119F3FEB1F9F0F81BD7

Star Wars - Rogue Squadron II - Rogue Leader

---------------------------------------------------------------------

5 ngc raw isos = 6.79 GB

5 ngc pakkiso'd = 6.13 GB = 0.66 GB saved over raw iso

5 ngc raw CDGroup'd = 2.40 GB GB = 3.73 GB saved over pakkiso

5 ngc compress'd CDGroup'd = 2.22 GB = 0.18 GB saved over raw CDGroup

---------------------------------------------------------------------

Star Wars - Rogue Squadron II - Rogue Leader (USA) md5 0BF391BEE90DA09D6042016D23E7A9B1

Star Wars - Rogue Squadron II - Rogue Leader (Europe) md5 15C49FF1DC836D19A1F275AC46EF4398

Star Wars - Rogue Squadron II - Rogue Leader (France) md5 776A1046F8C47D7AAD0CD732B53587CA

Star Wars - Rogue Squadron II - Rogue Leader (Spain) md5 BB7EEBBBB11FC8DC97F483A4DC3382CC

Star Wars - Rogue Squadron II - Rogue Leader (Germany) md5 FE32765CE6FCBB67FFD9E6F63D09CAE0

Tiger Woods PGA Tour 2004

---------------------------------------------------------------------

6 ngc raw isos = 8.14 GB

6 ngc pakkiso'd = 4.38 GB = 3.76 GB saved over raw iso

6 ngc raw CDGroup'd = 1.77 GB = 2.61 GB saved over pakkiso

6 ngc compress'd CDGroup'd = 1.17 GB = 0.6 GB saved over raw CDGroup

---------------------------------------------------------------------

Tiger Woods PGA Tour 2004 (Europe) (Disc 1) (v1.00) md5 2AE87A7103E8D24BD66C7908B455A863

Tiger Woods PGA Tour 2004 (USA) (Disc 2) md5 4DF313D253F54222944F382ECE74EF81

Tiger Woods PGA Tour 2004 (USA) (Disc 1) md5 81BA9B53904348A25DB2EE0AE46D3B6C

Tiger Woods PGA Tour 2004 (Europe) (Disc 2) (v1.01) md5 95FD974F151C4A48FD65E3F8C8158419

Tiger Woods PGA Tour 2004 (Europe) (Disc 2) (v1.00) md5 A53EE73D2A01BB136B5CC52703C8E228

Tiger Woods PGA Tour 2004 (Europe) (Disc 1) (v1.01) md5 DC8A8164E37CB0BAFD982A4D30946E8A

Resident Evil

---------------------------------------------------------------------

6 ngc raw isos = 8.15 GB

6 ngc pakkiso'd = 4.79 GB = 3.36 GB saved over raw iso

6 ngc raw CDGroup'd = 4.85 GB = nothing saved over pakkiso

6 ngc compress'd CDGroup'd = 3.59 GB = 1.26 GB saved over pakkiso

---------------------------------------------------------------------

Biohazard (Japan) (Disc 1) md5 20CB8D4CB322AA503D1B8A49C43CDEBF

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 2) md5 457944F833FC2F5E8FF394CFDF2E1B7C

Resident Evil (USA) (Disc 2) md5 7DEFD099E98944BC93684D4733BFE68B

Resident Evil (USA) (Disc 1) md5 BDD0FE3848C4AB1441DC6C9EE209426B

Biohazard (Japan) (Disc 2) md5 BFBF8E0F249CF8DD8FCB913793301A8C

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 1) md5 C581FAB5FD10F55B76188E86194199C1

Mario Party 4

---------------------------------------------------------------------

4 ngc raw isos = 5.43 GB

4 ngc pakkiso'd = 4.37 GB = 1.06 GB saved over raw iso

4 ngc raw CDGroup'd = 2.71 GB = 1.66 GB saved over pakkiso

4 ngc compress'd CDGroup'd = 2.19 GB = 0.52 GB saved over raw CDGroup

---------------------------------------------------------------------

Mario Party 4 (USA) (v1.01) md5 01DE13AD52F1554975E7F316370BA086

Mario Party 4 (Europe) (En,Fr,De,Es,It) (v1.00) md5 5DC6F6949F09DA56FD0313CBDB85FCA5

Mario Party 4 (USA) (v1.00) md5 6EDA0E31DCF1622EF749C5CE8F1196A3

Mario Party 4 (Europe) (En,Fr,De,Es,It) (v1.02) md5 F6740FC00D9818AD1F161889382C7902

New version CDGroup v0.3.2

Differences from v0.3.1

• Old code cleanup. Some options (like deletefiles) removed

• Improve cross-platform compatibility

• Remove J7zip as a 7z decompressor (can't handle some 7z files, presumably solid archives)

• External compressor / decompressor support (t7z.exe currently in there)

Note: You can define your own external compressor / decompressor using options.ini

CDGroup v0.3.2

Looks like you ran out of ram because the piece size was too small for the size of the largest file being grouped. If you didn't actually run out of ram (just the ram java had access to), try executing with the -Xmx flag like so:
java -Xmx1000m -jar CDGroup.jar ... (to give the program 1000MB of ram to work with for example).

Note this is partly why multiple passes are not recommended, nor is an initial pass of anything other than 1. They eat ram when given a large enough file and / or small enough piece size. The standard settings should allow most people to merge most common file sizes (given 512MiB of ram it should easily handle full dual layer DVD sized images). Multiple passes will probably be removed. Initial pass may be kept in for experimentation.

How multiple passes works (and why you shouldn't use it)

The binary result from one pass is the input for the next. The only benefit (compared to changing the initial pass and keeping passes at 1) is decreased overhead (the size of the combined gsm files will be smaller). But multiple passes has disadvantages that far outweigh it:

• After each pass, sectors are more 'mixed' than they were before

• Maximum concurrent ram usage doubles wrt the biggest file after each pass

• The same data is being grouped again and again in a nested fashion

The concurrent ram usage is where there's a real problem. Maximum concurrent ram usage in bytes is estimated as the number of pieces in the biggest file being merged multiplied by 48 (plus some overhead). With normal settings (sector matching), a 1Gib file would have 524288 pieces, taking up roughly 25Mib of ram (non-contiguous). This is fine for any sensible image size.

Taking 2048 as an example, for each successive pass the piece size is halved so the number of pieces the file is split into has doubled. The program ran out of memory at pass '256', where maximum concurrent ram usage would be roughly 8*25 = 200MiB per GiB of the maximum file size. Using passes instead of initialpass makes this worse, as a large file could have been made earlier, increasing concurrent ram usage further.

Freearc testing

As for your freearc testing, the only reason I can see for the result is that the repeated sectors are pulled out of where they are and placed in gs0. This may mess with a compressor as it could put unrelated sectors close together (although the sectors are ordered as naturally as possible within the gs0). Something else to consider is that as the images are small, freearc may have been able to see matching sectors from two images next to each other. It seems unlikely that the freearc performance would scale to bigger images (like full DVD images for example).

There are some things I can see to improve compressibility (or at least the chances of improved compressibility):

• Instead of pulling repeated sectors out to gs0, remove them from everywhere except the first location they are present

• Swap the name with the extension in the resulting files. I know freearc can order files any which way, but t7z orders them by extension, which is possibly the worst way to order them with the files named as they currently are (this would only affect merges with CDs with multiple sector types, which currently are compressed in the order 2048.gs0, 2324.gs0, 2048.gs1, 2324.gs1 ...)

• Store gsm file in a different manner (gsm is the overhead when merging). There's a way to store the gsm such that it is the same size and does the same thing, but is more compressible

If the first is implemented it seems very unlikely that freearc will produce a smaller file from raw images than the merged images.

Anything else that anyone thinks can improve compressibility let me know here or in pm.

It's possible to implement extraction of a single image. However the current direction I'm heading in making the data more compressible means more of the files are needed to extract a single image (the first bullet point on the todo list above). This makes single image extraction fine for uncompressed files, but if they're compressed it's either messy (don't delete extracted working files) or wasteful (extracting multiple single images, deleting the working files each time means multiple decompression of the same file).

Something I'm toying with is having an external diff stage after grouping and before external compression. This could really reduce the benefit of single image extraction (if it takes 90% of the effort to extract a single image, you might as well go the last 10% to get the rest).