Note: Tiger Woods PGA Tour 2004, tried to diff, but no diffs were created, and as I expected, my compression on the old set was identical in size, so nothing gained, nothing lost.

I think your right, but since the Diffing stage, its actually beeter your way, before diffing was introduced your right zip handled the garbage better, in ngc and probably Wii we would have seen this.

Your new way is the best so far, for maximum savings. I am currently re doing all of those ngc to see how much is saved over a larger field.

Also, zip should not be smaller than 7z, unless maybe zip deals with incompressible material better and the stuff being compressed is mostly incompressible (perhaps xbox padding).

---------------------------------------------------------------------

6 ngc raw isos = 8.15 GB

6 ngc pakkiso'd = 4.79 GB = 3.36 GB saved over raw iso

6 ngc raw CDGroupv0.3.1 = 4.85 GB = nothing saved over pakkiso

6 ngc compress'd CDGroupv0.3.1 = 3.59 GB = 1.26 GB saved over pakkiso

6 ngc compress'd (built-in compression) CDGroupv0.4.2 = 2.14 GB = 1.45 GB saved over custom compress'd CDGroupv0.3.1

6 ngc raw (uncompress'd) CDGroupv0.4.2 = 2.85 GB = 2.00 GB saved over raw CDGroupv0.3.1

6 ngc compress'd (my own custom compression) CDGroupv0.4.2 = 2.21 GB = nothing saved over built-in compression. You win jamjam. The fact I saved a bit more on TimeSplitters was, it wasn't a good example. I prefer the compression you are using, even if its not the samllest size every single time (bad examples included), it still wins it for me

---------------------------------------------------------------------

Biohazard (Japan) (Disc 1) md5 20CB8D4CB322AA503D1B8A49C43CDEBF

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 2) md5 457944F833FC2F5E8FF394CFDF2E1B7C

Resident Evil (USA) (Disc 2) md5 7DEFD099E98944BC93684D4733BFE68B

Resident Evil (USA) (Disc 1) md5 BDD0FE3848C4AB1441DC6C9EE209426B

Biohazard (Japan) (Disc 2) md5 BFBF8E0F249CF8DD8FCB913793301A8C

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 1) md5 C581FAB5FD10F55B76188E86194199C1

---------------------------------------------------------------------

CDGroup v0.4.2

CDLibrary v0.4.2


Processing '2048'

 Grouping '2048' (6 files)
  Hashing 'Biohazard (Japan) (Disc 1).iso'
  Hashing 'Biohazard (Japan) (Disc 2).iso'
  Hashing 'Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 1).iso'
  Hashing 'Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 2).iso'
  Hashing 'Resident Evil (USA) (Disc 1).iso'
  Hashing 'Resident Evil (USA) (Disc 2).iso'
  Sorting sectors within images
  Merging image sector hashes
  Counting repeated sectors
  Create map from images to merged files
  Writing 2048.hsn
  Writing '2048.hsm'
  Writing hs1.2048
  Writing hs2.2048
  Writing hs3.2048
  Writing hs4.2048
  Writing hs5.2048
  Writing hs6.2048
 Group of 2048 byte sectors successful

'2048' successfully processed

Doing external diff on 2048 byte/sector files
  Diffing to 'hs2.2048'
  Diffing to 'hs3.2048'
  Diffing to 'hs4.2048'
  Diffing to 'hs5.2048'
  Diffing to 'hs6.2048'
Diff successful

Compressing files


torrent7z_0.9.1beta/Thu Jul 23 03:08:33 2009
using 7-Zip (A) 4.65  Copyright (c) 1999-2009 Igor Pavlov  2009-02-03

Scanning

Creating archive ngc\ngc.7z.tmp

Compressing  ngc\hs1.2048
Compressing  ngc\hs2.diff.2048
Compressing  ngc\hs3.diff.2048
Compressing  ngc\hs4.diff.2048
Compressing  ngc\hs5.diff.2048
Compressing  ngc\hs6.diff.2048
Compressing  ngc\2048.hsm
Compressing  ngc\2048.hsn

Everything is Ok

External compressor seems to have completed successfully

Size of original images: 8759869440 bytes
Size of merged uncompressed files: 5216682706 bytes (~59 % of original images)
Size of merged + diffed files: 3066362234 bytes (~35 % of original images)
Size of merged + diffed + compressed files: 2308632971 bytes (~26 % of original
images)

Time taken to group: 0 hours 10 minutes 0 seconds
Time taken to diff: 0 hours 13 minutes 52 seconds
Time taken to compress: 0 hours 22 minutes 58 seconds

CDGroup completed in 0 hours 46 minutes 51 seconds

Press any key to continue . . .

NOTE:

CDGroupv0.4.2's size of files = CDGroupv0.3.1 before Diffing and before compression (4.85 GB).

CDGroupv0.4.2's size of files after Diffing and before compression = 2.85 GB (42%) smaller after Diffing.

I'm going to have fun with custom compression - on those sizes. or not

I tested Timeslpitters (2 Discs).

When I say I got smaller size with the old version, I meant after packing my old version.

With the new version it made all the files ok, but I was reffering to after your programme packed it into one single 7z.

I guess the reason the old version was smaller was because I used better (more suitable comprssion) torrent7z is know to produce larger files than pakkiso, and I used custom pakkiso compression. I also used torrent zip, as this seemed to get the smallest size on every file except gs0.

I also tested on Metal Gear, the one I chose to leave uncompressed, their was a small saving using the new version as compared to the old version left uncompressed.

I will further test, for sure, If I get the same thing happed on another test old is better than new, I will unpak your single 7z and pakk it using the compression I found worked the best with the old version, and post those findings.

If so, how much bigger is the result?

Is both a h<a>.diff.<b> and h<a>.<b> present after program completion, where <a> is some number, and <b> is some number?

If true, then there was a problem with cleanup midway through program execution. 'h2.2048' (an example of h<a>.<b>) is the sector store for the second image. 'h2.diff.2048' would be the diff from 'h1.2048' to 'h2.2048'. Only one of these need to be stored to be able to rebuild the second image.

Otherwise, is a h<a>.diff.<b> present after program execution?

If false, then the diff stage was found to not be beneficial. In this case, at worst the merged files should only be marginally bigger.

If true, then for at least one image, diffing was deemed beneficial before compression (the 'diff' file was at most 7/8 the size of the 'to' file). It is possible that the compressed 'diff' file is bigger than the compressed 'to' file, but given the initial size difference seems unlikely (it may be true if using freearc on a smallish set of data like gaijin did before, but again this is unlikely to scale to bigger sets/images).

In any event, please be more descriptive in your testing. Let me know the os, files used in the test, files present after execution had completed, the text output to screen during execution, etc. In my tests, the resulting size is at worst about the same as the old result. At best the file size is much smaller.

It seemed I was able to get the size smaller with my own compression and the old version, than this new version was able to get.

• New merged format

• New stage (external diff)

• Eliminated ram usage dependency on maximum image size

• Multiple passes removed when grouping

• Ram limit removed when ungrouping (obsolete)

The new format aims to be more compressible. The differences from the old format are:

• All extensions are now .h* instead of .g* (for clarity)

• There is no longer a repeated sector store (gs0). Instead, a repeated sector is stored in the first location it is present within the image stores (hs1, hs2 ...) (keep related data together)

• hsm files are stored differently (more compressible)

• Sector storage extension and name is swapped (may help some external compressors order sensibly by extension)

The new external diff stage is to remove repeated data that is not aligned to sector boundaries (shifted data for example). For some inputs this stage plays a good role in crunching down the file size, in others the previous grouping stage did most of the leg work.

CDGroup v0.4.2

Something I'm toying with is having an external diff stage after grouping and before external compression. This could really reduce the benefit of single image extraction (if it takes 90% of the effort to extract a single image, you might as well go the last 10% to get the rest).

And a little more tests:

Crisis Core - Final Fantasy VII (EN+US+US+DE+IT+ES+FR+JP+JP) (PSP) (14.6 GB)

FreeArc raw 9 isos = around 7-8 GB (maybe solid not work for big size)

CDGroup + packed  FreeArc = 2.19 GB

maniac version  - EU -> convert patches + packed FreeArc -> US+US+DE+IT+ES+FR+JP+JP = 2.16 GB (I made them early)

FreeArc totally lost

Resident Evil (USA+Europe+Japan) (GameCube) (8.15 GB)

FreeArc raw 6 isos = 3 GB

CDGroup + packed  FreeArc = 2.06 GB

There is a request if possible to make ability to extract one needed file, not unmerge all.

Note this is partly why multiple passes are not recommended, nor is an initial pass of anything other than 1. They eat ram when given a large enough file and / or small enough piece size. The standard settings should allow most people to merge most common file sizes (given 512MiB of ram it should easily handle full dual layer DVD sized images). Multiple passes will probably be removed. Initial pass may be kept in for experimentation.

How multiple passes works (and why you shouldn't use it)

The binary result from one pass is the input for the next. The only benefit (compared to changing the initial pass and keeping passes at 1) is decreased overhead (the size of the combined gsm files will be smaller). But multiple passes has disadvantages that far outweigh it:

• After each pass, sectors are more 'mixed' than they were before

• Maximum concurrent ram usage doubles wrt the biggest file after each pass

• The same data is being grouped again and again in a nested fashion

The concurrent ram usage is where there's a real problem. Maximum concurrent ram usage in bytes is estimated as the number of pieces in the biggest file being merged multiplied by 48 (plus some overhead). With normal settings (sector matching), a 1Gib file would have 524288 pieces, taking up roughly 25Mib of ram (non-contiguous). This is fine for any sensible image size.

Taking 2048 as an example, for each successive pass the piece size is halved so the number of pieces the file is split into has doubled. The program ran out of memory at pass '256', where maximum concurrent ram usage would be roughly 8*25 = 200MiB per GiB of the maximum file size. Using passes instead of initialpass makes this worse, as a large file could have been made earlier, increasing concurrent ram usage further.

Freearc testing

As for your freearc testing, the only reason I can see for the result is that the repeated sectors are pulled out of where they are and placed in gs0. This may mess with a compressor as it could put unrelated sectors close together (although the sectors are ordered as naturally as possible within the gs0). Something else to consider is that as the images are small, freearc may have been able to see matching sectors from two images next to each other. It seems unlikely that the freearc performance would scale to bigger images (like full DVD images for example).

There are some things I can see to improve compressibility (or at least the chances of improved compressibility):

• Instead of pulling repeated sectors out to gs0, remove them from everywhere except the first location they are present

• Swap the name with the extension in the resulting files. I know freearc can order files any which way, but t7z orders them by extension, which is possibly the worst way to order them with the files named as they currently are (this would only affect merges with CDs with multiple sector types, which currently are compressed in the order 2048.gs0, 2324.gs0, 2048.gs1, 2324.gs1 ...)

• Store gsm file in a different manner (gsm is the overhead when merging). There's a way to store the gsm such that it is the same size and does the same thing, but is more compressible

If the first is implemented it seems very unlikely that freearc will produce a smaller file from raw images than the merged images.

Anything else that anyone thinks can improve compressibility let me know here or in pm.

Differences from v0.3.1

• Old code cleanup. Some options (like deletefiles) removed

• Improve cross-platform compatibility

• Remove J7zip as a 7z decompressor (can't handle some 7z files, presumably solid archives)

• External compressor / decompressor support (t7z.exe currently in there)

Note: You can define your own external compressor / decompressor using options.ini

CDGroup v0.3.2

TEST: Lost Kingdoms II (USA+Europe+Japan) (1.36 + 1.36 + 1.36 GB)

Solid 7z raw isos = Lost Kingdoms II (USA+Europe+Japan).7z  -  2678068917 (2.49 GB)

CDGroup + packed 7z = Lost Kingdoms II (USA+Europe+Japan).7z - 2676988096 (2.49 GB)

FreeArc raw isos = Lost Kingdoms II (USA+Europe+Japan).arc     - 1685900893 (1.56 GB)

CDGroup + packed FreeArc = Lost Kingdoms II (USA+Europe+Japan).arc  -  1706886824 (1.58 GB)

If do patches USA->Europe->Japan will be even smaller (maniac version )