Merge program for CD and DVD images (Page 2 of 2)

Are you sure you used the same compressor + settings as the one you used before?

If so, how much bigger is the result?

Is both a h<a>.diff.<b> and h<a>.<b> present after program completion, where <a> is some number, and <b> is some number?

If true, then there was a problem with cleanup midway through program execution. 'h2.2048' (an example of h<a>.<b>) is the sector store for the second image. 'h2.diff.2048' would be the diff from 'h1.2048' to 'h2.2048'. Only one of these need to be stored to be able to rebuild the second image.

Otherwise, is a h<a>.diff.<b> present after program execution?

If false, then the diff stage was found to not be beneficial. In this case, at worst the merged files should only be marginally bigger.

If true, then for at least one image, diffing was deemed beneficial before compression (the 'diff' file was at most 7/8 the size of the 'to' file). It is possible that the compressed 'diff' file is bigger than the compressed 'to' file, but given the initial size difference seems unlikely (it may be true if using freearc on a smallish set of data like gaijin did before, but again this is unlikely to scale to bigger sets/images).

In any event, please be more descriptive in your testing. Let me know the os, files used in the test, files present after execution had completed, the text output to screen during execution, etc. In my tests, the resulting size is at worst about the same as the old result. At best the file size is much smaller.

Resident Evil - CDGroupv0.3.1 vs. CDGroupv0.4.2 - (built-in compression) vs. (my own custom compression)

---------------------------------------------------------------------

6 ngc raw isos = 8.15 GB

6 ngc pakkiso'd = 4.79 GB = 3.36 GB saved over raw iso

6 ngc raw CDGroupv0.3.1 = 4.85 GB = nothing saved over pakkiso

6 ngc compress'd CDGroupv0.3.1 = 3.59 GB = 1.26 GB saved over pakkiso

6 ngc compress'd (built-in compression) CDGroupv0.4.2 = 2.14 GB = 1.45 GB saved over custom compress'd CDGroupv0.3.1

6 ngc raw (uncompress'd) CDGroupv0.4.2 = 2.85 GB = 2.00 GB saved over raw CDGroupv0.3.1

6 ngc compress'd (my own custom compression) CDGroupv0.4.2 = 2.21 GB = nothing saved over built-in compression. You win jamjam. The fact I saved a bit more on TimeSplitters was, it wasn't a good example. I prefer the compression you are using, even if its not the samllest size every single time (bad examples included), it still wins it for me

---------------------------------------------------------------------

Biohazard (Japan) (Disc 1) md5 20CB8D4CB322AA503D1B8A49C43CDEBF

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 2) md5 457944F833FC2F5E8FF394CFDF2E1B7C

Resident Evil (USA) (Disc 2) md5 7DEFD099E98944BC93684D4733BFE68B

Resident Evil (USA) (Disc 1) md5 BDD0FE3848C4AB1441DC6C9EE209426B

Biohazard (Japan) (Disc 2) md5 BFBF8E0F249CF8DD8FCB913793301A8C

Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 1) md5 C581FAB5FD10F55B76188E86194199C1

---------------------------------------------------------------------

CDGroup v0.4.2

CDLibrary v0.4.2


Processing '2048'

 Grouping '2048' (6 files)
  Hashing 'Biohazard (Japan) (Disc 1).iso'
  Hashing 'Biohazard (Japan) (Disc 2).iso'
  Hashing 'Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 1).iso'
  Hashing 'Resident Evil (Europe) (En,Fr,De,Es,It) (Disc 2).iso'
  Hashing 'Resident Evil (USA) (Disc 1).iso'
  Hashing 'Resident Evil (USA) (Disc 2).iso'
  Sorting sectors within images
  Merging image sector hashes
  Counting repeated sectors
  Create map from images to merged files
  Writing 2048.hsn
  Writing '2048.hsm'
  Writing hs1.2048
  Writing hs2.2048
  Writing hs3.2048
  Writing hs4.2048
  Writing hs5.2048
  Writing hs6.2048
 Group of 2048 byte sectors successful

'2048' successfully processed

Doing external diff on 2048 byte/sector files
  Diffing to 'hs2.2048'
  Diffing to 'hs3.2048'
  Diffing to 'hs4.2048'
  Diffing to 'hs5.2048'
  Diffing to 'hs6.2048'
Diff successful

Compressing files


torrent7z_0.9.1beta/Thu Jul 23 03:08:33 2009
using 7-Zip (A) 4.65  Copyright (c) 1999-2009 Igor Pavlov  2009-02-03

Scanning

Creating archive ngc\ngc.7z.tmp

Compressing  ngc\hs1.2048
Compressing  ngc\hs2.diff.2048
Compressing  ngc\hs3.diff.2048
Compressing  ngc\hs4.diff.2048
Compressing  ngc\hs5.diff.2048
Compressing  ngc\hs6.diff.2048
Compressing  ngc\2048.hsm
Compressing  ngc\2048.hsn

Everything is Ok

External compressor seems to have completed successfully

Size of original images: 8759869440 bytes
Size of merged uncompressed files: 5216682706 bytes (~59 % of original images)
Size of merged + diffed files: 3066362234 bytes (~35 % of original images)
Size of merged + diffed + compressed files: 2308632971 bytes (~26 % of original
images)

Time taken to group: 0 hours 10 minutes 0 seconds
Time taken to diff: 0 hours 13 minutes 52 seconds
Time taken to compress: 0 hours 22 minutes 58 seconds

CDGroup completed in 0 hours 46 minutes 51 seconds

Press any key to continue . . .

NOTE:

CDGroupv0.4.2's size of files = CDGroupv0.3.1 before Diffing and before compression (4.85 GB).

CDGroupv0.4.2's size of files after Diffing and before compression = 2.85 GB (42%) smaller after Diffing.

I'm going to have fun with custom compression - on those sizes. or not

Also, zip should not be smaller than 7z, unless maybe zip deals with incompressible material better and the stuff being compressed is mostly incompressible (perhaps xbox padding).

I think your right, but since the Diffing stage, its actually beeter your way, before diffing was introduced your right zip handled the garbage better, in ngc and probably Wii we would have seen this.

Your new way is the best so far, for maximum savings. I am currently re doing all of those ngc to see how much is saved over a larger field.