New version, v0.0.4.

  • Reads hsa files (the new merged format)

  • Log file now created in directory containing jar file, instead of wherever it was executed from

CDGDatCheck v0.0.4

New in v0.4.2:

  • New merged format

  • New stage (external diff)

  • Eliminated ram usage dependency on maximum image size

  • Multiple passes removed when grouping

  • Ram limit removed when ungrouping (obsolete)

The new format aims to be more compressible. The differences from the old format are:

  • All extensions are now .h* instead of .g* (for clarity)

  • There is no longer a repeated sector store (gs0). Instead, a repeated sector is stored in the first location it is present within the image stores (hs1, hs2 ...) (keep related data together)

  • hsm files are stored differently (more compressible)

  • Sector storage extension and name is swapped (may help some external compressors order sensibly by extension)

The new external diff stage is to remove repeated data that is not aligned to sector boundaries (shifted data for example). For some inputs this stage plays a good role in crunching down the file size, in others the previous grouping stage did most of the leg work.

CDGroup v0.4.2

It's possible to implement extraction of a single image. However the current direction I'm heading in making the data more compressible means more of the files are needed to extract a single image (the first bullet point on the todo list above). This makes single image extraction fine for uncompressed files, but if they're compressed it's either messy (don't delete extracted working files) or wasteful (extracting multiple single images, deleting the working files each time means multiple decompression of the same file).

Something I'm toying with is having an external diff stage after grouping and before external compression. This could really reduce the benefit of single image extraction (if it takes 90% of the effort to extract a single image, you might as well go the last 10% to get the rest).

Looks like you ran out of ram because the piece size was too small for the size of the largest file being grouped. If you didn't actually run out of ram (just the ram java had access to), try executing with the -Xmx flag like so:
java -Xmx1000m -jar CDGroup.jar ... (to give the program 1000MB of ram to work with for example).

Note this is partly why multiple passes are not recommended, nor is an initial pass of anything other than 1. They eat ram when given a large enough file and / or small enough piece size. The standard settings should allow most people to merge most common file sizes (given 512MiB of ram it should easily handle full dual layer DVD sized images). Multiple passes will probably be removed. Initial pass may be kept in for experimentation.

How multiple passes works (and why you shouldn't use it)

The binary result from one pass is the input for the next. The only benefit (compared to changing the initial pass and keeping passes at 1) is decreased overhead (the size of the combined gsm files will be smaller). But multiple passes has disadvantages that far outweigh it:

  • After each pass, sectors are more 'mixed' than they were before

  • Maximum concurrent ram usage doubles wrt the biggest file after each pass

  • The same data is being grouped again and again in a nested fashion

The concurrent ram usage is where there's a real problem. Maximum concurrent ram usage in bytes is estimated as the number of pieces in the biggest file being merged multiplied by 48 (plus some overhead). With normal settings (sector matching), a 1Gib file would have 524288 pieces, taking up roughly 25Mib of ram (non-contiguous). This is fine for any sensible image size.

Taking 2048 as an example, for each successive pass the piece size is halved so the number of pieces the file is split into has doubled. The program ran out of memory at pass '256', where maximum concurrent ram usage would be roughly 8*25 = 200MiB per GiB of the maximum file size. Using passes instead of initialpass makes this worse, as a large file could have been made earlier, increasing concurrent ram usage further.

Freearc testing

As for your freearc testing, the only reason I can see for the result is that the repeated sectors are pulled out of where they are and placed in gs0. This may mess with a compressor as it could put unrelated sectors close together (although the sectors are ordered as naturally as possible within the gs0). Something else to consider is that as the images are small, freearc may have been able to see matching sectors from two images next to each other. It seems unlikely that the freearc performance would scale to bigger images (like full DVD images for example).

There are some things I can see to improve compressibility (or at least the chances of improved compressibility):

  • Instead of pulling repeated sectors out to gs0, remove them from everywhere except the first location they are present

  • Swap the name with the extension in the resulting files. I know freearc can order files any which way, but t7z orders them by extension, which is possibly the worst way to order them with the files named as they currently are (this would only affect merges with CDs with multiple sector types, which currently are compressed in the order 2048.gs0, 2324.gs0, 2048.gs1, 2324.gs1 ...)

  • Store gsm file in a different manner (gsm is the overhead when merging). There's a way to store the gsm such that it is the same size and does the same thing, but is more compressible

If the first is implemented it seems very unlikely that freearc will produce a smaller file from raw images than the merged images.

Anything else that anyone thinks can improve compressibility let me know here or in pm.

New version CDGroup v0.3.2

Differences from v0.3.1

  • Old code cleanup. Some options (like deletefiles) removed

  • Improve cross-platform compatibility

  • Remove J7zip as a 7z decompressor (can't handle some 7z files, presumably solid archives)

  • External compressor / decompressor support (t7z.exe currently in there)

Note: You can define your own external compressor / decompressor using options.ini

CDGroup v0.3.2

CDGDatCheck (dat check program for CDGroup merged files)

Checks a set of merged files (merged with CDGroup) against a standard dat (DTD or CLR). The hsa files have to be available (uncompressed) for the check. Doesn't uncompress/unmerge the merged set, just reads the hsa files for a quick check.

  • Dat file parameter can be uncompressed, or compressed in a .zip file

  • Optionally pass a zip containing cues, treated as part of the merged set

  • -c option to ignore cues in merged set and dat

Usage: Requires java to run. Can be used as supplied in three ways:

  • Use jar file directly
    java -jar CDGDatCheck.jar dir_to_merge dat_file cue_zip

  • Use bat file (windows)
    CDGDatCheck.bat dir_to_merge dat_file cue_zip

  • Use sh file (linux)
    CDGDatCheck.sh dir_to_merge dat_file cue_zip

CDGDatCheck v0.0.4

New version, v0.3.1

  • New file type, GSA (stores metadata of original images)

  • GSN files now generic, GSA is the only place actual titles are stored

  • Improved 7z extraction, extract direct to required location

Note: The new file type means the merged files are slightly different to those made with v0.2.x. The new format makes the gsn and def files generic. Now you only need to edit the gsa file (and any cues/gdis if they exist) for renaming images, and an extra hash check is performed for validation.

This is probably the final format of the merged files. I can't see many reasons to change it from here.

CDGroup v0.3.1

New version: v0.2.10

Differences from v0.2.1:

  • Better memory usage

  • Basic support to extract from zip and 7z files when unmerging (does not handle compressed files containing directories)

  • Partial sector matching option without having to do multiple passes (not used by default)

  • Bug fixes

  • Better description tongue

Note: Default behaviour has changed. There is now only one pass by default instead of three. If you want partial sector matching, it is recommended to use partial matching without multiple passes.

CDGroup v0.2.10

@tossEAC
In a single pass, the data is split into pieces. Any identical pieces are stored only once, with another file listing all the places the piece is duplicated. Large zeroed files will be reduced to a single piece with a large list of locations.

If by false lba you mean the file system describes files beyond the size limit of the image, this will have no effect as the file system is not read.

@Nexy
If you have many related images, you can merge them with this to save space. Many regional variants of the same game for example would be a good candidate for merging.

The tools are jar files, or jar files wrapped in exes. Both require java to be installed.[/strikethrough]

In my opinion having .net dependencies or similar is much more annoying. Java is near ubiquitous. What is it you don't like about java?

CDGroup merges related disc images to reduce combined file size.

This program takes a directory of images, processing them to remove redundancies to save space. It does this in four stages:

  • Stage 1: Split CD images (2352 byte/sector images) into their component parts. Model non-data parts where possible (ecc, edc, etc)

  • Stage 2: Group each data sector size to remove repeated aligned data (de-duplicate sectors)

  • Stage 3: Diff grouped files to remove repeated non-aligned data

  • Stage 4: Compress resulting files

Notes:

  • Stage 1 takes advantage of redundancies in the CD format

  • Stage 2 takes advantage of aligned redunduncies across related images

  • Stage 3 takes advantage of non-aligned redunduncies across related images

  • Stage 4 takes advantage of uncompressed data in the images

  • Each stage acts on the output of the previous stage

Features:

  • Reduces storage footprint by de-duplication of sectors (unique sectors are only stored once)

  • CDs, DVDs and Blu-Rays can be merged (any .iso with 2048 bytes/sector, and CDs .bin with 2352 bytes/sector)

  • Smart handling of 2352 byte/sector CD images (recognises audio, M0, M1, M2, M2F1, M2F2 sector types. Removes non-data portions of sectors such as ecc if it can be regenerated losslessly)

  • A mixture of 2352 and 2048 byte/sector images can be merged together

  • Identical input images should result in identical merged files

  • Images represented within the merged files can be renamed without unmerging (names stored in a text file *.hsa)

  • Supports merging combined image size of up to 4TiB

  • External diff support (jojodiff)

  • External compressor / decompressor support (7z)

  • Settings chosen to try and balance size reduction with practical time / hardware constraints


Usage: Requires java to run. Can be used as supplied in three ways:

  • Use jar files directly
    java -jar CDGroup.jar dir_to_merge
    java -jar unCDGroup.jar hsa_file

  • Use bat files (windows)
    CDGroup.bat dir_to_merge
    unCDGroup.bat hsa_file

  • Use sh files (linux)
    CDGroup.sh dir_to_merge
    unCDGroup.sh hsa_file

Note: v0.4.x marks a new format, different to that of v0.3.x.

CDGroup v0.4.2

Thank you, much quicker than I expected. Which simple vol 51 should be used, and if the shinobi alt. title is to be used is it complete just missing a )?

Can someone write out the normal titles and alternative titles if applicable please. Thanks.

http://img585.imageshack.us/img585/4854/sam1557w.jpg

http://img856.imageshack.us/img856/8836/sam1558z.jpg

http://img189.imageshack.us/img189/1659/sam1561e.jpg

http://img215.imageshack.us/img215/2267/sam1563r.jpg

http://img703.imageshack.us/img703/6108/sam1565b.jpg

http://img846.imageshack.us/img846/1889/sam1567a.jpg

http://img194.imageshack.us/img194/7319/sam1569.jpg

http://img824.imageshack.us/img824/8445/sam1572wb.jpg

http://img833.imageshack.us/img833/426/sam1574n.jpg

http://img834.imageshack.us/img834/1791/sam1576g.jpg

http://img854.imageshack.us/img854/469/sam1578.jpg

http://img706.imageshack.us/img706/7610/sam1580.jpg

http://img15.imageshack.us/img15/9908/sam1582c.jpg

http://img525.imageshack.us/img525/439/sam1584b.jpg

http://img403.imageshack.us/img403/3964/sam1586o.jpg

http://img827.imageshack.us/img827/8335/sam1588.jpg

http://img14.imageshack.us/img14/817/sam1590.jpg

@JackIngles
Follow the redump dreamcast guide here, and either use CDRwin as it says, or use DCDumper in its place.

@amarok
I can add a flag to let you dump between any two sector ranges, will it be useful though? Data tracks are aligned because the drive does it automatically, what you suggest is to dump as audio, then descramble + align in software. This can be done (ice does it for dense gd-rom session, maybe it will work generally), but if ice can't something else will have to be found which does.

BTW I will get round eventually to a 0.43a with some additions, been busy.

It would require a rewrite, as the sections are currently stored in an array as the size is known from the parameters passed to it. The benefit is minimal, as all you're doing is not rereading good sectors around a bad sector, and with a granularity of -i500 this doesn't mean much. You would only see a good benefit if using a high interval on a tricky disc.

Also, if the program gets to the end and says clean the disc (or you've closed it before it's finished execution), there's no need to delete the files it's made. Run the program again and it will continue where it left off (be sure to pass the same -i and -c parameters, and leave every file where it is). Any good sections won't be dumped again, and the successful reads of undumped sections will be used to try to get a match. Like in the above log file for house of the dead 2.

edit: Yes smile

@tossEAC
That's a relief, I'm still trying to dump mine with CDRwin as it doesn't want to play nice sad
Ring is "T-36901M-0145 MT B01"

@amarok
Thinking about it, if you used a cut-off sector with 0.3a-0.41a, it would produce a bad dump even if you used -i26. Those versions are toxic.

Putting all first checksums into an initial file would be annoying to implement given how it works now. How about the threshold for the interval increases instead? A minimum interval of 260 sectors would produce a maximum of 1940 hash files, less if a cut-off is used. Will keep the program flexible (~0.6MB per section is small enough for those tricky discs imo), and minimise undesired behaviour.

Here's the logs of some successful reads (matching redump), to show what I've seen using 0.42a with a "TSSTcorp DVD-ROM SH-D162C".
Observations:

  • Retrying coupled with fake reads seems to work in many cases

  • Step back doesn't seem to help much, may set the default to false next time unless others have positive results

  • Persistence pays off. The fourth and 12th successful read of 446261-549150  for house of the dead 2 matched, and the resulting track matched redump

tossEAC if you're resuming from a dump made with 0.3a-0.41a, just delete and restart, what was dumped is likely junk. Also, can you retry dumping this please? Mine matches except for the last track, which has an md5 hash of d564c50937f50a051038b0e62a8bb1f4.


Chu-chu Rocket (JPN)

---
 DCdumper.exe d -c446261 -df 

Handle acquired.
Load disc: Done.
Sector map created.

..................:::::::::::::::: PASS 1 ::::::::::::::::..................
Reading section 1: 044990-055278 - Initial dump.
Reading section 2: 055279-065567 - Initial dump.
Reading section 3: 065568-075856 - Initial dump.
Reading section 4: 075857-086145 - read error.
Fake read. Retry - Initial dump.
Reading section 5: 086146-096434 - Initial dump.
Reading section 6: 096435-106723 - Initial dump.
Reading section 7: 106724-117012 - Initial dump.
Reading section 8: 117013-127301 - Initial dump.
Reading section 9: 127302-137590 - Initial dump.
Reading section 10: 137591-147879 - Initial dump.
Reading section 11: 147880-158168 - Initial dump.
Reading section 12: 158169-168457 - Initial dump.
Reading section 13: 168458-178746 - Initial dump.
Reading section 14: 178747-189035 - Initial dump.
Reading section 15: 189036-199324 - Initial dump.
Reading section 16: 199325-209613 - Initial dump.
Reading section 17: 209614-219902 - Initial dump.
Reading section 18: 219903-230191 - read error.
Fake read. Retry - Initial dump.
Reading section 19: 230192-240480 - Initial dump.
Reading section 20: 240481-250769 - Initial dump.
Reading section 21: 250770-261058 - Initial dump.
Reading section 22: 261059-271347 - Initial dump.
Reading section 23: 271348-281636 - Initial dump.
Reading section 24: 281637-291925 - Initial dump.
Reading section 25: 291926-302214 - Initial dump.
Reading section 26: 302215-312503 - Initial dump.
Reading section 27: 312504-322792 - Initial dump.
Reading section 28: 322793-333081 - Initial dump.
Reading section 29: 333082-343370 - Initial dump.
Reading section 30: 343371-353659 - Initial dump.
Reading section 31: 353660-363948 - Initial dump.
Reading section 32: 363949-374237 - Initial dump.
Reading section 33: 374238-384526 - Initial dump.
Reading section 34: 384527-394815 - Initial dump.
Reading section 35: 394816-405104 - Initial dump.
Reading section 36: 405105-415393 - read error.
Fake read. Retry - Initial dump.
Reading section 37: 415394-425682 - Initial dump.
Reading section 38: 425683-435971 - Initial dump.
Reading section 39: 435972-446260 - Initial dump.
Reading section 40: 446261-549150 - Initial dump.

..................:::::::::::::::: PASS 2 ::::::::::::::::..................
Reading section 1: 044990-055278 - read error.
Fake read. Retry - MATCH: a92dd9e9699a8379ff6c44af9a38a7e8
Reading section 2: 055279-065567 - MATCH: f0538b6cbcf5f9f61dfac93557f9f4c8
Reading section 3: 065568-075856 - MATCH: a2a1f0d21d31843db2d27821d34407e7
Reading section 4: 075857-086145 - Non-Match.
Reading section 5: 086146-096434 - Non-Match.
Reading section 6: 096435-106723 - read error.
Fake read. Retry - Non-Match.
Reading section 7: 106724-117012 - MATCH: faea304d78d0f14b12eeec13a20575b0
Reading section 8: 117013-127301 - MATCH: a6108f55cc1619d7976d03b5f91e3ef2
Reading section 9: 127302-137590 - Non-Match.
Reading section 10: 137591-147879 - MATCH: 154762f11fb0da7e4c8c98929c84580d
Reading section 11: 147880-158168 - MATCH: 79a614f30a2423ff93fbdd4d0cd76b48
Reading section 12: 158169-168457 - MATCH: c1a903e5d1160c36625084196c13a923
Reading section 13: 168458-178746 - MATCH: 19b94c5f160fa695baf9163645b25f57
Reading section 14: 178747-189035 - MATCH: 68bc3f6e9a8d783565cbc98537401bdd
Reading section 15: 189036-199324 - MATCH: b450f8028fab40ba8a43c1d13537b40d
Reading section 16: 199325-209613 - MATCH: 84304ae7c4022e25ef80906301539b85
Reading section 17: 209614-219902 - MATCH: e4b6545d043a4913fd36a461a11b8eac
Reading section 18: 219903-230191 - Non-Match.
Reading section 19: 230192-240480 - MATCH: 2ed041596b88e26ffdd11f3f1497afe1
Reading section 20: 240481-250769 - MATCH: 6e4207510b72d64f2fdaf3a6e240b78e
Reading section 21: 250770-261058 - MATCH: 571d2edda501a781e5aac9d0c0a103da
Reading section 22: 261059-271347 - MATCH: e524a36ca2c1a9a1882e246caafb0106
Reading section 23: 271348-281636 - MATCH: 6dd29591019be9b77a4ea482ea3a3a4f
Reading section 24: 281637-291925 - MATCH: d72947d92a61f98ebcd8180c44a36e1b
Reading section 25: 291926-302214 - MATCH: cecc5174d242dcf7316203b2c8889f30
Reading section 26: 302215-312503 - MATCH: f1b3d9da4e6cadc9115d78eae4166e50
Reading section 27: 312504-322792 - MATCH: eb3e1326d4caf53f7a1ad95ddcf29bb1
Reading section 28: 322793-333081 - MATCH: 3baebdb7c7efc7df3771473335d08f58
Reading section 29: 333082-343370 - MATCH: dcb0ce8e933f0a6aac6596edb52d231c
Reading section 30: 343371-353659 - MATCH: 498d10ef9a163a1133a507f782f26924
Reading section 31: 353660-363948 - MATCH: 2e8551952fc2fc87a4d3bf14df734ec1
Reading section 32: 363949-374237 - MATCH: a422a968d40389aa90ee3cb74e07feed
Reading section 33: 374238-384526 - MATCH: 6a30b3a24f5c3c6c053a34ca566c07ce
Reading section 34: 384527-394815 - MATCH: 4e4c496027d120d75748ee4ecb53730b
Reading section 35: 394816-405104 - MATCH: 3144ebd6a0950581c0a6af6b50b40f61
Reading section 36: 405105-415393 - Non-Match.
Reading section 37: 415394-425682 - MATCH: c00e28033b1f658a1630a0ad1f4ee0e3
Reading section 38: 425683-435971 - MATCH: b2849ae119d944b41180606537fc3a9c
Reading section 39: 435972-446260 - MATCH: 952047e1b49b4c09e5d3b0e2c7a1f6a7
Reading section 40: 446261-549150 - read error.
Fake read. Retry - read error.
Fake read. Retry - read error.
Fake read. Retry - read error.

..................:::::::::::::::: PASS 3 ::::::::::::::::..................
Reading section 4: 075857-086145 - read error.
Fake read. Retry - Non-Match.
Reading section 5: 086146-096434 - MATCH: f087063b0a04bc10422f1dbfa135d1b1
Reading section 6: 096435-106723 - MATCH: b871804ebba263d92e6ad6c0bcb2872a
Reading section 9: 127302-137590 - read error.
Fake read. Retry - MATCH: 55c6e9c1d3b69dd78f9c70bdcf750283
Reading section 18: 219903-230191 - MATCH: e652ab44bbfba009861e4ae5b15fdb4b
Reading section 36: 405105-415393 - MATCH: 53dafd12f85ee96ed6c316be28c0518a
Reading section 40: 446261-549150 - read error.
Fake read. Retry - read error.
Fake read. Retry - read error.
Fake read. Retry - read error.

..................:::::::::::::::: PASS 4 ::::::::::::::::..................
Reading section 4: 075857-086145 - Non-Match.
Reading section 40: 446261-549150 - read error.
Fake read. Retry - MATCH: 1fc695f2294cd8ff0092c7190fc9067e

..................:::::::::::::::: PASS 5 ::::::::::::::::..................
Reading section 4: 075857-086145 - MATCH: 2de4b96d33c75bb14206e93dd0030aad
Creating dense.bin: Done. Pass this to ice.exe

Get bass (JPN)

---
 DCdumper.exe DCdumper.exe d -c446261 -df 

Handle acquired.
Load disc: Done.
Sector map created.

..................:::::::::::::::: PASS 1 ::::::::::::::::..................
Reading section 1: 044990-055278 - Initial dump.
Reading section 2: 055279-065567 - Initial dump.
Reading section 3: 065568-075856 - Initial dump.
Reading section 4: 075857-086145 - Initial dump.
Reading section 5: 086146-096434 - Initial dump.
Reading section 6: 096435-106723 - Initial dump.
Reading section 7: 106724-117012 - Initial dump.
Reading section 8: 117013-127301 - Initial dump.
Reading section 9: 127302-137590 - Initial dump.
Reading section 10: 137591-147879 - Initial dump.
Reading section 11: 147880-158168 - Initial dump.
Reading section 12: 158169-168457 - Initial dump.
Reading section 13: 168458-178746 - Initial dump.
Reading section 14: 178747-189035 - Initial dump.
Reading section 15: 189036-199324 - Initial dump.
Reading section 16: 199325-209613 - Initial dump.
Reading section 17: 209614-219902 - Initial dump.
Reading section 18: 219903-230191 - Initial dump.
Reading section 19: 230192-240480 - Initial dump.
Reading section 20: 240481-250769 - Initial dump.
Reading section 21: 250770-261058 - Initial dump.
Reading section 22: 261059-271347 - Initial dump.
Reading section 23: 271348-281636 - Initial dump.
Reading section 24: 281637-291925 - Initial dump.
Reading section 25: 291926-302214 - Initial dump.
Reading section 26: 302215-312503 - Initial dump.
Reading section 27: 312504-322792 - Initial dump.
Reading section 28: 322793-333081 - Initial dump.
Reading section 29: 333082-343370 - Initial dump.
Reading section 30: 343371-353659 - Initial dump.
Reading section 31: 353660-363948 - Initial dump.
Reading section 32: 363949-374237 - Initial dump.
Reading section 33: 374238-384526 - Initial dump.
Reading section 34: 384527-394815 - Initial dump.
Reading section 35: 394816-405104 - Initial dump.
Reading section 36: 405105-415393 - Initial dump.
Reading section 37: 415394-425682 - Initial dump.
Reading section 38: 425683-435971 - Initial dump.
Reading section 39: 435972-446260 - Initial dump.
Reading section 40: 446261-549150 - Initial dump.

..................:::::::::::::::: PASS 2 ::::::::::::::::..................
Reading section 1: 044990-055278 - read error.
Fake read. Retry - read error.
Fake read. Retry - Non-Match.
Reading section 2: 055279-065567 - MATCH: 0278c134f97f19e6f6bc8c390dc7a354
Reading section 3: 065568-075856 - MATCH: 64609ff036ed439e481ab12b3a99ea98
Reading section 4: 075857-086145 - MATCH: 855054d3fadb33728a41d8c944d85188
Reading section 5: 086146-096434 - MATCH: ab80d76c9a189a32664ebfdcdc424664
Reading section 6: 096435-106723 - MATCH: 24f972792a9be6a9f2281c3ea5e62b95
Reading section 7: 106724-117012 - MATCH: ba0310c899c44eb88dd7a6167c4d5dc3
Reading section 8: 117013-127301 - MATCH: 7383f215d15cb11c7740a651002a9bc0
Reading section 9: 127302-137590 - MATCH: 27527a1b9fc83c1a493901220806c7ad
Reading section 10: 137591-147879 - MATCH: 949aa40a59e44f360f52e22bfc2c89b1
Reading section 11: 147880-158168 - MATCH: b4d91a7b835c619f9195dd9191d4054e
Reading section 12: 158169-168457 - MATCH: 4e990abe072fbe37249d5a4fae946e4b
Reading section 13: 168458-178746 - MATCH: 734d5fbd9a058f981c4fd6cfd50ac0ee
Reading section 14: 178747-189035 - MATCH: 019dbd97d55c4d5c13eb40e9ed54250b
Reading section 15: 189036-199324 - MATCH: 596c2268f6dd2195e6c5bd4bf243848d
Reading section 16: 199325-209613 - MATCH: f021c0b016e79511850c52adcb6bae2e
Reading section 17: 209614-219902 - MATCH: 39e829a6a7e71c0844b29bcf8d7cf79f
Reading section 18: 219903-230191 - MATCH: 1db3c0f5ef7b5888e16085bce6fb3d36
Reading section 19: 230192-240480 - MATCH: a7e8b07c58566248c3c8f1eea2fd8348
Reading section 20: 240481-250769 - MATCH: 05cf4016a9c7f13ed12de3c3ac447810
Reading section 21: 250770-261058 - MATCH: f5433c95724932c6d65805ef57b5be9d
Reading section 22: 261059-271347 - MATCH: 87d01b53121a86f480d309ca8a66f2d5
Reading section 23: 271348-281636 - MATCH: 436934278f427d968022568c770f33a6
Reading section 24: 281637-291925 - MATCH: 575bda78d7689624d401ab53633c75df
Reading section 25: 291926-302214 - MATCH: 6e3c2d772d560328bdefaa8c1dfd242b
Reading section 26: 302215-312503 - MATCH: a212c8d8adc030f96bb9f00c969b2f6c
Reading section 27: 312504-322792 - MATCH: 855a758f7167b6700a300956e5b2bad2
Reading section 28: 322793-333081 - MATCH: 7fc281d84fc53a92a3584dc3adbde099
Reading section 29: 333082-343370 - MATCH: 999b6b0ad6e87bde8e8a453628572d46
Reading section 30: 343371-353659 - MATCH: 61687ca1828b8b7b2f658b6d8fb1f0fb
Reading section 31: 353660-363948 - MATCH: 3d176e2001b76d284408b6129925582f
Reading section 32: 363949-374237 - MATCH: c49a831d2b5bb7ed9a1b1a8f465b268d
Reading section 33: 374238-384526 - MATCH: 10803284c464f82675ab0d2bc16a8b1b
Reading section 34: 384527-394815 - Non-Match.
Reading section 35: 394816-405104 - MATCH: 15511e654013f6c5dccf5b36ea787397
Reading section 36: 405105-415393 - Non-Match.
Reading section 37: 415394-425682 - MATCH: bd2b3025b64dff6fbc7c780a6660a7dd
Reading section 38: 425683-435971 - MATCH: 0463a09e601db973b2d289322377a2ed
Reading section 39: 435972-446260 - MATCH: ba8a902ab45e28286d5193509b76b46c
Reading section 40: 446261-549150 - MATCH: 01c9fef76f8cde79b9908468cea30157

..................:::::::::::::::: PASS 3 ::::::::::::::::..................
Reading section 1: 044990-055278 - read error.
Fake read. Retry - read error.
Fake read. Retry - MATCH: f3880579b3c784bf8e10a68068d2862b
Reading section 34: 384527-394815 - MATCH: d25ed6c7f7d7e2a904c5532e774dc639
Reading section 36: 405105-415393 - MATCH: e6a1c8849eaad1a5aefe302500929e22
Creating dense.bin: Done. Pass this to ice.exe

house of the dead 2 (JPN)

---
 DCdumper.exe DCdumper.exe d -c446261 -df 

Handle acquired.
Load disc: Done.
Sector map created.

..................:::::::::::::::: PASS 1 ::::::::::::::::..................
Reading section 1: 044990-055278 - read error.
Fake read. Retry - Initial dump.
Reading section 2: 055279-065567 - Initial dump.
Reading section 3: 065568-075856 - Initial dump.
Reading section 4: 075857-086145 - Initial dump.
Reading section 5: 086146-096434 - Initial dump.
Reading section 6: 096435-106723 - Initial dump.
Reading section 7: 106724-117012 - Initial dump.
Reading section 8: 117013-127301 - Initial dump.
Reading section 9: 127302-137590 - Initial dump.
Reading section 10: 137591-147879 - Initial dump.
Reading section 11: 147880-158168 - Initial dump.
Reading section 12: 158169-168457 - Initial dump.
Reading section 13: 168458-178746 - Initial dump.
Reading section 14: 178747-189035 - Initial dump.
Reading section 15: 189036-199324 - Initial dump.
Reading section 16: 199325-209613 - Initial dump.
Reading section 17: 209614-219902 - Initial dump.
Reading section 18: 219903-230191 - Initial dump.
Reading section 19: 230192-240480 - Initial dump.
Reading section 20: 240481-250769 - Initial dump.
Reading section 21: 250770-261058 - Initial dump.
Reading section 22: 261059-271347 - Initial dump.
Reading section 23: 271348-281636 - Initial dump.
Reading section 24: 281637-291925 - Initial dump.
Reading section 25: 291926-302214 - Initial dump.
Reading section 26: 302215-312503 - Initial dump.
Reading section 27: 312504-322792 - Initial dump.
Reading section 28: 322793-333081 - Initial dump.
Reading section 29: 333082-343370 - Initial dump.
Reading section 30: 343371-353659 - Initial dump.
Reading section 31: 353660-363948 - Initial dump.
Reading section 32: 363949-374237 - Initial dump.
Reading section 33: 374238-384526 - Initial dump.
Reading section 34: 384527-394815 - Initial dump.
Reading section 35: 394816-405104 - Initial dump.
Reading section 36: 405105-415393 - Initial dump.
Reading section 37: 415394-425682 - Initial dump.
Reading section 38: 425683-435971 - Initial dump.
Reading section 39: 435972-446260 - Initial dump.
Reading section 40: 446261-549150 - Initial dump.

..................:::::::::::::::: PASS 2 ::::::::::::::::..................
Reading section 1: 044990-055278 - read error.
Fake read. Retry - MATCH: 757fdf03de428e95183f49749b33bbe7
Reading section 2: 055279-065567 - MATCH: 2cc8c975e637301b0cb370b4d5faa6da
Reading section 3: 065568-075856 - MATCH: a825a14b50a9e68c3da70b6489d8cd4b
Reading section 4: 075857-086145 - MATCH: 85533519128522066628f0cf0d802051
Reading section 5: 086146-096434 - MATCH: 6f421197040158378914cbb7175fe850
Reading section 6: 096435-106723 - MATCH: 9b4bdd1dc02f9106be1d0942d93c2a1d
Reading section 7: 106724-117012 - MATCH: 0debbeb08c02ce881a1500da13ee19c5
Reading section 8: 117013-127301 - MATCH: 8fb1e76eb733b5b8a69fe6f36623ccaf
Reading section 9: 127302-137590 - MATCH: 9865074bc9e9827fa576484bcc3e8b85
Reading section 10: 137591-147879 - MATCH: 0da6ec27c0c551fe8b9f9c526f27279b
Reading section 11: 147880-158168 - MATCH: 4f4c8c2d62bfac7d13d48305f5a0eb49
Reading section 12: 158169-168457 - MATCH: baa873dc53126af030706f3dc7ebac4f
Reading section 13: 168458-178746 - Non-Match.
Reading section 14: 178747-189035 - Non-Match.
Reading section 15: 189036-199324 - Non-Match.
Reading section 16: 199325-209613 - MATCH: d4d1157f282e61b0e4de7bdd556e1056
Reading section 17: 209614-219902 - MATCH: 4cc95037efdb23bb339ff029817619a2
Reading section 18: 219903-230191 - MATCH: 8cc863a1ab5e1a5647ca679cf7d5b00c
Reading section 19: 230192-240480 - Non-Match.
Reading section 20: 240481-250769 - Non-Match.
Reading section 21: 250770-261058 - Non-Match.
Reading section 22: 261059-271347 - MATCH: 3ab8e5e2367f3cec692995de8149680f
Reading section 23: 271348-281636 - MATCH: e16d2ea28af29c4fe4285fcbfc095909
Reading section 24: 281637-291925 - Non-Match.
Reading section 25: 291926-302214 - Non-Match.
Reading section 26: 302215-312503 - MATCH: a5216a09d546738dbc71cef4feec2975
Reading section 27: 312504-322792 - MATCH: 339e5f2f20afbaf862dd57204e9c4188
Reading section 28: 322793-333081 - MATCH: f3d5c4f0345e0ef906b2b250a5da703a
Reading section 29: 333082-343370 - MATCH: f977679c3d0123c9db7244e30e526011
Reading section 30: 343371-353659 - MATCH: 899b54ee7f76efda5a834bce922f1f2c
Reading section 31: 353660-363948 - MATCH: 549d3589a0d532369731c9e0146b2693
Reading section 32: 363949-374237 - MATCH: cba59f563c8fd483c80e5a78d75e6e85
Reading section 33: 374238-384526 - MATCH: ee0da6d4c8f23be696188a07b1777168
Reading section 34: 384527-394815 - MATCH: cf2a5c5173720e6e8e3c4a96ec7f9c79
Reading section 35: 394816-405104 - MATCH: 55ae7dd516fcd41267ae34f1e71771f6
Reading section 36: 405105-415393 - MATCH: aaca0ffe36bfe484895476dec3f67e01
Reading section 37: 415394-425682 - MATCH: 10a65d7134a7fcd70f5d4d83546fd732
Reading section 38: 425683-435971 - MATCH: f430c67ad81cffa491a2a1c8774091d2
Reading section 39: 435972-446260 - MATCH: 1159b4e2115822c1093683c250110e4d
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 3 ::::::::::::::::..................
Reading section 13: 168458-178746 - MATCH: e536350ff663568b46f988e58d5f04ca
Reading section 14: 178747-189035 - MATCH: 0e63c4f185065338976415db5afa1e02
Reading section 15: 189036-199324 - MATCH: a1fb6c80af27da67a8f39ffa0dab8cd6
Reading section 19: 230192-240480 - MATCH: 448bab402d33cad431f676e94c2d475a
Reading section 20: 240481-250769 - MATCH: eaeea09ce87800c4372fb4d797512d34
Reading section 21: 250770-261058 - Non-Match.
Reading section 24: 281637-291925 - Non-Match.
Reading section 25: 291926-302214 - MATCH: 36d0072bd7034284a4e56a54d4b64909
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 4 ::::::::::::::::..................
Reading section 21: 250770-261058 - MATCH: e451a2b1d102b132308b73810da7f54b
Reading section 24: 281637-291925 - MATCH: dac75dd50afeb4a1afd36ffcfe8ad220
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 5 ::::::::::::::::..................
Reading section 40: 446261-549150 - read error.
Fake read. Retry - Non-Match.

..................:::::::::::::::: PASS 6 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 7 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 8 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 9 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 10 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.
Sector read error between sectors 446261 and 549150 inclusive.
Clean the disc and try again.
---

---
 DCdumper.exe DCdumper.exe d -c446261 -df 

Handle acquired.
Load disc: Done.
Sector map created.

..................:::::::::::::::: PASS 1 ::::::::::::::::..................
Reading section 40: 446261-549150 - Non-Match.

..................:::::::::::::::: PASS 2 ::::::::::::::::..................
Reading section 40: 446261-549150 - MATCH: bc476676b73e635d1a952d3109718d4d
Creating dense.bin: Done. Pass this to ice.exe

New version (0.42a)

20110919 (0.42a)
+FIX:Read errors apparent since 0.3a
+FIX:Writing hashes to file

Have tested and found dumps to match in 0.42a which didn't in 0.41a. There may still be errors in dumping, lets hope everyone had the same problem as me  big_smile

The next version should just be for new functions, or more read fixes if necessary. Next version doesn't create/open the log file unless something will be written. Other possibilities I'm considering:

  • Send stop start  commands to drive when there are many adjacent read errors (exiting program, doing this then running program seems to kickstart it sometimes, could just be waiting awhile or some other factor though)

  • Specify 1x read speed (currently not, although sptd does seem to go a lot slower naturally)

  • Suggestions?

Fair enough about ice numbers in brackets not being definite errors and just edc/ecc checks. Knowing that, I see why the ice log that made me think of validating by ice is wrong (because the input was wrong). Each section was (0.3a-0.41a) filled with the first 26 sectors of the section, so the audio tracks after a data track had thousands of ice errors as they were actually repeated mode1 sectors from the data track. My testing of 0.3a-0.41a must only have gone so far as checking the ice log, of what happened to be single track games, oops. A complete garbage dump but it would have compressed well  lol

Thank you all for testing, information and suggestions. Keep it coming.

Redid a dump that was failing ice on 0.41a, this is the result:

ice @20090609 / themabus@inbox.lv
---------------------------------
dense.bin
---------------------------------
Accessing                       : ok
Seeking 1st valid Mode1 sector  : ok
 LBA found                      : 44990
 @file offset                   : $0000004C
 Scrambled                      : TRUE
Seeking LBA 45000               : ok
 Combined offset (samples)      : 5899
Parsing IP.BIN                  : ok
 Writting 'ip.txt'              : ok
Parsing TOC                     : ok
 TOC entries                    : 20
 Writting 'redump.cue'          : ok
---------------------------------
03  Mode1   45000  375571  330572 ok
04  Audio  375572  383649    8078 ok
05  Audio  383650  397093   13444 ok
06  Audio  397094  399944    2851 ok
07  Audio  399945  406033    6089 ok
08  Audio  406034  413385    7352 ok
09  Audio  413386  415457    2072 ok
10  Audio  415458  419294    3837 ok
11  Audio  419295  428565    9271 ok
12  Audio  428566  453103   24538 ok
13  Audio  453104  478127   25024 ok
14  Audio  478128  485726    7599 ok
15  Audio  485727  513636   27910 ok
16  Audio  513637  523073    9437 ok
17  Audio  523074  525264    2191 ok
18  Audio  525265  525788     524 ok
19  Mode1  525789  549149   23361 ok (75)
---------------------------------
done

It matches db so great, but will test more and add some functions before new binary. The question is, does every multi-track dump have 75 errors on the last track? Does this number vary at all? Is there any other time ice indicates track errors where the track is actually valid?

Found a problem with the reading, will test before posting a new binary. If the problem is how I think it is, why it doesn't mess up everything is a mystery. For any who still want to test 0.41a, use -i26 as it should bypass the problem I can see.

DeviceIOControl is called to read each section 26 sectors at a time, and you seem correct that it messes up sometime after the first DeviceIOControl call.

Ultra-speed chunk generation sounds very bad, but plausible if windows keeps a record of the last successful read attempt.

Now no log is created unless the arguments are parsed correctly and it will try to dump something.

Come on man. Vague descriptions won't help at all. From that all I can give is vague advice. Problems with starting may well be the same problems that occur with the guide (sometimes you have to try the trap disc multiple times or stop start before it'll take).

What do you mean by hung half way through? An error box popped up? It sat there looking pretty? It started getting read errors all the time?

Please follow this:
Use -df to not delete the .hsh or temp section files.
If any complete sections are saved to hdd, create an md5 file for comparison with the .hsh files
Find the dcd.log so I can see your console output
Send me everything. md5, .hsh, dcd.log, your drive, the game title/region/redump entry
If it hangs, where, what was the last thing it said it was doing
Write down what any error messages say
Less is not more. Send me whatever you have even if you think it's useless. If I have nothing, I can do nothing.

Problems I have found with my own testing:
Some discs have a lot of read errors right from the start of execution, with the odd dump but largely read errors. Exiting and starting the program a few times has got some working. Once they got started they executed ok.

Some discs dump fine and dense.bin is created, but ice.exe shows that the dump has errors. I had hoped that by verifying the dump that most of those errors would go (on the premise that bad dumps due to scratches would rarely match). It seems successful matching reads don't always mean correct reads. Suggestions as to how to reduce this issue? Of course it's possible to detect when ice gives errors and try to rerip, but anything I can do at the ripping stage?

Please run the program to completion and try to get a match with the db. And explain how the matches have BIG differences.

Also post the dump you're trying, the log file etc.

New version (0.4a)
Added some read options/functionality, and cleaned code a little.

-ft -ff - Fake reads: Fake read nearby before re-read to try and kick program out of read errors
          Default true

-st -sf - Step back: Fake reads after a pass to step to beginning
          Default: true

-t#     - Reread attempts: Max number of re-read attempts on a read error
          Min: 0  Max: 20  Default: 3

Hopefully this version fixes a lot of the previous issues with read errors.

edit: 0.41a suppresses some strings that shouldn't be shown on fake reads.

New version (0.3a).

Usage: DCdumper.exe drive_letter [options]

20110917 (0.3a)
+No longer uses akrip32, now DeviceIO (SPTD)
+No longer uses startstop.exe, now DeviceIO (Load device)
+Section hashes stored to file as successful reads are completed (better resume function)
+Log file created (mirrors console)
+Pass drive letter, not scsi id

Thanks themabus, much cleaner now.

@themabus
Thank you, I'll look into it. My choice of akrip was just because it was the first thing I found that worked, but it seems people are having problems.

@tossEAC
I will try your suggestions if necessary, fake reads particularly sound potentially useful. No I will not be your xml code monkey bitch tongue
I have a black drive but I think it is C version.