1 Topic by jamjam (edited by jamjam 2013-03-14 19:40:07)

  • Registered: 2009-07-27
  • Posts: 352

Process images from certain systems for efficient archival (at present Gamecube, PS3, Wii, Xbox).

Features:

  • Can merge multiple releases from the same system into one set of files

  • For systems with padding (Gamecube, Wii, Xbox), the padding is split into a separate file. GParse can recreate the original image (with the padding file) or scrubbed image (without the padding file) on decode

  • For systems with encryption (PS3, Wii), it is decrypted

  • All of these factors can increase compression efficiency

Notes:

  • The game data (dec_file) should be compressed after processing, the padding (pad_file) shouldn't as it is (or at least should be) incompressible

  • Supported systems: Gamecube, Wii, Xbox, PS3

  • This is not a compressor. It can be a pre-processor to a compressor

help wrote:

GParse r27

Usage:

Encode (all except ps3):
GParse e <system> pad_file dec_file grp_file game_1 game_2 game_3 ...

Encode (ps3):
GParse e <system> pad_file dec_file grp_file game_1 key_1 game_2 key_2 ...

Decode:
GParse d pad_file dec_file chk_file game_1 game_2 game_3 ...

<system> : wii for wii, ngc for gamecube, xbox for xbox, ps3 for ps3 (all lowercase)
pad_file : The file the padding will be/is located in
dec_file : The file the decrypted game data will be/is located in
grp_file : Stores names, sizes, hashes
chk_file : Allows external source to check status of execution
game_#   : The next game to process
key_#    : The key used to decrypt/encrypt game_# (disc_key, not d1)

Note: Even systems without padding (ps3) have to include the pad_file in the
      commandline. It keeps things consistent and simple. Specify it as NULL

'-' in place of the file path indicates read from stdin (or write to stdout)
'NULL' in place of the file path indicates ignore file

On encode, pad_file can be stdout, NULL or a file path
On encode, dec_file must be a file path
On encode, grp_file can be NULL or a file path
On decode, pad_file can be NULL or a file path
On decode, dec_file can be stdin or a file path
On decode, chk_file can be NULL or a file path

Changes from r25 to r26

  • Fixed Xbox processing, which treated some data as padding (which means all existing Xbox processing outputs garbage when decoding as scrubbed, so existing stuff needs to be redone)

  • Breaking change to Xbox processing (which I figured didn't matter as all Xbox processing to date has been wrong anyway) to allow current (3629408 sector) and potentially new (3820880 sector) redump images to be processed together (and leave the door open for future non-breaking changes for other sizes)

Changes from r26 to r27

  • Added ps3 as a supported system (input are original aka redump aka encrypted images, which need to be decrypted)

  • OpenMP is a new requirement to allow multi-threading (for windows this means vcomp100.dll should be supplied with the program)

  • PS3 is currently the only system which uses multi-threading

GParse-r27

PS3Dec (decrypt ps3 images), PS3DumpCheck (check integrity), GetKey (dump PS3 metadata), DatSplit (split redump dats), GPack (compress related images together)