Merged Compression Tests (Page 1 of 2)

yes, it's symetrical algorithm and decompression takes pretty much this same time as compression (and memory also..)

no way PAQs are disqualifed because i dont have all day to wait for each to finish

I agree, for distribution there will need to found better method which will at least consume less memory... I have in mind splitting all discs into f.e. 100mb parts and then merging with proper sorting.. with 100mb it would be enough to have 150-200mb of RAM for repetition filter - I'll test this method soon

cmon man, I've been recently doing massive recompression on about 1700 discs and it took barely 2 weeks with ultra 7-zip on 2x2.5ghz and compressing with this same method inside FreeArc would be about 20% faster

topkat, idea is effective but it's deffinately too much job with editing those files, everything must be done manually, and you're loosing original file naming and cues, personally I would not want such mess...

good method should be done by one command, don't change anything in original files and decompress files to their original state, and I have some new ideas about that...

Updates:
- low-memory methods using split have been added to test (config "Arc 6" needs only ~250mb for decompression)
- approximate compression time added to the results

However... there is still some manual work with splitting... would be nice to have it done by automatically before compression and auto-joined after decompression

dunno, to me ImageDiff results look most appealing
it would depend on decompression speed difference, which one of two i'd consider the best

splitting in parts adds another layer the same as ImageDiff and both speed and compression is worse
otherwise it's just too slow, imho.

when i said years i mean single game from such merged 7z+ecm+flac (or tak)+ImageDiff set
should decompress in about 2-3 minutes or so
it's most frequent scenario, nobody really needs whole set decompressed at once,
and still ImageDiff would be faster than 10 minutes, i guess.
so, very roughly:
let it be 20 minutes overhead on game
let it be 100 such games (or decompressions user commits)
and they're shared by 1000 people
100/3 = 33 hours = 1.4 days
1.4 * 1000 = years
sure they would download faster, but download is done passively in background
when in contrast decompression takes full PC load and needs interaction from user

...moreover.

If someone makes a little tool that set date like packiso and we choose a winrar version that people must use to archive (you don't have to share it, and telling people "use v4.00" is not illegal) probably every archive will have only a few kbytes or less to be fixed.

Doesn't 7zip have a greater compression over RAR and wasn't there a purpose for keeping the tracks in separate files even when torrenting? What this is suggesting is going to kill everything that has been worked on with packIso, especially with keeping the tracks separate so people can get only the tracks they need without having to download the whole image or even seed the tracks they currently have while downloading the ones they may not. Not to mention all the current torrents that have been made using packIso (which is being accepted by the torrenting community from what I can see).

I'll admit that the packIso format may not be the best compression but it is at least much better then the current alternative torrentzip and still offers the ability to keep tracks separate and keep the files the same for sharing purposes. The time it takes to unpack a packIso archive is also a big help unlike what is being suggested here where if I wanted to say play a game or make a patch I would have to wait at least 30 minutes just to decompress it which wouldn't be very helpful at all for me even if I were to only access them very rarely.

As far as setting the date, that is what the rmdtrash.exe file does in packIso. It sets the dates on the files themselves before they are compressed so that all of the 7zip archives have the same date for each files on everyone's computer. It also clears the archive bit set by the OS in the file system to further clear out unwanted data for compression.

Everyone seems to be so worked up over using clrmamepro that you have forgotten the purpose of the archives to begin with and although I can understand why you would want to use it the way you are going about this seems to be a bit far fetched as far as actual usability for everyone. I've already mentioned that adding packIso checksums is all we would have to do in order to get a dat for packIsoed files while still maintaining  our original compression. As long as there is someone that has a copy of the game that matches the database then they can make a packIso archive and post the checksums of the files and in the case of the data track the checksum of the ecm'd file in the 7zip archive should be taken instead of the archive itself so that file name changes won't effect anything.

The compression percentage is not something that we need to worry about attaining nor should we worry about whether or not the archives are compatible with any emulator or front-end application. The purpose of the archives is just to store them in a way that makes it easy for everyone to get and decompress for use however they see fit after they get the archive and right now packIso is the only thing I can see that really offers that at this time.

creating Diffs and restoring original game from those takes a lot of time and user effort, even more than compressing/decompressing
as I proved you can merge images ultra fast with FreeArc LZMA (merging 5gb into 380mb in about 10 min - is that long ?? and decompressing everything to its original state in 5~8 min)

right now in present moment of time merging with low-memory decompression needs manual splitting, however ... I wrote to the author of FreeArc, he is open to suggestions and theres a pretty good chance that he will include splitting feature into one of next FA releases
+ add to that ECM filtering automatically done inside FreeArc
and imagine... in near future it will be possible to merge with even less than 256mb of RAM and with just a single command-line or few clicks inside GUI

if you would have audio tracks stored as .wav you could configure FA to compress those for you with Ape format and .bin with LZMA
...and if you would not want to convert to .wav you could create archive with 2-steps:
1. create + add all data tracks with LZMA method
2. add to previously created archive all audio with Ape codec
isn't that convenient ?

and dont forget, merging right now is only experimental
I'm only trying to show some possibilities, maybe someone will try those for saving HDD space only only remember to test data after compression if you deal with unstable alpha versions of soft

7-Zip does much better with data than RAR
RAR does much better with audio than 7-Zip
APE does even better with audio than RAR, because it is dedicated audio algorithm
APE is very fast and works with low-memory
thats basically it...

and when it comes to audio... imho Ape is a good format, it is fast, well known and supported by many other software, yes you can get better ratio with TAK... but so what ? you can get even better (+ faster) with LA

oh, i see - if you'd extract single file from archive
for ImageDiff you'd need to run it through ECM and ImageDiff
but for FreeArc only ECM (and then join files)
but still decompression should be notably longer for FreeArc since it's 5.5gb vs 700mb
so what's lost on ImageDiff should be regained on decompression

ok, so it will be more convenient
but it's possible to make an frontend or a script for commandline applications - many do that now
there are quite a lot of programs (for cd recording, video/audio encoding/processing and so on)
that actually are only GUIs for command line *nix applications

but for your example it's 2nd case, right?
there wouldn't be 2gb archive, it would be 350mb.
so ImageDiff would have 5 minutes to complete in, and actually it shouldn't be much slower than joining files.
(it basically is joining of files)

edit:
so anyway such merged set (7z+ecm+ImageDiff+tak) seems very good idea to me
from example above it's 8 times compression improvement over PackIso
it would be rare of course to have so many versions merged
but generally it would still save a lot of time and space
(even single title decompression from this set on average shouldn't be slower than PackIso, i think,
because of speed gain from TAK and would TAK implement support for RAW audio data
sox.exe could be eliminated, as it's used now to add/remove RIFF header,
which means every audio track is basically copied after extraction,
which is about the same as doing ImageDiff)
kudos to cHrI8l3 for that

is this FreeArc?

ok, so i guess, it's how fast single version from 378mb FreeArc would extract
could you test extraction of .ecm + one .imageDiff from 357mb .7z and 344mb .nz, please then
unecm would be the same, and ImagePatch would be slightly slower than joining, i guess

edit:
noooo, wait
ummm it says 7z

that's what i thought:
1st: FreeArc - prettyslow
2nd: 7z + ImageDiff - fast

i mean when you have files decoded from ImageDiff .7z you'd still beat FA by about 4-5 minutes
plus compression is better