26 Reply by cHrI8l3

(43 replies, posted in General discussion)

First of all it looks like this program is still in development and the current version is an alpha build so I don't think it would be a good idea to migrate everything to this just yet until the program is more perfected by the author.

yes its still unstable release, and every time you want to store an archive, you should execute test after compression (-t switch)

Basically if we are going to use this to help spread our dumps we need to make sure it's going to be usable and accepted by everyone first (or at least the majority of people that know what they are doing) before we start migrating everything to a new format.

yes yes i agree ! dont use freearc yet for anything official, just try it for personal use for now, make used to it, learn basic usage etc...
It will be however possible to create bundles of FreeArc with other compressors (like ECM) and config, lets say f.e. "FreeArc 0.5 (Redump Pirate Release)" and every thing user will need to do is to unpack it, run FreeArc.exe, select and extract archives made with ECM tongue no need for installation, no need for messing with config, thats one example...

so in 'Arc 6' ther's ECMs of all 8 images -> diff
decompression and reverse of diff on all 8 images would complete in 7-8 minutes, leaving ECMs, right?

yes there are 8 .ecm files (no diffs!!) understand one thing... im looking for a method that will not involve diffs, I added diff method in tests only for comparison of archive sizes that you can get with and without diff, and as you can see both moethods giving pretty much this same size (with few MB in favour of diff and greater speed/convenience in favour of repetition filter...)

can you make a filter chain then, that would produce one ECM and diffs, alike to 'ImageDiffs+ECM+7z' ?

I dont thinks so, FreeArc config does not treat every file inside archive separately, but as a merged solid bundle, for patches you would need more compliacated algorithm that will select pairs of files of create patches from those pairs...

I do not think this is meant to be used for spreading the images. Instead it is meant as to store many games in very little space.

hell yeah smile for everyone who is short on hdd space and dont want too much mess with images

Go to forum: General discussion

27 Reply by cHrI8l3

(43 replies, posted in General discussion)

what about records with FreeArc+NanoZip

lets make one thing clear... FreeArc is a compression suite (not compressor!!) it allows you to link different algorithms
it has built-in some nice algorithms like LZMA (the one 7z uses), repetition filter, exe filters, delta...
and you can configure it to work with almost any other command line compressors/data filters, f.e. ECM, NanoZip, Precomp, APE, WinRAR, etc... whatever you need
and then.. you can create your own packing profiles by linking compressors with filters etc.

you can run for example arc.exe a -mecm+rep:1gb+lzma:128 -dp"C:\Working Dir" -- "C:\Working Dir\arc.arc" "file1" "file2"
and it will first ecm both files, then filter repetitions within 1gb range, and then compress it to LZMA with 128mb dictionary
its simple smile

if you can talk author into integrating ecm then - it's even better

it is already possible, but ecm have some issues on large sets of data... it can be used on <2gb files thou
might be worked out in one of next releases

(7 is about 10 - 7 approximates to 10) = symmetrical

wtf smile repetition filter takes most of the time, decompressings from LZMA takes less than half of that time

Try it, deffinately !!
download and run installer of v0.50: http://freearc.org/Download.aspx
download and unpack over installed version update pack: http://www.haskell.org/bz/arc1.arc (recommended)

Edit:
if you want you can store your own configurations in arc.ini, here is one of mine:
cso7=ecm+rep:1gb+lzma:128mb:max:bt4:273
and you run it with -mcso7 switch

and... following can also be configured and run when you have audio stored as wav:
packiso= $obj => ecm+7z, $wav => ape

and. FA has also a GUI but you can not run custom configurations from GUI yet.. however it will be soon resolved

in short... its soft for PRO's and thats why I though redump staff might be interested tongue are you ? smile

Go to forum: General discussion

28 Reply by cHrI8l3

(43 replies, posted in General discussion)

as you said yourself FreeArc use symmetric compression algorithm which 7z does not

I never said that! quite opposite when you look at the config table you will see that fast, low-memory method from FreeArc use this same algorithm as 7z - LZMA !! and FA has faster implementation of LZMA than 7z
LZMA is assymetric
symmetric compressors are used in NanoZip

ImageDiff uses 2 methods for creating patches, Ive been using patches created with "best diff type" option and "slow match" for smallest patches

unpack 650mb from 350mb 7z ~80 seconds

unpack 1 versions of FF IX (in .ecm format) from 1 archive (7z) in 80 seconds (no ImageDiffs here!!)

but it's overkill imho - 5gb of RAM is not sane and probably same could be done then with ramdisk

i mean it's all good to have all those features in one program but decompression algorithm itself is way slower
so that's where it ends, imho

man you should educate yourself or at least play some with those algorithms before make such statements

I know my english is not perfect, but I already wrote all informations which are needed to understand the whole idea
decompression tests might by confusing and the whole diffs, because I did not intended to make those in the first place
I will think about it and maybe add those unpacking times to table with results , perhaps that will be easier to understand ;/

Go to forum: General discussion

29 Reply by cHrI8l3

(43 replies, posted in General discussion)

themabus, you have no idea about how those methods really work do you
no worries, lets start with imagediff, which if you used at least once you would know that its slower than simple decompression

unecm "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin.ecm" ~75 sec
imadiff "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin" into version "(F)(Disc 1 of 4)[SLES-02966]" ~5 min

imagediff is patching in 3 basic steps
1. calc md5 (hdd speed) to validate integrity of foreign image
2. create sector map of foreign image (also hdd speed)
3. rebuild image while comparing imagediff's sector map to sector map of foreign image (and this one is "slightly" 2-3x slower than just reading whole image)

it reads whole original image at least 3 times!

and now compare it to extracting ALL 8 files to their original form (without any patching) in 8 (eight) minutes from FreeArc archive (+ unecm on 8 files which is even slower than decompression = 8 *75 = ~10 minutes )

FreeArc is ultra fast because of its method:
- first decompress 380mb archive into repetition filters dictionary + non-common data (in this step you dont have 5gb un-merged set, only ~700mb dictionary + ~50mb of data)
- next repetition filter is working in a way which you though imagediff should work - rebuild files with common parts which are stored in dictionary and are loaded into RAM for fast access (and here is done the magic of un-merging into 5gb)
- rebuilded data is stored in cache and then written onto hdd

and as I already wrote from the beginning "no need for storing by ImageDiff when merging with repetition filter is much more conveniant" big_smile

Go to forum: General discussion

30 Reply by cHrI8l3

(43 replies, posted in General discussion)

but for your example it's 2nd case, right?
there wouldn't be 2gb archive, it would be 350mb.
so ImageDiff would have 5 minutes to complete in, and actually it shouldn't be much slower than joining files.

I dont know if understanded your point correctly... example on which I based those decompression times is based on compression WITHOUT using ImageDiff (config "Arc 6" from base topic)

this is what that 380mb archive contains inside:
http://img12.imageshack.us/img12/269/clipboard05h.th.jpg

and those ~2gb 7z is total of 8 packiso archives ( 8*350mb )

ps. in a few days ill do merging test on game with audio tracks smile that may be interesting...

Go to forum: General discussion

31 Reply by cHrI8l3

(43 replies, posted in General discussion)

LA is very-very slow

as far as Ive been testing its slightly faster than ape (using maximum possible settings) hmm need to check that again...

but for FreeArc only ECM (and then join files)

Ive been trying to explain that FreeArc will be soon able to UnECM (+ Join) data automatically after decompression - so the only thing user will need to do is click on "Decompress" to restore original .bin's, without further processing files...

but still decompression should be notably longer for FreeArc since it's 5.5gb vs 700mb

unpack 5.2gb (8 files) from 380mb FA archive ~7 minutes
unpack 650mb from 350mb 7z ~80 seconds
unpack 8 x 650mb from 2gb 7z ... 8*80 = big_smile
...times checked on external usb drive, with internal SATA should be better... wink

Go to forum: General discussion

32 Reply by cHrI8l3

(43 replies, posted in General discussion)

dunno, to me ImageDiff results look most appealing
it would depend on decompression speed difference, which one of two i'd consider the best

creating Diffs and restoring original game from those takes a lot of time and user effort, even more than compressing/decompressing
as I proved you can merge images ultra fast with FreeArc LZMA (merging 5gb into 380mb in about 10 min - is that long ?? and decompressing everything to its original state in 5~8 min)

right now in present moment of time merging with low-memory decompression needs manual splitting, however ... I wrote to the author of FreeArc, he is open to suggestions and theres a pretty good chance that he will include splitting feature into one of next FA releases
+ add to that ECM filtering automatically done inside FreeArc
and imagine... in near future it will be possible to merge with even less than 256mb of RAM and with just a single command-line or few clicks inside GUI

if you would have audio tracks stored as .wav you could configure FA to compress those for you with Ape format and .bin with LZMA
...and if you would not want to convert to .wav you could create archive with 2-steps:
1. create + add all data tracks with LZMA method
2. add to previously created archive all audio with Ape codec
isn't that convenient ?

and dont forget, merging right now is only experimental
I'm only trying to show some possibilities, maybe someone will try those for saving HDD space smile only only remember to test data after compression if you deal with unstable alpha versions of soft tongue

Doesn't 7zip have a greater compression over RAR and wasn't there a purpose for keeping the tracks in separate files even when torrenting?

7-Zip does much better with data than RAR
RAR does much better with audio than 7-Zip
APE does even better with audio than RAR, because it is dedicated audio algorithm
APE is very fast and works with low-memory
thats basically it...

and when it comes to audio... imho Ape is a good format, it is fast, well known and supported by many other software, yes you can get better ratio with TAK... but so what ? you can get even better (+ faster) with LA

Go to forum: General discussion

33 Reply by cHrI8l3

(43 replies, posted in General discussion)

topkat, idea is effective but it's deffinately too much job with editing those files, everything must be done manually, and you're loosing original file naming and cues, personally I would not want such mess... smile

good method should be done by one command, don't change anything in original files and decompress files to their original state, and I have some new ideas about that...

Updates:
- low-memory methods using split have been added to test (config "Arc 6" needs only ~250mb for decompression)
- approximate compression time added to the results

However... there is still some manual work with splitting... would be nice to have it done by automatically before compression and auto-joined after decompression

Go to forum: General discussion

34 Reply by cHrI8l3

(43 replies, posted in General discussion)

so it's 30-40 minutes to decompress those?

it's still too much imho
maybe for archiving, when kept to oneself - accessed infrequently and space is really an issue
but otherwise (if such archives would be distributed)

I agree, for distribution there will need to found better method which will at least consume less memory... I have in mind splitting all discs into f.e. 100mb parts and then merging with proper sorting.. with 100mb it would be enough to have 150-200mb of RAM for repetition filter - I'll test this method soon

those minutes multiplied with thousands of CDs and thousands of copies would turn into years

cmon man, I've been recently doing massive recompression on about 1700 discs and it took barely 2 weeks with ultra 7-zip on 2x2.5ghz big_smile and compressing with this same method inside FreeArc would be about 20% faster

Go to forum: General discussion

35 Reply by cHrI8l3

(43 replies, posted in General discussion)

You did not try PAQ?

no way big_smile PAQs are disqualifed because i dont have all day to wait for each to finish big_smile

Go to forum: General discussion

36 Reply by cHrI8l3

(43 replies, posted in General discussion)

it (NanoZip) would decompress slower than 7z, though
right?

yes, it's symetrical algorithm and decompression takes pretty much this same time as compression (and memory also..)

Go to forum: General discussion

37 Topic by cHrI8l3

(43 replies, posted in General discussion)

Job: merge different versions of this same game
Goal: find best method that will require smallest user effort for compression and decompression (and perhaps don't use too much of system memory)

Configurations used:
http://img8.imageshack.us/img8/1701/clipboard04n.jpg

Test #1:
Game: [PSX] Final Fantasy IX (Disc 1) (8 versions)
Versions order used for merging: U v1.0, U v1.1, E, F, G, I, S, J
Results table:
http://img17.imageshack.us/img17/8871/clipboard03s.jpg
Brief summary:
- 5,5gb - uncompressed
- 2,8gb - PackIso
- 379mb - ECM+Split:100mb+Rep:200mb+LZMA:32mb in ~10 minutes (~250mb for decompression!)
- 375mb - ECM+Rep:1gb+LZMA:128mb in ~25 minutes (~1gb for decompression)
- 354mb - ECM+Rep:1gb+NanoZip:1.5gb in ~40 minutes (~1.5gb for decompression)
- 344mb - ECM+Split:100mb+Rep:1gb+NanoZip:1gb in ~30 minutes (~1gb for decompression)

One version of game & ImageDiff for others:
- 358mb - 7-Zip:192mb
- 344mb - NanoZip:1.5gb

Some notes:
- times were measured on dual core 2.5ghz, and accuracy is about 90%
- it will be possible to use ECM inside FreeArc, but we need to wait for next more stable version - that will reduce whole process into one command

Basic idea of splitting:
- split data into parts, f.e. v1.bin.001, v1.bin.002, v2.bin.001, v2.bin.002
- add all to archive sorted by extension and name: v1.bin.001, v2.bin.001, v1.bin.002, v2.bin.002
- apply repetition filter with at least twice large dictionary than part size (so if you have parts for 100mb you need at least 200mb dictionary for it to work good)

Conclusions:
- no need for storing by ImageDiff when merging with repetition filter is much more conveniant
- with splitting you can achive amazing results with cost of convenience...
- buy more RAM - there's never too much when it comes to compression

Go to forum: General discussion

38 Reply by cHrI8l3

(13 replies, posted in News)

Welcome Macarro smile Thx for interest in this project

Go to forum: News