Repository navigation
2x faster compression #108
Description
I've done some tests to determine whether Conor's fastlzma2 library could actually benefit precomp. The answer is yes and by a long shot. The tests have proven that the novel algorithm provides about 2x faster compression, all while compressing up to 10% more and using less memory than multi-threaded lzma. So it's basically double speed for free.
Because of this...
The library can compress using many threads without dividing the input into large chunks which require the duplication of the match-finder tables and chains. Extra memory used per thread is typically no more than a few megabytes.
... the difference on ratio is more visible when working on a high number of threads, which on mt-lzma usually means chunking the input and losing long range compression, but with flzma2 doesn't.
Activity
M-Gonzalo commentedon Apr 18, 2020
Precomp mt-lzma compression VS xz-fastlzma2 comparison and report:
Sumary:
Details:
===========================================================
zpaqgui-win32.win32.x86_64.zip (pre-compresed):
On ARM Cortex A53, 8 threads:
On x86-64 Intel Atom, 4 threads:
On x86-64 Intel Xeon, 2 threads:
===========================================================
APKs.zip (pre-compresed)
On ARM Cortex A53, 8 threads:
On x86-64 Intel Atom, 4 threads:
On x86-64 Intel Xeon, 2 threads:
===========================================================
These are only two datasets that are representative of several other more informal tests during the past months.
BTW: fxz uses ~900-1000m RAM on the default configuration, instead of the ~1300 that precomp does, although it can be configured to use a bigger dictionary and, in doing so, provide even better compression.