Available Languages:

SourceForge.net Logo

SuperREP: huge-dictionary LZ77 preprocessor

Description

SuperREP is the first LZ77 compressor that supports dictionaries larger than RAM available. Default settings (-l512) allows to process files that are 20x larger than RAM size. Memory requirements are proportional to 1/L, so by increasing -l value it's possible to process even larger files.

Memory usage

When compressing, memory usage for hash: filesize/L*20 + filesize/L*4 + roundup(filesize/L*4 * 5/4) + roundup(filesize/(L*8) * 4) where roundup() rounds up to the next power of 2 and first memory block (i.e. *20) used only in -m1 mode. So, overall, hash includes 3 or 4 chunks. Besides hash, compression uses 3*8mb buffers.

Decompression uses only 2*8mb buffers and no hash. Repeated data copied directly from output file, though, so you need to have enough RAM used for disk cache in order to make decompression fast.

User reports

History/Downloads

Downloads contain sources and Win32, Win64, Linux-i386, Linux-x64 executables.

Version Date Download Improvements
1.5May 11, 2010Download .zip
  • -m1: old method (compression memory = 6-7% of filesize, check matches by SHA1 digest)
  • -m2: new, default method (compression memory = 2-3% of filesize, check matches by rereading old data)
  • -index option - keep index of compressed data in separate file in order to improve compression ratio
  • 64-bit executable that's still 100% compatible but faster than 32-bit one
1.0Dec 15, 2009Download .zip
  • -delete option that delete source file after successful (de)compression
  • checking of -l value
0.8Nov 24, 2009Download .zip
  • better compression due to improved hashing and compressed format
  • faster compression on files <1gb
  • MD5 integrity checking on decompressed data
  • first 8 bytes of compressed file now contains SREP signature, helping programs like unix magic
  • exit code == 0 on success
0.7Nov 23, 2009Download .zip
  • reduced memory usage down to 6-8% of filesize. For example, 24gb file needs 256+256+960 mb memory chunks
  • now hash keeps address of the last chunk with the same contents
  • hashing improved a little
  • fixed WinXP crashing bug
0.6Nov 23, 2009Download .zip
  • fixed 64-bit version, now it properly handles files >2gb
  • fixed decompresion with non-default -l
  • -s prints stats after each block
0.5Nov 23, 2009Download .zip
  • first version that was able to compress and extract data

Other links