gzip, bzip2 and xz

[Carter Bullard]

Hey Jesper,
We can add very easily, if there is a library, and an easy way to detect the type of compression.
In the routine ArgusReadConnection(), when we start to read a file or a stream, we test the first set of bytes to see if the stream is compressed.  Currently we just call bzip2.1, gzip.1 or zcat.1, based on the test.  If xz can fit into this type of strategy, then easy to add.

Is there a magic.5 type of entry for xz files ??? That’s the best test if this type of strategy can work.

Carter

On Nov 19, 2013, at 9:13 AM, Jesper Skou Jensen <jesper.skou.jensen at uni-c.dk> wrote:

> Hi guyz,
> 
> At the moment I'm using gzip to compress my argus logfiles but I would like to save more harddisk-space.
> 
> One option is to use bzip2, but it's much much slower than gzip both on compress/write but most important on decompress/read, and because of that it's not really a viable option for me.
> 
> xz on the other hand is VERY slow at compressing (doesn't matter for me, it's just a batch job in the background) but decompresses just as fast at gzip. The major advantage with xz is the huge save in compressed filesize you get. A few quick tests shows 36% saved when compared to bzip2 and 41% when compared to gzip.
> 
> File compressor;Bytes
> uncompressed argus file;105268632
> gzip;39122641
> bzip2;36041236
> xz;22977764
> xz with -e and -9; 22600708
> 
> As you can see there's only a very small gain with the last "xz -e -9", and it takes MUCH longer to compress, so it's probably not useful in real-life situations. But the xz itself looks promising.
> 
> Unfortunately Argus clients doesn't read xz by default and I would have to make my own wrapper for this to work. Would it be possible to add xz as native supported compression format for the Argus clients?
> 
> 
> --
> Regards
> Jesper
> 




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131119/627a2473/attachment.bin>