gzip, bzip2 and xz

[David Edelman]

xz does have a magic entry. 

# file driveWarning.txt.xz
	driveWarning.txt.xz: XZ compressed data

--Dave


-----Original Message-----
From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
[mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
Behalf Of Carter Bullard
Sent: Tuesday, November 19, 2013 10:22 AM
To: Jesper Skou Jensen
Cc: Argus
Subject: Re: [ARGUS] gzip, bzip2 and xz

Hey Jesper,
We can add very easily, if there is a library, and an easy way to detect the
type of compression.
In the routine ArgusReadConnection(), when we start to read a file or a
stream, we test the first set of bytes to see if the stream is compressed.
Currently we just call bzip2.1, gzip.1 or zcat.1, based on the test.  If xz
can fit into this type of strategy, then easy to add.

Is there a magic.5 type of entry for xz files ??? That's the best test if
this type of strategy can work.

Carter

On Nov 19, 2013, at 9:13 AM, Jesper Skou Jensen
<jesper.skou.jensen at uni-c.dk> wrote:

> Hi guyz,
> 
> At the moment I'm using gzip to compress my argus logfiles but I would
like to save more harddisk-space.
> 
> One option is to use bzip2, but it's much much slower than gzip both on
compress/write but most important on decompress/read, and because of that
it's not really a viable option for me.
> 
> xz on the other hand is VERY slow at compressing (doesn't matter for me,
it's just a batch job in the background) but decompresses just as fast at
gzip. The major advantage with xz is the huge save in compressed filesize
you get. A few quick tests shows 36% saved when compared to bzip2 and 41%
when compared to gzip.
> 
> File compressor;Bytes
> uncompressed argus file;105268632
> gzip;39122641
> bzip2;36041236
> xz;22977764
> xz with -e and -9; 22600708
> 
> As you can see there's only a very small gain with the last "xz -e -9",
and it takes MUCH longer to compress, so it's probably not useful in
real-life situations. But the xz itself looks promising.
> 
> Unfortunately Argus clients doesn't read xz by default and I would have to
make my own wrapper for this to work. Would it be possible to add xz as
native supported compression format for the Argus clients?
> 
> 
> --
> Regards
> Jesper
> 




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6283 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131121/000bcc93/attachment.bin>