gzip, bzip2 and xz

[Jesse Bowling]

On 11/21/2013 08:53 PM, David Edelman wrote:
> xz does have a magic entry. 
> 
> # file driveWarning.txt.xz
> 	driveWarning.txt.xz: XZ compressed data
> 

My /usr/share/file/magic file says:

# http://tukaani.org/xz/xz-file-format.txt
0       ustring         \xFD7zXZ\x00            xz compressed data
!:mime  application/x-xz

Cheers,

Jesse

> --Dave
> 
> 
> -----Original Message-----
> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
> Behalf Of Carter Bullard
> Sent: Tuesday, November 19, 2013 10:22 AM
> To: Jesper Skou Jensen
> Cc: Argus
> Subject: Re: [ARGUS] gzip, bzip2 and xz
> 
> Hey Jesper,
> We can add very easily, if there is a library, and an easy way to detect the
> type of compression.
> In the routine ArgusReadConnection(), when we start to read a file or a
> stream, we test the first set of bytes to see if the stream is compressed.
> Currently we just call bzip2.1, gzip.1 or zcat.1, based on the test.  If xz
> can fit into this type of strategy, then easy to add.
> 
> Is there a magic.5 type of entry for xz files ??? That's the best test if
> this type of strategy can work.
> 
> Carter
> 
> On Nov 19, 2013, at 9:13 AM, Jesper Skou Jensen
> <jesper.skou.jensen at uni-c.dk> wrote:
> 
>> Hi guyz,
>>
>> At the moment I'm using gzip to compress my argus logfiles but I would
> like to save more harddisk-space.
>>
>> One option is to use bzip2, but it's much much slower than gzip both on
> compress/write but most important on decompress/read, and because of that
> it's not really a viable option for me.
>>
>> xz on the other hand is VERY slow at compressing (doesn't matter for me,
> it's just a batch job in the background) but decompresses just as fast at
> gzip. The major advantage with xz is the huge save in compressed filesize
> you get. A few quick tests shows 36% saved when compared to bzip2 and 41%
> when compared to gzip.
>>
>> File compressor;Bytes
>> uncompressed argus file;105268632
>> gzip;39122641
>> bzip2;36041236
>> xz;22977764
>> xz with -e and -9; 22600708
>>
>> As you can see there's only a very small gain with the last "xz -e -9",
> and it takes MUCH longer to compress, so it's probably not useful in
> real-life situations. But the xz itself looks promising.
>>
>> Unfortunately Argus clients doesn't read xz by default and I would have to
> make my own wrapper for this to work. Would it be possible to add xz as
> native supported compression format for the Argus clients?
>>
>>
>> --
>> Regards
>> Jesper
>>
> 
> 
> 
>