Removing possibly unused metadata?

[MN]

Formerly, for data that we kept long-term, rounding time stamps to the
nearest 1/4 or 1/8 of a second reduced entropy sufficiently to make a
significant difference in compressed file sizes (this will not help on
non-compressed argus files).  I can send the old code if desired, but
it was for an older version of Argus.

Now we save our longer term data in ascii format, saving just the fields
that we want, and using a combination of -p and RA_TIME_FORMAT.

Consider using xz instead of bzip2, especially if you look at the log
files frequently, as the decompression time is significantly less - at
the cost of longer compression times.  Note xz defaults to '-6'.

We've been keeping more than a years worth of data on roughly ten 1-4g/s
links.

- mike

On Oct 28, 2011, at 5:06 PM, Jason Carr wrote:

> We write argus data into five minute chunked files.  We typically have +1G
> files for those 5 minutes.  Is there any metadata that we might be able to
> purge to decrease the size significantly?
> 
> I normally only care about StartTime, flags, pro to, src/dst
> {mac,ip,port}, direction, packets, bytes, state, and user data in either
> direction.
> 
> I already gzip compress the files, I tried using bzip2 on a few test files
> and got a 1.1G file down to 500M instead of 539M, but I'm looking for a
> larger compression and/or size difference.
> 
> Thanks,
> 
> Jason