Removing possibly unused metadata?

[Carter Bullard]

Hey Jason,
You should be able to remove the mac section, if you generated it, as that
doesn't generally have long term significance, assuming you realized that
the IP / mac pairing isn't anomalous.  You can generate the IP/mac report without
a massive hit, and then remove the mac DSRs.  You can remove the net DSR.
It contains the rtp and the tcp performance data that generally isn't of long term
interest, although it does have important forensics relevance if you're
concerned about TCP hijack detection or other TCP bad things.  But for many
situations, you can remove it.   The encaps DSR isn't big but it is 8 bytes
per record, and may not be a long lived one.

These would remove some significant bytes, and shouldn't be generally missed.

Carter

On Oct 28, 2011, at 5:06 PM, Jason Carr wrote:

> We write argus data into five minute chunked files.  We typically have +1G
> files for those 5 minutes.  Is there any metadata that we might be able to
> purge to decrease the size significantly?
> 
> I normally only care about StartTime, flags, pro to, src/dst
> {mac,ip,port}, direction, packets, bytes, state, and user data in either
> direction.
> 
> I already gzip compress the files, I tried using bzip2 on a few test files
> and got a 1.1G file down to 500M instead of 539M, but I'm looking for a
> larger compression and/or size difference.
> 
> Thanks,
> 
> Jason
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20111103/37e1866f/attachment.bin>