Why does argus use so much disk space?

[carter at qosient.com]

Hey Kieran,
Yes, I need to work on the docs. I don't have access to code right this second, and its been a while since I looked at rastrip(),  so take all info in this reply with a HUGE grain of salt, so to speak.  

So, experiment with options like "-m -net" (or something like it) to remove the tcp network DSR.  That should reduce the size quite a bit.  Or try "-m time flow metric" to see if you get something useful.  I think the "usage" output, when you use the "-h" option, will have all the keywords.

Yes, you can pipe rastrip().  Try something like this:
   rastrip -S server -w - | rasplit 

With the appropriate options.

Carter


Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: Kieran Rhysling <krhysling at yahoo.com>
Date: Thu, 30 Nov 2006 11:59:27 
To:carter at qosient.com
Cc:Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] Why does argus use so much disk space?

carter at qosient.com wrote:
> Hey Kieran,
> The TCP stats are pretty extensive.  Base sequence numbers, window performance, window sizes, current window bytes, etc...   
> 
> Use rastrip() to remove data that you're not interested in.  I remember that there is an outstanding issue with rastrip(), but it should allow you to test stripping the data to the bare bones, so to speak.
> 
> Carter
> 
> Carter Bullard
> QoSient LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax  
> 
> -----Original Message-----
> From: Kieran Rhysling <krhysling at yahoo.com>
> Date: Wed, 29 Nov 2006 09:18:14 
> To:argus-info at lists.andrew.cmu.edu
> Subject: [ARGUS] Why does argus use so much disk space?
> 
> Hi,
> 
> I'm hoping to replace snort's tcp stream stats with argus to get
> information on udp and icmp transactions in addition to tcp.
> 
> However, argus 3.0 rc35 uses about 3.0GB per day to capture tcp
> alone compared to snort using about 600MB for snort's tcp stream stats.
> 
> I have explicitly set ARGUS_CAPTURE_DATA_LEN=0 and left most of the
> rest of argus.conf at defaults.
> 
> I know I can shrink that 3.0GB to about 1.2GB with bzip2 -9 but it's
> still a lot for my server limitations and the amount of historical data
> I need to keep online.
> 
> Is the big difference in size between snort and argus due solely to all
> the additional fields argus captures that snort does not?
> 
> Is there a way to customize which fields argus captures for each
> transaction? For instance, could I just capture the classic 5 tuple
> and src/dst packets and bytes?
> 
> Or is argus the wrong tool for what I'm trying to do and I should look
> at something like nprobe instead?
> 
> By the way, sampling is not an answer for me. I need every transaction.
> 
> Thanks in advance,
> Kieran
> 
Thanks, Carter. That sounds like what I need. I didn't notice it before
because I was reading the man pages rather than looking at the included
binaries. Should have known better since the documentation isn't final
yet. :-)

Is there any documentation of what each DSR is? All I have to go on is
the rastrip usage info which is a little terse. I'm not sure what to
remove and what to leave.

Also, is it possible to use rastrip in a pipe with argus to avoid saving
the undesired info to disk rather than removing it later?

Thanks for all your help (and your hard work on argus),
Kieran