argus 3 not combining flows?

Thu Oct 26 07:17:11 EDT 2006

You're files are too big.  Flip files faster, we flip every 5 minutes, or split the files to about 500M and racluster and then merge them back.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: mnewton at stanford.edu (MN)
Date: Wed, 25 Oct 2006 13:24:26 
To:Carter Bullard <carter at qosient.com>
Cc:argus-info at lists.andrew.cmu.edu
Subject: Re: [ARGUS] argus 3 not combining flows?

Hi - if we run racluster on one of our files:

# /usr/local/src/argus-clients-3.0.0.rc.33/bin/racluster -r argus.21.bz2 -w mn-argus.21.bz2
out of memory

That is, the racluster resident/virtual size goes up rapidly (very roughly 
400M/minute) until it hits 4+g, at which point it core dumps.  (The core
dump takes over an half an hour to write out.)

- mike

On Tue, Oct 24, 2006 at 10:34:16PM -0400, Carter Bullard wrote:
> Hey Mike,
> These look ok,  your ARGUS_FAR_STATUS_INTERVAL is 5 seconds, so
> you're getting multiple records for long lasting flows.  How does it  
> look
> after you run racluster() on the file?  You can specify a longer  
> interval,
> but that will cause argus() to use a little more memory to support the
> flow cache.   Maybe 20, or 30 seconds will help?
> 
> Carter
> 
> 
> On Oct 24, 2006, at 9:06 PM, MN wrote:
> 
> >
> >Hi - I suspect that I've just missed something fundamental...
> >
> >With Argus 3.0, our flow files are 3 times the size of Argus 2.0.
> >It looks like Argus is not putting flows together.  I've included
> >one example below (with IPs replace for privacy reasons).
> >
> >Our argus.conf file on this sensor is as:
> >
> >ARGUS_ACCESS_PORT=0
> >ARGUS_DAEMON=yes
> >ARGUS_DEBUG_LEVEL=0
> >ARGUS_FILTER=""
> >ARGUS_FILTER_OPTIMIZER=yes
> >ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
> >ARGUS_FLOW_TYPE="Bidirectional"
> >ARGUS_GENERATE_APPBYTE_METRIC=yes
> >ARGUS_GENERATE_JITTER_DATA=no
> >ARGUS_GENERATE_MAC_DATA=yes
> >ARGUS_GENERATE_RESPONSE_TIME_DATA=no
> >ARGUS_INTERFACE=eth1
> >ARGUS_MAR_STATUS_INTERVAL=60
> >ARGUS_MONITOR_ID=`hostname`
> >ARGUS_OUTPUT_FILE=/logs/argus.out
> >ARGUS_PID_PATH="/var/run"
> >ARGUS_SET_PID=yes
> >
> >(all flows tcp)
> >[...]
> >05:50:05.5,05:50:05.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:50:15.5,05:50:15.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:50:25.5,05:50:25.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:50:32.0,05:50:35.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,4,3,240,612,CON
> >05:50:45.6,05:50:45.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,1,1,54,58,CON
> >05:50:55.6,05:50:55.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,1,1,62,58,CON
> >05:51:06.5,05:51:06.8,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,195,58,CON
> >05:51:15.0,05:51:16.0,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,2,192,195,CON
> >05:51:25.7,05:51:26.1,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:51:35.7,05:51:36.0,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:51:45.8,05:51:46.1,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:51:55.8,05:51:56.1,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:52:05.8,05:52:06.8,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,4,2,309,116,CON
> >05:52:25.9,05:52:26.2,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:52:35.9,05:52:36.2,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:52:45.9,05:52:46.2,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:52:55.9,05:52:56.3,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:53:06.0,05:53:06.3,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:53:16.0,05:53:16.3,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:53:36.0,05:53:36.3,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:53:46.1,05:53:46.1,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,1,1,62,58,CON
> >05:53:56.1,05:53:56.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:54:06.1,05:54:06.8,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,2,257,116,CON
> >05:54:16.1,05:54:16.4,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:54:26.2,05:54:26.5,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:54:45.7,05:54:46.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,3,192,249,CON
> >05:54:56.2,05:54:56.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:55:06.3,05:55:06.7,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,2,257,116,CON
> >05:55:16.3,05:55:16.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:55:26.3,05:55:26.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:55:36.3,05:55:36.3,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,1,1,62,58,CON
> >05:55:56.3,05:55:56.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:56:06.3,05:56:06.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:56:16.3,05:56:16.6,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:56:26.3,05:56:26.7,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:56:36.4,05:56:36.7,10.10.197.51,2855,?>, 
> >1.11.12.13,7227,2,0,116,0,CON
> >05:56:56.4,05:56:56.7,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:57:06.4,05:57:06.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,2,116,116,CON
> >05:57:16.5,05:57:16.9,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:57:26.5,05:57:26.8,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:57:36.6,05:57:37.0,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,2,1,116,58,CON
> >05:57:46.3,05:57:47.2,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,3,192,249,CON
> >05:58:06.4,05:58:07.0,10.10.197.51,2855,<?>, 
> >1.11.12.13,7227,3,2,257,116,CON
> >[...]
> >
> >Any ideas?
> >- mike
> >
> 
> 
> 
> 