argus suggestions please

[Michael Hornung]

Thanks Carter, this is what I was hoping to hear!  You guessed my setup 
exactly, though I've got a problem with what you sent, and I suspect it 
may be related to the amount of data.  The box I'm using to do processing 
is x86 linux (RHEL5) with 2 x dual core 2ghz CPUs and 4GB of RAM.

% du -hs archive/2007/10/04
20G     archive/2007/10/04

% racluster -R archive/2007/10/04 -M norep -w foo -- ip
Segmentation fault

strace shows:
brk(0x4e0be000)                         = 0x4e09d000
mmap2(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, 
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, 
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, 
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, 
-1, 0) = -1 ENOMEM (Cannot allocate memory)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++


I ran top while racluster was running and it seems that the process runs 
out of memory, and I'm nearing the system's limits...so what can be done 
about this?

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 7551 argus     20   0 3062m 2.9g  836 R   76 83.2   0:38.64 racluster

-Mike

On Fri, 5 Oct 2007 at 14:19, Carter Bullard wrote:

|Hey Michael,
|I think asking these questions are great!!!  As it gets examples into the
|mailing list,
|where people can search etc....
|
|So, you have a daily directory and you want a report based on IP top talkers.
|Lets say the directory is in the standard argus archive format, and we'll do
|yesterday.
|Here is the set of commands that I would use:
|
|  racluster -R archive/2007/10/04 -M norep -w - -- ip | \
|  racluster -M rmon -m saddr -w - | \
|  rasort -m bytes -s saddr trans:10 sbytes:14 dbytes:14
|
|So what does this do:
|  racluster -R archive... -M norep -w - -- ip      This program will read in a
|days worth of IP data and assemble all the flow status
|      reports into individual flow report.  We need to do this because you said
|you wanted
|      to know how many flows there were.  The "-M norep" option sez don't
|report the
|      merge statistics for aggregations.  This allows for a single record to be
|      tallied as a single flow.  And we write the output to stdout.
|
|  racluster -M rmon -m saddr -w -
|     This program will read in the stream of single flow reports from stdin and
|generate
|      the top talker stats.  The rmon option pushes the identifiers to the src
|fields, and
|      the -m option , and write the output to stdout.
|
|  rasort -m bytes -s saddr trans:10 sbytes:14 dbytes:14
|     This program sorts the output based on total bytes for each top talker.
|      and prints out the IP address, the number of flows, the bytes transmitted
|by
|      the talker and the bytes received.
|
|  Now if you want the top 20 talkers, you need to select the first 20 records
|  from the rasort(), to do this:
|  racluster -R archive/2007/10/04 -M norep -w - -- ip | \
|  racluster -M rmon -m saddr -w - | \
|  rasort -m bytes -w - |\
|  ra -N 20 -s saddr trans:10 sbytes:14 dbytes:14
|
|
|If you try this and get something weird, send mail!!  It would be
|good if we can get a "standard" set of calls that people understand.
|
|Carter
|
|Michael Hornung wrote:
|> I have an ra reading from a remote argus collector 24x7, and every 5 minutes
|> the argus file is archived; at the end of a day I have 290 files representing
|> the traffic from that day.
|> 
|> Let's say I want to make a list of the top talkers, sorted by total bytes
|> transferred.  Given those top talkers, I want to see the following as text,
|> and/or alternately graphed, for each top talker:
|> 
|> IP
|> # flows
|> # bytes rcvd
|> # bytes sent
|> 
|> Can you recommend a command-line that's going to give me this?  The profusion
|> of argus utilities and a lack of examples is making this hard for me.
|> Thanks.
|> 
|> -Mike
|> 
|
|