Argus TopN

[carter at qosient.com]

Hey Keir,
The racluster is swapping.  If you called racluster() with "-m saddr" it shouldn't be that bad.  How did you call racluster()?

Most will split the data into smaller sets, cluster the subsets, and then cluster the clusters if there really are that many IP addrs.  And you can use rasqlinsert() to use mysql as the backing store if you are really seeing that many IP addrs.

Carter

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Keir Novik <novik at sfu.ca>
Date: Wed, 8 Sep 2010 20:18:11 
To: <carter at qosient.com>
Cc: Argus<argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] Argus TopN

No problem, you set me on the right track and I got it working.  As Peter Van Epp added in a private reply the memory requirements can be large, e.g. trying to analyze a day of our traffic the racluster process is currently using 4.2 GB memory and hasn't finished yet at 22 hours CPU time.

Regards,
Keir


On 2010-09-08, at 4:08 AM, carter at qosient.com wrote:

> Hey Keir,
> I made a mistake in my answer, as I forgot a "-w -" on the racluster() call.  A better answer is:
> 
> $ racluster -m saddr -M rmon  -r file -w -  - ip  | rasort -m bytes -s stime saddr spkts dpkts sbytes dbytes | head 
> 
> Thanks to Jesse on the list for keeping me straight.
> Carter 
> 
> ------Original Message------
> From: Keir Novik
> To: Carter Bullard
> Subject: Re: [ARGUS] Argus TopN
> Sent: Sep 7, 2010 7:36 PM
> 
> Thanks!
> 
> Regards,
> Keir
> 
> 
> On 2010-09-06, at 5:12 PM, carter at qosient.com wrote:
> 
>> Hey Keir,
>> The rmon functions are now in all the clients, and TopN is done using racluster() and rasort().
>> 
>> $ racluster -m saddr -M rmon  -r file - ip  | rasort -m bytes -s stime saddr spkts dpkts sbytes dbytes | head 
>> If you have any problems, send email,
>> Carter
>> 
>> 
>> Carter 
>> 
>> Sent from my Verizon Wireless BlackBerry
>> 
>> -----Original Message-----
>> From: Keir Novik <novik at sfu.ca>
>> Sender: argus-info-bounces+carter=qosient.com at lists.andrew.cmu.edu
>> Date: Mon, 6 Sep 2010 13:15:16 
>> To: Argus<argus-info at lists.andrew.cmu.edu>
>> Subject: [ARGUS] Argus TopN
>> 
>> What's the best way to do a TopN report (bytes per IP address) in Argus 3?  In Argus 2 I would do 
>> 
>> $ ramon -M TopN -n -s bytes -r file |head
>>    StartTime             Addr       InPkt    OutPkt    InBytes      OutBytes    
>> 2005-04-11 08:17:13         197.0.1.1 816971   395562    1132802297   22705854
>> 2005-04-11 10:17:15         1.0.12.15 28536    61199     1543399      85490108
>> 2005-04-11 09:30:06          1.0.12.5 25119    52212     1358400      73443503
>> 2005-04-11 09:56:37         1.0.12.11 21878    45413     1182885      63713137
>> 2005-04-11 10:39:30         1.0.12.19 22040    44806     1191633      63260385
>> 2005-04-11 09:24:27          1.0.12.4 15251    30746     824536       43076452
>> 2005-04-11 08:55:28          1.0.12.1 16233    30346     877564       42943674
>> 2005-04-11 10:06:41         1.0.12.13 14598    30647     789762       42933338
>> 2005-04-11 09:38:26          1.0.12.8 14286    30553     772436       42723656
>> 
>> In Argus 3, the thoughts I've had are
>> 
>> (a) use "racount - host a.b.c.d" for each IP address in turn, which is fine for a few IP addresses but doesn't scale, or
>> 
>> (b) use "racluster -m daddr - dst net a.b.c.d/e", "racluster -m saddr - src net a.b.c.d/e", and write a script of my own to add up the results.
>> 
>> but is there a better way?
>> 
>> Regards,
>> Keir
>> 
>> 
> 
> 
> 
> 
> Sent from my Verizon Wireless BlackBerry