Argus TopN

[Peter Van Epp]

On Thu, Sep 09, 2010 at 10:57:11AM +0000, carter at qosient.com wrote:
> Hey Keir,
> The racluster is swapping.  If you called racluster() with "-m saddr" it shouldn't be that bad.  How did you call racluster()?
> 
> Most will split the data into smaller sets, cluster the subsets, and then cluster the clusters if there really are that many IP addrs.  And you can use rasqlinsert() to use mysql as the backing store if you are really seeing that many IP addrs.
> 
> Carter
>

	Carter is very probably right as the likely machine that this is running
on has 4 gigs of physical ram. Keir: assuming this is one of the IBMs it will
(I think anyway) take more ram, it was just too expensive back in the day. 
These days its old and likely cheap so more ram in the machine is your first
good bet :-). The machine and OS are both 64 bit so more rem will also be 
actually usable (4 gigs is as much as a 32 bit machine can usefully use). 
	This is why the original perl traffic scripts post process once an 
hour, they ran out of ram primarily because of port scans (the network sees
or did, several full scans on any given day on a class B and about 20 class Cs).
One possibility is using topn to collect the top 100 or so hosts every hour 
writing to file then do a topn of 30 (to match the current scripts) at the end 
of the day then post process only those 30 hosts through the entire 24 hours of
data with a more restricted set of hosts to keep memory usage down. The 
scanning and smtp volume alerts for local hosts would need a separate pass 
filtered by the border router MAC to get only outbound from our net traffic. 
While the current scripts collect scan data from outside sources it isn't 
really very useful for anything and could easily be omitted. Given the new 
capability to use multiple filters writing to different output files, you may 
be able to arrange this in one pass through the original file (because I/O time
on the file read is signinfigant too) although that may run in to memory issues
as well (its a try it and see operation :-)). 

Peter Van Epp