Clustering flows within a specific time interval

[manaf gharaibeh]

Hi,

I have huge Argus files (each with records of flows for an entire day). I am trying to gather statistics like the number of flows, number of different sources, or source packets that target the same destination within a given interval of time like 1 minute. I use the following command line within a Perl script to cluster flows based on destination then sort the result of that based on the number of source packets to destinations:
`racluster -nw - @arglist -m daddr -t @timeIneterval |rasort -u -m spkts -s daddr stime ltime dur spkts srate -c, > spktsSorted.dat`; 

where @arglist contains user command-line options, mainly the name of the input argus file. And @timeIneterval contains a time interval in a form like i1293864155+60s. The result is saved to spktsSorted.dat file in a comma separated format.

Now here is my problem: The argus files I have are originally sorted based on the ending time of a flow rather than the starting time of that flow. So when I run the racluster command, it will have no clue where are the flows that fall within the specified interval. It will simply search through the whole argus file, which is very expensive with huge files like the ones I'm working with. I used the option -N to limit the number of flows that racluster should find, and that reduced the time needed by the command significantly. But this is not a good solution since I might loose some flows. Or if the integer with the -N is larger than the number of flows the satisfy the specified constrains then I will have the original expensive exhaustive search problem.

So the question is: how can I cluster flows based on destination host IP within a specific time interval in a reasonable time, that is to cluster flows that were active during an interval that starts at x and ends at y based on their destination IP addresses?  
 
-Manaf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120111/e6afdce6/attachment.html>