Memory Throttling and Big O Impact of "filter"

[Carter Bullard]

Hey dn1nj4,
Two things in your email, but the big ticket is that I think you may need to drop the " -M rmon " from your command.

Your question first:
The concept of the idle timer is to push inactive flows out of the cache, and it is the best way to cap memory.  
You are basically aggregating within a 10 minute (600s) sliding window.  It generally will provide the best data reduction,
but if there is a later record that should be merged into one of the inactive flows, then the aggregation doesn't really work.
There will be multiple records for a given key, thus incomplete aggregation.

Using mysql as a backing cache does do very well.  You run rasqlinsert() with your aggregation key,
and tell it to use a database as a cache, and when its done, you have a database table of your aggregated flows that you can use rasql() to read, create your file, and then your done.

   rasqlinsert -m saddr proto sport -M cache -w mysql://user@localhost/db/tempTable
   rasql -r mysql://user@localhost/db/tempTable -w argus.outfile

OK, with regard to your " -M rmon ".  This is a method for giving your relevant data that pertains to a single object.
So, just for clarity, with a single flow like this:

   tcp X.xport -> Y.yport

with your aggregation, the -M rmon option will generate these two records:

   tcp X xport yport
   tcp Y yport xport

and tally them.  I do this type of aggregation, but I always limit the keys to the Src side of the metrics.
What are you trying to track ?

Carter


On Feb 4, 2013, at 4:09 AM, Digital Ninja <dn1nj4 at gmail.com> wrote:

> Good morning Carter,
> 
> I am trying to use racluster (v3.0.6.2) to periodically consolidate about 1500 argus files (~6GB) with the following options: 
> 
> -M rmon -m saddr proto sport dport -L0 -Z s 
> 
> And this runs pretty well.  It takes about 20 minutes and consumes about 40% of my RAM before completing.  However, I can forsee in the future where I may need to double the amount of data consolidation that occurs and I was looking for some measures to try to limit the amount of memory argus will use.  
> 
> I ran across an old post of a similar vein here:
> http://permalink.gmane.org/gmane.network.argus/8306
> and tried to implement the following basic racluster.conf filter: 
> 
> filter="" mode="saddr proto sport dport" status=0 idle=600
> 
> When I run racluster with this configuration file, memory utilization was indeed capped.  It never reached over 3%.  However, the process also ran for over 8 hours and never completed (I had to kill it as that is not an acceptable solution).
> 
> So my questions are: 
> 1. What is the best way to limit memory utilization by racluster?
> 2. What should the expected big O impact of adding filters be?
> 3. Why is adding the filter causing the jump from 20 minutes to > 8 hours?
> 
> Thanks in advance,
> dn1nj4

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130204/2e983083/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130204/2e983083/attachment.bin>