Memory Throttling and Big O Impact of "filter"

Digital Ninja dn1nj4 at
Mon Feb 4 10:24:55 EST 2013

Thanks Carter,

Is there no way to do this without backend caching?  MySQL is an option I'd
like to avoid in this instance.

For the "-M rmon" option, I am indeed intending to generate a report of:
proto Xaddr xport yport
proto Yaddr yport xport

This is an intermediate step used in generating a list of unique IP
addresses and all ports on which they communicated.  The resulting metadata
is used for analytics where both source and destination ports are used as

That said, I am still interested to know #2 & #3.  More specifically #2 is
a curiosity, but #3 is a "have to figure out" in order to move forward.


On Mon, Feb 4, 2013 at 9:53 AM, Carter Bullard <carter at> wrote:

> Hey dn1nj4,
> Two things in your email, but the big ticket is that I think you may need
> to drop the " -M rmon " from your command.
> Your question first:
> The concept of the idle timer is to push inactive flows out of the cache,
> and it is the best way to cap memory.
> You are basically aggregating within a 10 minute (600s) sliding window.
>  It generally will provide the best data reduction,
> but if there is a later record that should be merged into one of the
> inactive flows, then the aggregation doesn't really work.
> There will be multiple records for a given key, thus incomplete
> aggregation.
> Using mysql as a backing cache does do very well.  You run rasqlinsert()
> with your aggregation key,
> and tell it to use a database as a cache, and when its done, you have a
> database table of your aggregated flows that you can use rasql() to read,
> create your file, and then your done.
>    rasqlinsert -m saddr proto sport -M cache -w
> mysql://user@localhost/db/tempTable
>    rasql -r mysql://user@localhost/db/tempTable -w argus.outfile
> OK, with regard to your " -M rmon ".  This is a method for giving your
> relevant data that pertains to a single object.
> So, just for clarity, with a single flow like this:
>    tcp X.xport -> Y.yport
> with your aggregation, the -M rmon option will generate these two records:
>    tcp X xport yport
>    tcp Y yport xport
> and tally them.  I do this type of aggregation, but I always limit the
> keys to the Src side of the metrics.
> What are you trying to track ?
> Carter
> On Feb 4, 2013, at 4:09 AM, Digital Ninja <dn1nj4 at> wrote:
> Good morning Carter,
> I am trying to use racluster (v3.0.6.2) to periodically consolidate about
> 1500 argus files (~6GB) with the following options:
> -M rmon -m saddr proto sport dport -L0 -Z s
> And this runs pretty well.  It takes about 20 minutes and consumes about
> 40% of my RAM before completing.  However, I can forsee in the future where
> I may need to double the amount of data consolidation that occurs and I was
> looking for some measures to try to limit the amount of memory argus will
> use.
> I ran across an old post of a similar vein here:
> and tried to implement the following basic racluster.conf filter:
> filter="" mode="saddr proto sport dport" status=0 idle=600
> When I run racluster with this configuration file, memory utilization was
> indeed capped.  It never reached over 3%.  However, the process also ran
> for over 8 hours and never completed (I had to kill it as that is not an
> acceptable solution).
> So my questions are:
> 1. What is the best way to limit memory utilization by racluster?
> 2. What should the expected big O impact of adding filters be?
> 3. Why is adding the filter causing the jump from 20 minutes to > 8 hours?
> Thanks in advance,
> dn1nj4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the argus mailing list