Memory Throttling and Big O Impact of "filter"

[Digital Ninja]

Thanks Carter,

Is there no way to do this without backend caching?  MySQL is an option I'd
like to avoid in this instance.

For the "-M rmon" option, I am indeed intending to generate a report of:
proto Xaddr xport yport
proto Yaddr yport xport

This is an intermediate step used in generating a list of unique IP
addresses and all ports on which they communicated.  The resulting metadata
is used for analytics where both source and destination ports are used as
discriminators.

That said, I am still interested to know #2 & #3.  More specifically #2 is
a curiosity, but #3 is a "have to figure out" in order to move forward.

dn1nj4


On Mon, Feb 4, 2013 at 9:53 AM, Carter Bullard <carter at qosient.com> wrote:

> Hey dn1nj4,
> Two things in your email, but the big ticket is that I think you may need
> to drop the " -M rmon " from your command.
>
> Your question first:
> The concept of the idle timer is to push inactive flows out of the cache,
> and it is the best way to cap memory.
> You are basically aggregating within a 10 minute (600s) sliding window.
>  It generally will provide the best data reduction,
> but if there is a later record that should be merged into one of the
> inactive flows, then the aggregation doesn't really work.
> There will be multiple records for a given key, thus incomplete
> aggregation.
>
> Using mysql as a backing cache does do very well.  You run rasqlinsert()
> with your aggregation key,
> and tell it to use a database as a cache, and when its done, you have a
> database table of your aggregated flows that you can use rasql() to read,
> create your file, and then your done.
>
>    rasqlinsert -m saddr proto sport -M cache -w
> mysql://user@localhost/db/tempTable
>    rasql -r mysql://user@localhost/db/tempTable -w argus.outfile
>
> OK, with regard to your " -M rmon ".  This is a method for giving your
> relevant data that pertains to a single object.
> So, just for clarity, with a single flow like this:
>
>    tcp X.xport -> Y.yport
>
> with your aggregation, the -M rmon option will generate these two records:
>
>    tcp X xport yport
>    tcp Y yport xport
>
> and tally them.  I do this type of aggregation, but I always limit the
> keys to the Src side of the metrics.
> What are you trying to track ?
>
> Carter
>
>
> On Feb 4, 2013, at 4:09 AM, Digital Ninja <dn1nj4 at gmail.com> wrote:
>
> Good morning Carter,
>
> I am trying to use racluster (v3.0.6.2) to periodically consolidate about
> 1500 argus files (~6GB) with the following options:
>
> -M rmon -m saddr proto sport dport -L0 -Z s
>
> And this runs pretty well.  It takes about 20 minutes and consumes about
> 40% of my RAM before completing.  However, I can forsee in the future where
> I may need to double the amount of data consolidation that occurs and I was
> looking for some measures to try to limit the amount of memory argus will
> use.
>
> I ran across an old post of a similar vein here:
>
> http://permalink.gmane.org/gmane.network.argus/8306
>
> and tried to implement the following basic racluster.conf filter:
>
> filter="" mode="saddr proto sport dport" status=0 idle=600
>
> When I run racluster with this configuration file, memory utilization was
> indeed capped.  It never reached over 3%.  However, the process also ran
> for over 8 hours and never completed (I had to kill it as that is not an
> acceptable solution).
>
> So my questions are:
> 1. What is the best way to limit memory utilization by racluster?
> 2. What should the expected big O impact of adding filters be?
> 3. Why is adding the filter causing the jump from 20 minutes to > 8 hours?
>
> Thanks in advance,
> dn1nj4
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130204/351bf08b/attachment.html>