Big O Impact of Filters

[Jason]

I'm testing the 3.0.2.27 now.  Duplicating the original test in this thread
produced much more reasonable results.  When I run against a larger test
data set though (around 40 input files), I am getting the following error:

*** glibc detected *** racluster: corrupted double-linked list:
0x000000001e900470 ***

The error is the same each time I run the test.



On Thu, May 15, 2014 at 10:40 PM, Carter Bullard <carter at qosient.com> wrote:

> Hey Jason,
> So I uploaded argus-clients-3.0.7.27 that has a complete fix in
> for the problem you reported.  FYI, the problem was that we were
> calling the queue timeout management routines on every flow,
> which, interestingly, really crushed the routine when the idle
> timers and status timers were both turned on, both not in the
> same order of magnitude, and a specific filter gets a large
> number of hits in a short period of time...
>
> That of course is / was really stupid, not really a bug, but kinda of a
> bug.
>
> OK, the fix that is now in has independent logic for managing the idle
> and status timeouts. Each filter entry get a complete aggregation
> engine, and processing queue, so we can use an efficient idle timeout
> processing strategy, but we need to process the status timeouts
> independently, which we now do once every second.
>
> Hopefully things are working better for you now.
>
> Carter
>
> On May 15, 2014, at 11:30 AM, Carter Bullard <carter at qosient.com> wrote:
>
> Hey Jason,
> Could you give this version of racluster() a run to see if it does
> what you want ???  The principal difference is that the output of
> this new racluster() will have records a bit more out of order
> that the other version.
>
> With streaming data, you may not get status reports timely (like
> within 0.25 seconds of the status timer expiration) but you will
> get correct status record reporting, driven by the idle timeout
> period.  I’ll improve this behavior later today.
>
> Sorry for any inconvenience, and thanks for pushing on this !!!!
>
> Carter
>
> <racluster.c>
>
> On May 15, 2014, at 10:20 AM, Carter Bullard <carter at qosient.com> wrote:
>
> Hey Jason,
> Found the problem, and its a poor design assumption on my part.
> Its a kind of a thrash between the status timer and the idle timer.
> This does not affect rabins() or radium(), just racluster().
>
> Fixing it now.
>
> Carter
>
> On May 14, 2014, at 5:53 PM, Jason <dn1nj4 at gmail.com> wrote:
>
> Hi Carter,
>
> So I asked a very similar question last year (
> http://comments.gmane.org/gmane.network.argus/9110), but I can't seem to
> find a response.  I apologize if I'm just missing something or have just
> forgotten.
>
> I am trying once again to understand why there is such a significant
> impact on the length of time it takes to run racluster when leveraging
> filters.  Here is the racluster.conf file I am testing:
>
> filter="udp and port domain" model="saddr daddr proto sport dport"
> status=600 idle=10
> filter="udp" model="saddr daddr proto sport dport" status=600 idle=60
> filter="" model="saddr daddr proto sport dport" status=600 idle=600
>
> And here are two runs against a single argus file.  The only difference is
> whether or not I'm using the racluster.conf:
>
> $ time racluster -f racluster.conf -r infile.bin -w outfile.bin -M rmon -u
> -c "," -m saddr proto sport dport -L0 -Z s -s stime saddr proto sport dport
> sbytes runtime dbytes trans state - not arp
>
> real    2m42.935s
> user    2m39.274s
> sys     0m3.288s
>
> $ time racluster -r infile.bin -w outfile.bin -M rmon -u -c "," -m saddr
> proto sport dport -L0 -Z s -s stime saddr proto sport dport sbytes runtime
> dbytes trans state - not arp
>
> real    0m1.054s
> user    0m0.944s
> sys     0m0.108s
>
> Why does the filtered option take exponentially longer?
>
> Thanks!
> Jason
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140516/5c884f02/attachment.html>