Big O Impact of Filters

[Carter Bullard]

Hey Jason,
Thanks for testing this.  Any chance you can run using gdb to see where its
running into trouble ???  To compile with symbols:

   % touch .devel
   % ./configure
   % make clean
   % make

If it breaks in with the same error, then type

   (gdb) where

That should be very helpful !!!!

Carter

On May 16, 2014, at 8:42 AM, Jason <dn1nj4 at gmail.com> wrote:

> I'm testing the 3.0.2.27 now.  Duplicating the original test in this thread produced much more reasonable results.  When I run against a larger test data set though (around 40 input files), I am getting the following error: 
> 
> *** glibc detected *** racluster: corrupted double-linked list: 0x000000001e900470 ***
> 
> The error is the same each time I run the test.
> 
> 
> 
> On Thu, May 15, 2014 at 10:40 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Jason,
> So I uploaded argus-clients-3.0.7.27 that has a complete fix in
> for the problem you reported.  FYI, the problem was that we were
> calling the queue timeout management routines on every flow,
> which, interestingly, really crushed the routine when the idle
> timers and status timers were both turned on, both not in the
> same order of magnitude, and a specific filter gets a large
> number of hits in a short period of time...
> 
> That of course is / was really stupid, not really a bug, but kinda of a bug. 
> 
> OK, the fix that is now in has independent logic for managing the idle
> and status timeouts. Each filter entry get a complete aggregation
> engine, and processing queue, so we can use an efficient idle timeout
> processing strategy, but we need to process the status timeouts 
> independently, which we now do once every second.
> 
> Hopefully things are working better for you now.
> 
> Carter
> 
> On May 15, 2014, at 11:30 AM, Carter Bullard <carter at qosient.com> wrote:
> 
>> Hey Jason,
>> Could you give this version of racluster() a run to see if it does
>> what you want ???  The principal difference is that the output of
>> this new racluster() will have records a bit more out of order
>> that the other version.  
>> 
>> With streaming data, you may not get status reports timely (like
>> within 0.25 seconds of the status timer expiration) but you will
>> get correct status record reporting, driven by the idle timeout
>> period.  I’ll improve this behavior later today.
>> 
>> Sorry for any inconvenience, and thanks for pushing on this !!!!
>> 
>> Carter
>> 
>> <racluster.c>
>> 
>> On May 15, 2014, at 10:20 AM, Carter Bullard <carter at qosient.com> wrote:
>> 
>>> Hey Jason,
>>> Found the problem, and its a poor design assumption on my part.
>>> Its a kind of a thrash between the status timer and the idle timer.
>>> This does not affect rabins() or radium(), just racluster().
>>> 
>>> Fixing it now.
>>> 
>>> Carter
>>> 
>>> On May 14, 2014, at 5:53 PM, Jason <dn1nj4 at gmail.com> wrote:
>>> 
>>>> Hi Carter,
>>>> 
>>>> So I asked a very similar question last year (http://comments.gmane.org/gmane.network.argus/9110), but I can't seem to find a response.  I apologize if I'm just missing something or have just forgotten.
>>>> 
>>>> I am trying once again to understand why there is such a significant impact on the length of time it takes to run racluster when leveraging filters.  Here is the racluster.conf file I am testing: 
>>>> 
>>>> filter="udp and port domain" model="saddr daddr proto sport dport" status=600 idle=10
>>>> filter="udp" model="saddr daddr proto sport dport" status=600 idle=60
>>>> filter="" model="saddr daddr proto sport dport" status=600 idle=600
>>>> 
>>>> And here are two runs against a single argus file.  The only difference is whether or not I'm using the racluster.conf:
>>>> 
>>>> $ time racluster -f racluster.conf -r infile.bin -w outfile.bin -M rmon -u -c "," -m saddr proto sport dport -L0 -Z s -s stime saddr proto sport dport sbytes runtime dbytes trans state - not arp 
>>>>  
>>>> real    2m42.935s 
>>>> user    2m39.274s 
>>>> sys     0m3.288s 
>>>>  
>>>> $ time racluster -r infile.bin -w outfile.bin -M rmon -u -c "," -m saddr proto sport dport -L0 -Z s -s stime saddr proto sport dport sbytes runtime dbytes trans state - not arp 
>>>>  
>>>> real    0m1.054s 
>>>> user    0m0.944s 
>>>> sys     0m0.108s
>>>> 
>>>> Why does the filtered option take exponentially longer?
>>>> 
>>>> Thanks!
>>>> Jason
>>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140516/deaf958f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140516/deaf958f/attachment.bin>