Big O Impact of Filters

[Carter Bullard]

Not seeing this problem with memory after reading some files here.
I've run through about 15000 at a time, with no problems.
I'll do a memory check this weekend !!!

Carter

> On May 16, 2014, at 8:42 AM, Jason <dn1nj4 at gmail.com> wrote:
> 
> I'm testing the 3.0.2.27 now.  Duplicating the original test in this thread produced much more reasonable results.  When I run against a larger test data set though (around 40 input files), I am getting the following error: 
> 
> *** glibc detected *** racluster: corrupted double-linked list: 0x000000001e900470 ***
> 
> The error is the same each time I run the test.
> 
> 
> 
>> On Thu, May 15, 2014 at 10:40 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Jason,
>> So I uploaded argus-clients-3.0.7.27 that has a complete fix in
>> for the problem you reported.  FYI, the problem was that we were
>> calling the queue timeout management routines on every flow,
>> which, interestingly, really crushed the routine when the idle
>> timers and status timers were both turned on, both not in the
>> same order of magnitude, and a specific filter gets a large
>> number of hits in a short period of time...
>> 
>> That of course is / was really stupid, not really a bug, but kinda of a bug. 
>> 
>> OK, the fix that is now in has independent logic for managing the idle
>> and status timeouts. Each filter entry get a complete aggregation
>> engine, and processing queue, so we can use an efficient idle timeout
>> processing strategy, but we need to process the status timeouts 
>> independently, which we now do once every second.
>> 
>> Hopefully things are working better for you now.
>> 
>> Carter
>> 
>>> On May 15, 2014, at 11:30 AM, Carter Bullard <carter at qosient.com> wrote:
>>> 
>>> Hey Jason,
>>> Could you give this version of racluster() a run to see if it does
>>> what you want ???  The principal difference is that the output of
>>> this new racluster() will have records a bit more out of order
>>> that the other version.  
>>> 
>>> With streaming data, you may not get status reports timely (like
>>> within 0.25 seconds of the status timer expiration) but you will
>>> get correct status record reporting, driven by the idle timeout
>>> period.  I’ll improve this behavior later today.
>>> 
>>> Sorry for any inconvenience, and thanks for pushing on this !!!!
>>> 
>>> Carter
>>> 
>>> <racluster.c>
>>> 
>>>> On May 15, 2014, at 10:20 AM, Carter Bullard <carter at qosient.com> wrote:
>>>> 
>>>> Hey Jason,
>>>> Found the problem, and its a poor design assumption on my part.
>>>> Its a kind of a thrash between the status timer and the idle timer.
>>>> This does not affect rabins() or radium(), just racluster().
>>>> 
>>>> Fixing it now.
>>>> 
>>>> Carter
>>>> 
>>>>> On May 14, 2014, at 5:53 PM, Jason <dn1nj4 at gmail.com> wrote:
>>>>> 
>>>>> Hi Carter,
>>>>> 
>>>>> So I asked a very similar question last year (http://comments.gmane.org/gmane.network.argus/9110), but I can't seem to find a response.  I apologize if I'm just missing something or have just forgotten.
>>>>> 
>>>>> I am trying once again to understand why there is such a significant impact on the length of time it takes to run racluster when leveraging filters.  Here is the racluster.conf file I am testing: 
>>>>> 
>>>>> filter="udp and port domain" model="saddr daddr proto sport dport" status=600 idle=10
>>>>> filter="udp" model="saddr daddr proto sport dport" status=600 idle=60
>>>>> filter="" model="saddr daddr proto sport dport" status=600 idle=600
>>>>> 
>>>>> And here are two runs against a single argus file.  The only difference is whether or not I'm using the racluster.conf:
>>>>> 
>>>>> $ time racluster -f racluster.conf -r infile.bin -w outfile.bin -M rmon -u -c "," -m saddr proto sport dport -L0 -Z s -s stime saddr proto sport dport sbytes runtime dbytes trans state - not arp 
>>>>>  
>>>>> real    2m42.935s 
>>>>> user    2m39.274s 
>>>>> sys     0m3.288s 
>>>>>  
>>>>> $ time racluster -r infile.bin -w outfile.bin -M rmon -u -c "," -m saddr proto sport dport -L0 -Z s -s stime saddr proto sport dport sbytes runtime dbytes trans state - not arp 
>>>>>  
>>>>> real    0m1.054s 
>>>>> user    0m0.944s 
>>>>> sys     0m0.108s
>>>>> 
>>>>> Why does the filtered option take exponentially longer?
>>>>> 
>>>>> Thanks!
>>>>> Jason
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140516/51db4bc7/attachment.html>