racluster crashes, and memory utilization

[Carter Bullard]

So I have a workaround, and I'd like to get some feedback.
I am going to propose a multi-stage process, but hopefully it
won't be to complex to be useful.

I have added the "-M rmon" option to ra(), which will enable:

    ra -r files -M rmon -w - | rasplit -M count 1M -w /tmp/tmpdir/\$snet

This will generate a good number of files in the /tmp/tmpdir directory,
(use any name you like).  These files will be named for the network
address of the population of data in the files, that are no bigger than
1 million records long.  If the data files are large, then additional
files with ".aaa" extensions will be build.  Then to aggregate:

    racluster -m saddr -M Replace -R /tmp/tmpdir

This will aggregate each of the intermediate files and replace them.
Use the -V flag to get racluster() to tell you what it did with each  
file.

To realize a single file with the aggregated data:

    ra -R /tmp/tmpdir -w /tmp/output.file

These will probably be sorted in string numeric order, but hopefully
that will be ok?

So, how does that sound as a workaround?  Can't currently do it with
the existing client distribution, so, ..., it should be up later  
tonight.

Carter



On Feb 26, 2007, at 3:34 PM, Carter Bullard wrote:

> Hey Michael,
> racluster() can be a hog when it comes to memory, and 1G isn't  
> really enough
> to aggregate all the IP addresses in a good network.  Now this may  
> seem
> perplexing, but when racluster() aggregates, each flow, which in  
> this case,
> is each IP address, tracks an amazing amount of information,  
> possibly too
> much information in this case.  And this problem is a thorn for a  
> lot of people
> on the list.
>
> It will take me a while to restructure the methods that racluster()  
> uses, to
> minimize the amount of memory, but it is of course doable.  If we  
> have a
> set of "what are we trying to do" with the output, then that may  
> help in getting
> an implementation out the door quickly.
>
> Same thing applies to sorting, so, ....., this is not a bad topic  
> for the list.
>
> I'm working on a workaround for ip addresses right now, so let me  
> give you
> an update in a few days.  The workaround is to use rasplit() to  
> write out the
> "-M rmon" data out into a number of files that are labeled using  
> the network
> address.  this will give us a set of files you can aggregate, and  
> then you can
> merge the data back, to have a single aggregated file.
>
> Hopefully this will work, and I'll have it up on the server tonight.
>
> Carter
>
>
> On Feb 26, 2007, at 2:10 PM, Michael Hornung wrote:
>
>> Hi, new member to the list here and a new argus user.  I'm running  
>> an argus probe and a separate collector, retrieving info via a  
>> SASL connection between the two, and the collector is writing  
>> files to disk.
>>
>> My collector is OpenBSD 4.0 on a P4 2.8Ghz with 1GB physical RAM  
>> and many times that in swap.  I'm running RC39.  When I try to  
>> combine several logs' worth of data (the log being archived when  
>> it reaches a given size) into one argus stream using 'racluster' I  
>> continually run out of memory when I do not expect to.  See an  
>> example:
>>
>> % ls -l ../archive/20070226-[45]
>> -rw-r--r--  1 argus  argus  287077860 Feb 26 09:50 ../archive/ 
>> 20070226-4
>> -rw-r--r--  1 argus  argus  295809628 Feb 26 10:00 ../archive/ 
>> 20070226-5
>>
>> % racluster -M rmon -m saddr -r ../archive/20070226-[45] -w clustered
>> racluster[11726]: 11:04:40.200048 ArgusMallocListRecord  
>> ArgusMalloc Cannot allocate memory
>> racluster[11726]: 11:04:40.200563 ArgusNewSorter ArgusCalloc error  
>> Cannot allocate memory
>> Segmentation fault (core dumped)
>>
>> (gdb) bt
>> #0  0x1c04473b in ArgusNewSorter ()
>> #1  0x1c00262f in RaParseComplete ()
>> #2  0x1c003b92 in ArgusShutDown ()
>> #3  0x1c028d69 in ArgusLog ()
>> #4  0x1c028b9e in ArgusMallocListRecord ()
>> #5  0x1c03afe2 in ArgusCopyRecordStruct ()
>> #6  0x1c002d0a in RaProcessThisRecord ()
>> #7  0x1c0029f5 in RaProcessRecord ()
>> #8  0x1c01adb8 in ArgusHandleDatum ()
>> #9  0x1c038713 in ArgusReadStreamSocket ()
>> #10 0x1c038adb in ArgusReadFileStream ()
>> #11 0x1c003829 in main ()
>>
>> Any ideas?  Thanks.
>>
>> _____________________________________________________
>>  Michael Hornung          Computing & Communications
>>  hornung at washington.edu   University of Washington
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20070226/5991f102/attachment.html>