racluster crashes, and memory utilization

[Carter Bullard]

Hey Michael,
racluster() can be a hog when it comes to memory, and 1G isn't really  
enough
to aggregate all the IP addresses in a good network.  Now this may seem
perplexing, but when racluster() aggregates, each flow, which in this  
case,
is each IP address, tracks an amazing amount of information, possibly  
too
much information in this case.  And this problem is a thorn for a lot  
of people
on the list.

It will take me a while to restructure the methods that racluster()  
uses, to
minimize the amount of memory, but it is of course doable.  If we have a
set of "what are we trying to do" with the output, then that may help  
in getting
an implementation out the door quickly.

Same thing applies to sorting, so, ....., this is not a bad topic for  
the list.

I'm working on a workaround for ip addresses right now, so let me  
give you
an update in a few days.  The workaround is to use rasplit() to write  
out the
"-M rmon" data out into a number of files that are labeled using the  
network
address.  this will give us a set of files you can aggregate, and  
then you can
merge the data back, to have a single aggregated file.

Hopefully this will work, and I'll have it up on the server tonight.

Carter


On Feb 26, 2007, at 2:10 PM, Michael Hornung wrote:

> Hi, new member to the list here and a new argus user.  I'm running  
> an argus probe and a separate collector, retrieving info via a SASL  
> connection between the two, and the collector is writing files to  
> disk.
>
> My collector is OpenBSD 4.0 on a P4 2.8Ghz with 1GB physical RAM  
> and many times that in swap.  I'm running RC39.  When I try to  
> combine several logs' worth of data (the log being archived when it  
> reaches a given size) into one argus stream using 'racluster' I  
> continually run out of memory when I do not expect to.  See an  
> example:
>
> % ls -l ../archive/20070226-[45]
> -rw-r--r--  1 argus  argus  287077860 Feb 26 09:50 ../archive/ 
> 20070226-4
> -rw-r--r--  1 argus  argus  295809628 Feb 26 10:00 ../archive/ 
> 20070226-5
>
> % racluster -M rmon -m saddr -r ../archive/20070226-[45] -w clustered
> racluster[11726]: 11:04:40.200048 ArgusMallocListRecord ArgusMalloc  
> Cannot allocate memory
> racluster[11726]: 11:04:40.200563 ArgusNewSorter ArgusCalloc error  
> Cannot allocate memory
> Segmentation fault (core dumped)
>
> (gdb) bt
> #0  0x1c04473b in ArgusNewSorter ()
> #1  0x1c00262f in RaParseComplete ()
> #2  0x1c003b92 in ArgusShutDown ()
> #3  0x1c028d69 in ArgusLog ()
> #4  0x1c028b9e in ArgusMallocListRecord ()
> #5  0x1c03afe2 in ArgusCopyRecordStruct ()
> #6  0x1c002d0a in RaProcessThisRecord ()
> #7  0x1c0029f5 in RaProcessRecord ()
> #8  0x1c01adb8 in ArgusHandleDatum ()
> #9  0x1c038713 in ArgusReadStreamSocket ()
> #10 0x1c038adb in ArgusReadFileStream ()
> #11 0x1c003829 in main ()
>
> Any ideas?  Thanks.
>
> _____________________________________________________
>  Michael Hornung          Computing & Communications
>  hornung at washington.edu   University of Washington
>

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20070226/94e2214a/attachment.html>