a memory oddness (possibly a Mac oddness)

[Carter Bullard]

Hey Karl,
Ok, I have some sorting solutions that we can work with, but they are 
specific
to attributes like time, because the probes generate partially sorted 
streams
based on time.  What other objects do you think you want to have fast 
sorting
support for?

Carter


Karl Tatgenhorst wrote:

>On Tue, 2007-02-06 at 10:29 -0500, Carter Bullard wrote:
>  
>
>>Hey Karl,
>>argus-3.0 records are larger, but not necessarily so, so we can
>>work on making that a priority to fix after we release the code.
>>Getting a sense of what we want to do in this area, like very
>>fast support for sorting, aggregating, merging, spliting, filtering
>>searching, x ing ..., whatever, will be good efforts in the coming
>>months.
>>
>>    
>>
>
>  Yes. We want very fast sorting. All other things can be accomplished
>through the clever use of network devices (ie... merging can be done
>through span ports etc...). 
>
>
>  
>
>>Until then, if you want to have multiple accesses to a set of
>>streams of argus data, radium() is the program of choice.  argus()
>>is designed to support multiple output streams, but its a strain,
>>as each record now needs to be copied.  There is a configuration
>>option to keep argus from supporting multiple readers at a time,
>>to avoid the additional processing burden to support them.  But,
>>failures are not the desired outcome, so we need to work on solving
>>that problem.
>>
>>Can't elaborate right now (in a meeting) but lets keep on this thread
>>to get it fixed.
>>
>>Carter
>>    
>>
>Radium was how I was using it when I had two streams of data. Now
>however I have one stream and radium seems not to like that.
>
>Karl
>
>  
>
>>
>>
>>
>>Karl Tatgenhorst wrote:
>>
>>    
>>
>>> I forgot that we had run into this very same problem. I found that
>>>Racluster was very piggish in it's use of memory. We had two seperate
>>>streams that we were merging into one, so we had border1.file and merge
>>>with border2.file. Racluster was never able to do it for us. 
>>>
>>> I will add to this one other problem we encounter with Argus. On our
>>>collector (the machine running ra, not argus) if we start a second argus
>>>socket program (rahist - ra etc...) the main ra process will eventually
>>>(within a short time) die. I find that very frustrating.
>>>
>>>Karl
>>>
>>>On Mon, 2007-02-05 at 20:44 -0500, Carter Bullard wrote:
>>> 
>>>
>>>      
>>>
>>>>Hmmm, racluster will aggregate across files, and so you may just be 
>>>>running out
>>>>of memory.  There are two things to consider.  Try running racluster() 
>>>>with a conf
>>>>file that sets an idle time.  This will flush out the very short tcp 
>>>>flows and scanners.
>>>>The other thing to do is have racluster() process each file 
>>>>individually, rather than
>>>>aggregate across all the files, by using the "-M ind" option, "process 
>>>>files independantly".
>>>>That will help, but it will not merge data that crosses files.
>>>>
>>>>Carter
>>>>
>>>>
>>>>Peter Van Epp wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>	I'm trying to run racluster against an archive. I started with a 
>>>>>script that does 24 hours (one hour at a time) and discovered that my 2 gig
>>>>>Mac ran out of memory a couple of files in. Rebooting (which seems to be 
>>>>>required to free VM) and running them individually seems to still run out
>>>>>of memory (perhaps indicating a memory leak) on about the second file:
>>>>>
>>>>>test4:/var/log/argus vanepp$ /usr/local/bin/racluster -r  /archive/argus3/com_argus.archive/2007/01/30/com_argus.2007.01.30.09.00.00.0.gz -w  /archive/argus3c/com_argus.archive/2007/01/30/com_argus.2007.01.30.09.00.00.0
>>>>>test4:/var/log/argus vanepp$ /usr/local/bin/racluster -r  /archive/argus3/com_argus.archive/2007/01/30/com_argus.2007.01.30.10.00.01.0.gz -w  /archive/argus3c/com_argus.archive/2007/01/30/com_argus.2007.01.30.10.00.01.0
>>>>>racluster(364) malloc: *** vm_allocate(size=1069056) failed (error code=3)
>>>>>racluster(364) malloc: *** error: can't allocate region
>>>>>racluster(364) malloc: *** set a breakpoint in szone_error to debug
>>>>>Bus error
>>>>>
>>>>>	I just started moving the source archive from the Mac to one of the 
>>>>>IBMs with 4 gigs of ram to see if the same thing will happen there. I'd 
>>>>>think that the racluster task ending should release all the memory its holding
>>>>>which doesn't seem to be happening (but may be a Mac issue rather than 
>>>>>argus). 
>>>>>
>>>>>Peter Van Epp / Operations and Technical Support 
>>>>>Simon Fraser University, Burnaby, B.C. Canada
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>> 
>>>
>>>      
>>>
>
>
>  
>