new software on the server, ..., almost done

[Carter Bullard]

So the only thing I can see that could cause you to use so much memory
when threads are enabled, is a poor scheduler.

The big difference between threaded vs non-threaded ra* programs, is
that in non-threaded, we process one record at a time, and so we have
a single record buffer per input that is used to do all the work.  If a
client program is a sorter or aggregator, it starts to allocate  
memory to
hold the record's cache, until either the cache is done or the program
terminates.

But with the threaded model, we have an input thread for each input, and
we have a RecordProcess(ing) thread, and we have a thread for each
output destination.  As a result, the input threads copy each record and
queue it for the RecordProcess(ing) thread.  The RecordProcess(ing)  
thread
is sitting on a timed_conditional_mutex, (20 mSec timeout) and the  
inputs
are suppose to wake up the thread when there are 100 items on the  
list, so
we're not waking up on every record.  When reading a file, the behavior
should be read 100 records and wake up the RecordProcess(ing) thread,
who should grab a chunk of records and process them, hopefully on  
another
processor.

So, have you done your test without .threads and reading a file?  If  
that is
cool, then we need to look at your scheduler.  I'll add more debug  
information
for the mutex's.  That may help.

Carter







On Sep 9, 2007, at 11:06 PM, Peter Van Epp wrote:

> 	Alas that doesn't look to be all thats wrong with clients.
>
> vanepp$ ps auxwww | grep ra3
> vanepp   26560  197.7 -36.4   786848 764120  p1  R+    7:59PM    
> 0:13.37 ra3 -r com_argus -n
> vanepp   26491   3.9 -0.1    32736   2600  p1  S     7:48PM    
> 0:20.02 /usr/local/bin/ra3 -S 192.75.244.191:560 -n -D4 -w /var/log/ 
> argus/com_argus
> vanepp   26600   0.0 -0.0    27812    112  p2  R+    8:00PM    
> 0:00.00 grep ra3
>
> vanepp$ ps auxwww | grep ra3
> vanepp   26560  195.6 -74.6  1815340 1564440  p1  R+    7:59PM    
> 0:30.84 ra3 -r com_argus -n
> vanepp   26491   3.4 -0.1    32736   2600  p1  S     7:48PM    
> 0:20.32 /usr/local/bin/ra3 -S 192.75.244.191:560 -n -D4 -w /var/log/ 
> argus/com_argus
> vanepp   26602   0.0 -0.0    27812    288  p2  R+    8:00PM    
> 0:00.00 grep ra3
>
> 	One ra instance is reading from the sensor and writing to file the
> other is reading the file the first is writing. The one reading the  
> file is
> using a huge amount of memory (although not at this moment  
> complaining about
> malloc errors).
>
> Peter Van Epp / Operations and Technical Support
> Simon Fraser University, Burnaby, B.C. Canada
>