argus crash problem

[Peter Van Epp]

On Thu, Jan 24, 2008 at 01:56:31PM -0500, Lei Wei wrote:
> Hello,
> 
> I've met with some crashing issue when using argus to continously 
> monitoring the dag interface. The version I'm now using is argus-3. It 
> happened several times that after about 10 hours of monitoring, the 
> argus process died by itself for some reason. The system is a 
> Freebsd5.4 on a 2G memory and 2G CPU machine. I periodically checked 
> the memory usage of argus process when it's running, it seemed that 
> most of the time it consumed under 400Mb of memory and above 50% CPU 
> usage, and sometime even 99% of CPU usage. Could anyone give me some 
> hint on what could possibly cause argus to crash and what I could do to 
> prevent it?
> 
> Thanks.
> Lei

	How fast and how loaded is  your link (i.e. gig, 10gig, 100megs per
second average, 2 or 3 gigs per second average :-))? As to the crashing
first place to look is in /var/log/messages since argus will write error 
conditions to syslog sometimes. I'm assuming if the dag is working that 
you have the latest argus 3.0 code (because dags haven't been working).
If there isn't anything interesting in /var/log/messages you can (perhaps,
because it will drive CPU load up) do in the argus src directory

touch .debug .devel
make clobber
./configure 
make

which enabled debug code in the argus. Then when running the argus
add a -D flag (I'd probably start with -D2 and see what that does)
it writes the debug info to stderr so you need to redirect it to a 
file (which can be a problem with a loaded CPU as well) but when a
crash happens it should at least tell us where it happened and assuming
you have corelimit set to a non zero value gdb on the core will give
Carter good information. 
	As a data point my test argus 3.0 machine looking at a gig link 
averaging about 130 megabits per second with 150 meg peaks on a quad core
IBM PPC Power5 with 4 gigs of ram running a Linux kernel with the ring 
buffer code (which is why I can get away without a dag) looks like this 
currently:

ps auxwwww | grep argus
root     10710 13.4  4.1 168256 164032 ?       SLs  Jan21 580:16 argus -Jd -P 560 -i eth0 -i eth1 -U 512 -m -F /scratch/argus.conf

so I'm going to guess your average traffic is a lot higher than mine :-). 
top on the machine looks like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
10710 root      15   0  196m 160m 1124 S   22  4.2 580:33.96 argus              

	I'll note that there is no local file activity on this machine, it 
writes the data to a socket and ra on another machine reads it and archives
it. Without dags the interfaces will lose packets due to bus contention. 
While you shouldn't be having that problem you could reduce the CPU load
by doing the same thing (write the data to a socket and read and archive 
it on another machine). I expect that if you run out of memory argus will
likely stop. 

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada