[ARGUS] Segmentation Fault with 2.0.6rc2 on FreeBSD 4.9-RELEASE

[Peter Van Epp]

	Is it possible to get a tcpdump of the input during one of these 
crashes? With that and tcpreplay on a test machine (a big test machine :-))
it may be possible to receate the crash. Touching .devel and .debug in the
argus source directory and recompiling with symbols might help some as well
(it also may slow you down enough to work even worse though).
	It sounds like you are already writing via a socket from the sensor
box to ra on another box (if not this is worth doing because the disk I/O on
a single box is known to cause at least packet loss).

netstat -i
netstat -m

after a crash would be good bets to see if the kernel is running out of mbufs.
I saw a kernel tuning page on the tcpreplay web page at sourceforge but a 
quick look at the FAQ only turned up "experiment with NMBCLUSTERS in the kernel
config file". I think there is another comment on boosting kernel buffer sizes
in general on the BSDs that may be worth looking at somewhere there.
	I assume you have an ioctl such as 

/sbin/sysctl debug.bpf_bufsize=524288

to boost the libpcap buffer to max size? I don't think any of these are likely
the base problem, but one or more might help if something ugly is happening 
before the traffic gets to argus.
	Do you see any messages in syslog about

ArgusWriteOutSocket(0x%x) Queue Count %d
ArgusWriteOutSocket(0x%x) failed to create file %s
ArgusWriteOutSocket(0x%x) Exceeded Maximum Errors
ArgusWriteOutSocket(0x%x) Queue Exceeded Maximum Limit

These are all in the area of code that should be the problem.

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Fri, Apr 09, 2004 at 05:08:44PM -0500, eric wrote:
> I have argus running under daemontools on freebsd 4.9-RELEASE. I've
> done to this to take care of argus' falling down when we are pushing
> large amounts of traffic through the network (140Mbps).
> 
> However, a corefile turns up the following...
> 
> GNU gdb 4.18 (FreeBSD)
> [...]
> Core was generated by `argus'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/lib/libwrap.so.3...(no debugging symbols
> found)...done.
> Reading symbols from /usr/lib/libpcap.so.2...(no debugging symbols
> found)...done.
> Reading symbols from /usr/lib/libm.so.2...(no debugging symbols
> found)...done.
> Reading symbols from /usr/lib/libc.so.4...(no debugging symbols
> found)...done.
> Reading symbols from /usr/libexec/ld-elf.so.1...(no debugging
> symbols found)...done.
> #0  0x8052918 in free ()
> (gdb) bt
> #0  0x8052918 in free ()
> #1  0x94 in ?? ()
> #2  0x804f632 in free ()
> #3  0x805272c in free ()
> #4  0x804ebe6 in free ()
> #5  0x804e32b in free ()
> #6  0x804ad6f in free ()
> #7  0x804a14e in free ()
> (gdb) 
> 
> And the error message is...
> 
> argus[16318]: ArgusProcessPacket () ArgusWriteOutSocket Failed 
>               to Multiplexor. Shuting Down
> 
> It can't be a bottleneck to the disks; iostat shows only about
> 10MB/sec are being written to disk during high usage.
> 
> Any thoughts are appreciated!
> 
> - Eric