[ARGUS] Segmentation Fault with 2.0.6rc2 on FreeBSD 4.9-RELEASE

[eric]

On Fri, 2004-04-09 at 16:10:16 -0700, Peter Van Epp proclaimed...

> 	Is it possible to get a tcpdump of the input during one of these 
> crashes? With that and tcpreplay on a test machine (a big test machine :-))
> it may be possible to receate the crash. Touching .devel and .debug in the
> argus source directory and recompiling with symbols might help some as well
> (it also may slow you down enough to work even worse though).

If I can find the exact timeframe, then yes, I could sniff for a few
minutes. But anything more and we run out of disk space.

> 	It sounds like you are already writing via a socket from the sensor
> box to ra on another box (if not this is worth doing because the disk I/O on
> a single box is known to cause at least packet loss).

Nope, we're writing right to a file; using the following parameter.

ARGUS_OUTPUT_FILE=/path/to/output/file

> netstat -i
> netstat -m
> 
> after a crash would be good bets to see if the kernel is running out of mbufs.

$ netstat -i
Name  Mtu   Network    Address            Ipkts Ierrs    Opkts Oerrs  Coll
em0   1500  <Link#1>  00:00:00:xx:xx:xx 1645131079 20564  0     0      0
em1   1500  <Link#2>  00:00:00:yy:yy:yy 1586377066 19322  0     0      0

$ netstat -m
770/1120/6144 mbufs in use (current/peak/max):
        770 mbufs allocated to data
768/840/1536 mbuf clusters in use (current/peak/max)
1960 Kbytes allocated to network (42% of mb_map in use)

> I saw a kernel tuning page on the tcpreplay web page at sourceforge but a 
> quick look at the FAQ only turned up "experiment with NMBCLUSTERS in the kernel
> config file". I think there is another comment on boosting kernel buffer sizes
> in general on the BSDs that may be worth looking at somewhere there.
> 	I assume you have an ioctl such as 
> 
> /sbin/sysctl debug.bpf_bufsize=524288
> 

Ok, I have debug.bpf_bufsize: 16384  -- should I increase this?

Here's the other performance tunings I've made...

kern.maxproc=10240
kern.maxprocperuid=7680
kern.maxusers=128
kern.ipc.somaxconn=1024
kern.ipc.nmbclusters=32768

> to boost the libpcap buffer to max size? I don't think any of these are likely
> the base problem, but one or more might help if something ugly is happening 
> before the traffic gets to argus.
> 	Do you see any messages in syslog about
> 
> ArgusWriteOutSocket(0x%x) Queue Count %d
> ArgusWriteOutSocket(0x%x) failed to create file %s
> ArgusWriteOutSocket(0x%x) Exceeded Maximum Errors
> ArgusWriteOutSocket(0x%x) Queue Exceeded Maximum Limit

Nope. Also, I increased the argus buffer size by using this patch in
src/server

--- server/ArgusUtil.c.orig    Mon Apr  5 01:42:44 2004
+++ server/ArgusUtil.c Mon Apr  5 01:42:50 2004
@@ -815,9 +815,9 @@ ArgusDeleteSocket (struct ArgusSocketStr
 #include <fcntl.h>

 #define ARGUS_MAXERROR         20000
-#define ARGUS_MAXWRITENUM      2048
+#define ARGUS_MAXWRITENUM      32768

-int ArgusMaxListLength = 262144;
+int ArgusMaxListLength = 1028576;

 int ArgusReadSocket (struct ArgusSocketStruct *asock, ArgusHandler ArgusThisHandler, void *data)

> These are all in the area of code that should be the problem.

Thanks for the help.

- Eric