argus instability problem
Peter Van Epp
vanepp at sfu.ca
Tue Jul 17 11:30:34 EDT 2007
On Tue, Jul 17, 2007 at 10:41:19AM -0400, Carter Bullard wrote:
> Hey Robin,
> Thanks!!! I think that it is affecting mixed 64/32 bit sites, where
> either
> argus or radium is running on a different architecture, at least I'm
> hoping
> that is the basic problem, as that is explainable ;o)
>
> Glad you're working well, but don't stop sending opinions/comments/
> suggestions,
> as I'm sure it could be better!!!
>
> Carter
>
>
> On Jul 17, 2007, at 2:45 AM, Robin Gruyters wrote:
>
> >Hi Carter,
> >
> >The last (running) version that i'm using here, downloaded on the
> >26 of June. Sinds then it's working fine. No problems at all.
> >
> >Here is some info about my system.
> >
> ># uname -a
> >FreeBSD nsm-01 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Apr 13
> >16:00:00 CEST 2007 root at se-mt-01:/data/obj/data/src_6_2/sys/
> >YIRDIS.NSM i386
> >
> ># ldd /usr/local/sbin/argus
> >/usr/local/sbin/argus:
> > libpcap.so.4 => /lib/libpcap.so.4 (0x480b0000)
> > libwrap.so.4 => /usr/lib/libwrap.so.4 (0x480d5000)
> > libm.so.4 => /lib/libm.so.4 (0x480dc000)
> > libc.so.6 => /lib/libc.so.6 (0x480f2000)
> >
> >/lib/libpcap.so.4:
> >/usr/lib/libwrap.so.4:
> > workarounds.c 1.6 96/03/19 16:22:25
> >/lib/libm.so.4:
> >/lib/libc.so.6:
> > key_call.c 1.25 94/04/24 SMI
> >
> ># cat /etc/argus.conf
> >ARGUS_MONITOR_ID="aaa.bbb.ccc.ddd"
> >ARGUS_ACCESS_PORT=5611
> >ARGUS_BIND_IP="127.0.0.1"
> >ARGUS_INTERFACE=ste6
> >ARGUS_COLLECTOR=yes
> >ARGUS_SETUSER_ID=sguil
> >ARGUS_SETGROUP_ID=sguil
> >ARGUS_SET_PID=yes
> >ARGUS_PID_PATH=/var/run/nsm
> >ARGUS_FLOW_STATUS_INTERVAL=60
> >ARGUS_GENERATE_MAC_DATA=yes
> >ARGUS_CAPTURE_DATA_LEN=128
> >ARGUS_DEBUG_LEVEL=0
> >
> >[process list]
> >sguil 610 0.0 0.4 12956 7816 ?? Ss 26Jun07 28:05.14 /
> >usr/local/sbin/argus -d -F /etc/argus.conf
> >
> >[bpfstat]
> > 610 ste6 p--s- 84126938 0 84126938 1298 0 argus
> >
> >
> >I use radium(8) to collect the data from the Argus daemon.
> >
> >Anyway, hope you can find the problem and fix it. If you need more
> >info, let me know.
> >
> >Kind regards,
> >
> >Robin Gruyters
> >Network and Security Engineer
> >Yirdis B.V.
> >I: http://yirdis.com
> >P: +31 (0)36 5300394
> >F: +31 (0)36 5489119
> >
> >
I'm running two gig links (but at the moment only capturing one) on
an IBM P510 Power5 64 bit PPC processor. Our commodity link is doing about 100
megabits average with a 150 meg traffic limit and its the one that fails.
I'm running SUSE 10.2 with the linux pf-ring kernel mods (and so am
fairly non standard :-)). The problem looks to be timing related in that if
I crank up the debug level it takes longer to fail. It also has only been
seen to fail on the busier of my two links (which is why the other is off at
the moment). I'm capturing with ra (rather than radium so far) on two machines
an Intel 32bit P3 running FreeBSD 6.2 and a quad core Mac G5 64 bit PPC box
but the failure looks to be in the argus sensor (I may try running another
sensor here on a dual Athelon 32 bit system also on SUSE with pf-ring and
see if that also dies). It looks like something in Mar processing corrupts
memory somehow because we get a record without a time dsr and then (usually)
a crash due to heap memory corruption. Below is the debug log (not quite
detailed enough yet, I just made changes and am trying again) from a crash
last night. Below that is a patch to common/argus_util.c for clients.rc.45
which makes the client exit when the sensor dies (the close isn't currently
decrementing the open connection counter) which may help you when the sensor
dies.
argus[25203]: 17 Jul 07 00:14:51.670811 ArgusMallocListRecord (668) start 0x101d2c80
argus[25203]: 17 Jul 07 00:14:51.670833 ArgusMallocListRecord (668) returning 0x16411620
argus[25203]: 17 Jul 07 00:14:51.670857 ArgusMallocListRecord (668) start 0x101d2c80
argus[25203]: 17 Jul 07 00:14:51.670879 ArgusMallocListRecord (668) returning 0x164118d0
argus[25203]: 17 Jul 07 00:14:51.670905 ArgusDeleteFlow (0x107a26d0) returning
argus[25203]: 17 Jul 07 00:14:51.670928 ArgusDeleteFlow (0x1141d3d0) returning
argus[25203]: 17 Jul 07 00:14:51.670949 ArgusDeleteFlow (0x10cd6790) returning
argus[25203]: 17 Jul 07 00:14:51.670970 ArgusDeleteFlow (0x10f93570) returning
argus[25203]: 17 Jul 07 00:14:51.671011 ArgusDeleteFlow (0x11075470) returning
argus[25203]: 17 Jul 07 00:14:51.671034 ArgusDeleteFlow (0x10695d30) returning
argus[25203]: 17 Jul 07 00:14:51.671056 ArgusDeleteFlow (0x1052a9f0) returning
argus[25203]: 17 Jul 07 00:14:51.671078 ArgusDeleteFlow (0x104d7a50) returning
argus[25203]: 17 Jul 07 00:14:51.671099 ArgusDeleteFlow (0x10280580) returning
argus[25203]: 17 Jul 07 00:14:51.671122 ArgusDeleteFlow (0x10787c60) returning
argus[25203]: 17 Jul 07 00:14:51.671144 ArgusDeleteFlow (0x10a97ab0) returning
argus[25203]: 17 Jul 07 00:14:51.671165 ArgusDeleteFlow (0x1c5f3790) returning
argus[25203]: 17 Jul 07 00:14:51.671187 ArgusDeleteFlow (0x1109d220) returning
argus[25203]: 17 Jul 07 00:14:51.671208 ArgusDeleteFlow (0x108311a0) returning
argus[25203]: 17 Jul 07 00:14:51.671230 ArgusDeleteFlow (0x10828470) returning
argus[25203]: 17 Jul 07 00:14:51.671251 ArgusDeleteFlow (0x10d32410) returning
argus[25203]: 17 Jul 07 00:14:51.671273 ArgusDeleteFlow (0x113e6a10) returning
argus[25203]: 17 Jul 07 00:14:51.671294 ArgusDeleteFlow (0x10366320) returning
argus[25203]: 17 Jul 07 00:14:51.671316 ArgusDeleteFlow (0x102093b0) returning
argus[25203]: 17 Jul 07 00:14:51.671338 ArgusDeleteFlow (0x109061d0) returning
argus[25203]: 17 Jul 07 00:14:51.671359 ArgusDeleteFlow (0x102f4800) returning
argus[25203]: 17 Jul 07 00:14:51.671380 ArgusDeleteFlow (0x1bce8d20) returning
argus[25203]: 17 Jul 07 00:14:51.671402 ArgusDeleteFlow (0x10312a10) returning
argus[25203]: 17 Jul 07 00:14:51.671424 ArgusDeleteFlow (0x104b09f0) returning
argus[25203]: 17 Jul 07 00:14:51.671446 ArgusDeleteFlow (0x10399260) returning
argus[25203]: 17 Jul 07 00:14:51.671467 ArgusDeleteFlow (0x12c80bf0) returning
argus[25203]: 17 Jul 07 00:14:51.671490 ArgusDeleteFlow (0x10714c50) returning
argus[25203]: 17 Jul 07 00:14:51.671511 ArgusDeleteFlow (0x1c0031d0) returning
argus[25203]: 17 Jul 07 00:14:51.671533 ArgusDeleteFlow (0x1bd867a0) returning
argus[25203]: 17 Jul 07 00:14:51.671555 ArgusDeleteFlow (0x12a21180) returning
argus[25203]: 17 Jul 07 00:14:51.671577 ArgusDeleteFlow (0x10df9bb0) returning
argus[25203]: 17 Jul 07 00:14:51.671599 ArgusDeleteFlow (0x1074c900) returning
argus[25203]: 17 Jul 07 00:14:51.671621 ArgusDeleteFlow (0x108430e0) returning
argus[25203]: 17 Jul 07 00:14:51.671643 ArgusDeleteFlow (0x1031e9b0) returning
argus[25203]: 17 Jul 07 00:14:51.671664 ArgusDeleteFlow (0x10822410) returning
argus[25203]: 17 Jul 07 00:14:51.671687 ArgusDeleteFlow (0x10e0e1f0) returning
argus[25203]: 17 Jul 07 00:14:51.671709 ArgusDeleteFlow (0x10a29530) returning
argus[25203]: 17 Jul 07 00:14:51.671730 ArgusDeleteFlow (0x1293b4a0) returning
argus[25203]: 17 Jul 07 00:14:51.671751 ArgusDeleteFlow (0x1045f490) returning
argus[25203]: 17 Jul 07 00:14:51.671773 ArgusDeleteFlow (0x10a77710) returning
argus[25203]: 17 Jul 07 00:14:51.671795 ArgusDeleteFlow (0x10f75e70) returning
argus[25203]: 17 Jul 07 00:14:51.671817 ArgusDeleteFlow (0x113db430) returning
argus[25203]: 17 Jul 07 00:14:51.671839 ArgusDeleteFlow (0x101e3dc0) returning
argus[25203]: 17 Jul 07 00:14:51.671861 ArgusDeleteFlow (0x1c245190) returning
argus[25203]: 17 Jul 07 00:14:51.671883 ArgusDeleteFlow (0x104e5370) returning
argus[25203]: 17 Jul 07 00:14:51.671904 ArgusDeleteFlow (0x10311e40) returning
argus[25203]: 17 Jul 07 00:14:51.671926 ArgusDeleteFlow (0x10312230) returning
argus[25203]: 17 Jul 07 00:14:51.671948 ArgusDeleteFlow (0x12b89570) returning
argus[25203]: 17 Jul 07 00:14:51.671969 ArgusDeleteFlow (0x10ff41d0) returning
argus[25203]: 17 Jul 07 00:14:51.671991 ArgusDeleteFlow (0x10ff45c0) returning
argus[25203]: 17 Jul 07 00:14:51.672012 ArgusDeleteFlow (0x113f7150) returning
argus[25203]: 17 Jul 07 00:14:51.672034 ArgusDeleteFlow (0x1c251e50) returning
argus[25203]: 17 Jul 07 00:14:51.672056 ArgusDeleteFlow (0x110a38e0) returning
argus[25203]: 17 Jul 07 00:14:51.672078 ArgusDeleteFlow (0x110f2270) returning
argus[25203]: 17 Jul 07 00:14:51.672100 ArgusDeleteFlow (0x110f2660) returning
argus[25203]: 17 Jul 07 00:14:51.672174 ArgusGenerateStatusMarRecord(0x101d19e0, 32) start 0x0
argus[25203]: 17 Jul 07 00:14:51.672215 ArgusMallocListRecord (668) start 0x101d2c80
argus[25203]: 17 Jul 07 00:14:51.672239 ArgusMallocListRecord (668) returning 0x16411b80
argus[25203]: 17 Jul 07 00:14:51.672260 ArgusGenerateStatusMarRecord(0x101d19e0, 32) after alloc 0x16411b80
argus[25203]: 17 Jul 07 00:14:51.672284 ArgusGenerateStatusMarRecord(0x101d19e0, 32) returning 0x16411b80
argus[25203]: 17 Jul 07 00:14:51.672306 ArgusMallocListRecord (668) start 0x101d2c80
argus[25203]: 17 Jul 07 00:14:51.672327 ArgusMallocListRecord (668) returning 0x16411e30
argus[25203]: 17 Jul 07 00:14:51.672353 ArgusMallocListRecord (668) start 0x101d2c80
argus[25203]: 17 Jul 07 00:14:51.672375 ArgusMallocListRecord (668) returning 0x164120e0
argus[25203]: 17 Jul 07 00:14:51.672413 ArgusFreeListRecord (0x1c07be20) start
argus[25203]: 17 Jul 07 00:14:51.672436 ArgusFreeListRecord (0x1c07be20) returning
argus[25203]: 17 Jul 07 00:14:51.672457 ArgusGenerateRecord: time dsr not set
argus[25203]: 17 Jul 07 00:14:51.672581 ArgusShutDown(SIGHUP)
Peter Van Epp / Operations and Technical Support
Simon Fraser University, Burnaby, B.C. Canada
*** common/argus_util.c.orig Fri Jul 13 13:09:21 2007
--- common/argus_util.c Fri Jul 13 13:11:17 2007
***************
*** 16093,16098 ****
--- 16093,16100 ----
ArgusLog (LOG_ERR, "ArgusCloseInput: close error %s", strerror(errno));
input->fd = -1;
}
+
+ parser->RaParseDone++;
if (parser->RaCloseInputFd && (input->file != NULL)) {
if (fclose (input->file))
More information about the argus
mailing list