RC.39 (and 40) 'argus' segfault on Fedora Core 6

Wed Feb 28 14:13:10 EST 2007

It appears I am cursed here.  =)
I tried running a 'tcpdump' capture through argus to try reproducing the 
segfaults I have been experiencing.  I got another segfault, but a 
different one.  Is this the right way to run a tcpdump pcap file through 
argus instead of reading through an interface?

	argus -X -r /home/argus/cap.dump

The tcpdump file was generated via:
	tcpdump -li eth1 -s128 -w cap.dump ''

# ls -l /home/argus/cap.dump
-rw-r--r-- 1 root root 4207648087 Feb 28 10:39 /home/argus/cap.dump

# argus -X -r /home/argus/cap.dump -w /dev/null
argus[29483]: 28 Feb 07 10:59:19.057577 started
Segmentation fault

gdb shows:

(gdb) run -X -r /home/argus/cap.dump -w /dev/null
Starting program: /usr/local/sbin/argus -X -r /home/argus/cap.dump -w /dev/null
argus[29540]: 28 Feb 07 11:05:21.421096 started

Program received signal SIGSEGV, Segmentation fault.
0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
    at ArgusSource.c:99
99         if ((inf->ArgusPd = pcap_open_live(device->name, 
src->ArgusSnapLen, !src->Arguspflag, 100, errbuf)) != NULL) {

(gdb) where
#0  0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
    at ArgusSource.c:99
#1  0x080574bc in ArgusGetPackets (src=0xb7da4008) at ArgusSource.c:1547
#2  0x0804b2eb in main (argc=6, argv=0xbfe608a4) at argus.c:460

(gdb) bt full
#0  0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
    at ArgusSource.c:99
        device = (struct ArgusDeviceStruct *) 0x0
        errbuf = 
"ï¿½002ï¿½\002ï¿½\002ï¿½b\aï¿½_\a\b\004\000\000\000ï¿½002ï¿½\004ï¿½F\235\000ï¿½216\004\bï¿½002ï¿½\230\b\bï¿½002ï¿½\214(!\000\000\000\000ï¿½\006\00028 
Feb 07 11:05:21.421096 
started\n\000\b\001\000\000\000ï¿½\235\000\000\000\000\020\214\005ï¿½001\000\000\000\000\020", 
'\0' <repeats 38 times>, 
"$\235\021\000\210ï¿½ï¿½\000\000\000\000ï¿½ï¿½ï¿½ï¿½ï¿½\235\000ï¿½\235\000ï¿½\204\004\bï¿½004ï¿½z\234\000\200H\235\000ï¿½ï¿½001\000\000\000\001\000\000\000\000\000\000\000ï¿½216\004\b\000\000\000\000ï¿½\n\b"...
#1  0x080574bc in ArgusGetPackets (src=0xb7da4008) at ArgusSource.c:1547
        ArgusReadMask = {__fds_bits = {0 <repeats 32 times>}}
        ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
        ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
        tmp = 6
        i = 0
        width = 0
        noerror = 1
        fd = 0
        found = 0
        up = 0
        wait = {tv_sec = 0, tv_usec = 20000}
#2  0x0804b2eb in main (argc=6, argv=0xbfe608a4) at argus.c:460
        commandlinew = 1
        doconf = 1
        dodebug = 0
        i = 6
        pid = 0
        tmparg = 0xbfe60bb0 "/dev/null"
        filter = 0x0
        statbuf = {st_dev = 0, __pad1 = 0, __st_ino = 0, st_mode = 3219523608, 
  st_nlink = 0, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, 
  st_size = -4618995228131459072, st_blksize = 10307272, 
  st_blocks = 134516652, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {
    tv_sec = -163754450, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 
0}, 
  st_ino = 0}
        host = (struct hostent *) 0x80a6720
        commandlinei = 0
        op = -1
        path = "/usr/local/sbin/argus", '\0' <repeats 8170 times>

(gdb) print device->name
Cannot access memory at address 0x4
(gdb) print src->ArgusSnapLen
$3 = 96
(gdb) print src->Arguspflag
$4 = 0

-Mike

On Wed, 28 Feb 2007 at 10:46, Michael Hornung wrote:

|I got a different segfault today:
|
|Program received signal SIGSEGV, Segmentation fault.
|0x0805940f in ArgusCreateFlowKey (model=0x87af008, flow=0x87b0290, 
|    hstruct=0x87b0200) at ArgusUtil.c:704
|704           hstruct->hash ^= *ptr++;
|
|
|(gdb) where
|#0  0x0805940f in ArgusCreateFlowKey (model=0x87af008, flow=0x87b0290, 
|    hstruct=0x87b0200) at ArgusUtil.c:704
|#1  0x0804e6af in ArgusProcessPacket (model=0x87af008, p=0x87b260a "", 
|    length=90, tvp=0xbfc03298, type=0) at ArgusModeler.c:1004
|#2  0x0805545d in ArgusEtherPacket (user=0xb7e2b008 "", h=0xbfc03298, 
|    p=0x87b260a "") at ArgusSource.c:608
|#3  0x08063e18 in pcap_read_linux ()
|#4  0x0805718c in ArgusGetPackets (src=0xb7e2b008) at ArgusSource.c:1477
|#5  0x0804b2eb in main (argc=1, argv=0xbfc03664) at argus.c:460
|
|
|(gdb) print hstruct->hash
|$1 = 4251857491
|(gdb) print ptr
|$2 = (unsigned int *) 0xc023000
|(gdb) print *ptr
|Cannot access memory at address 0xc023000
|
|
|Could this be a result of bad physical memory in this box?
|
|I captured traffic with tcpdump while running argus and will see if I can 
|reproduce the crash that way.
|
|-Mike
|
|On Wed, 28 Feb 2007 at 08:04, Michael Hornung wrote:
|
||(gdb) print ArgusMallocList
||$1 = (struct ArgusMemoryList *) 0x9e1d538
||(gdb) print ArgusMallocList->end
||$2 = (struct ArgusMemoryHeader *) 0x54a96bb8
||(gdb) print ArgusMallocList->end->nxt
||Cannot access memory at address 0x54a96bb8
||
||-Mike
||
||On Tue, 27 Feb 2007 at 23:30, Carter Bullard wrote:
||
|||This is very odd.  When you get to someplace in gdb, with a Segmentation
|||fault, try to find out what variable is having problems by printing the actual
|||values:
|||
|||  (gdb) print ArgusMallocList
|||  (gdb) print ArgusMallocList->end
|||  (gdb) print ArgusMallocList->end->nxt
|||
|||I would guess that ArgusMallocList doesn't exist, or is corrupted.
|||This can happen for a number of reasons, but it may be useful to
|||try to get a packet trace that generates your errors.  Maybe a
|||bit of data, but if we can replicate the problem, we can fix it.
|||
|||Carter
|||
|||
|||
|||Michael Hornung wrote:
|||
|||> On Tue, 27 Feb 2007 at 14:04, Peter Van Epp wrote:
|||> 
|||> |touch .devel
|||> |touch .debug
|||> |./configure
|||> |make clean |make
|||> |
|||> |in the top argus directory it will compile with debug symbols which will
|||> |get interesting data if you type "where" at the gdb prompt.
|||> 
|||> 
|||> # gdb /usr/local/sbin/argus
|||> GNU gdb Red Hat Linux (6.5-15.fc6rh)
|||> ...
|||> (gdb) run 2>run.log
|||> Starting program: /usr/local/sbin/argus 2>run.log
|||> 
|||> Program received signal SIGSEGV, Segmentation fault.
|||> 0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
|||> 1362                ArgusMallocList->end->nxt = mem;
|||> 
|||> 
|||> (gdb) where
|||> #0  0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
|||> #1  0x0805a744 in ArgusWriteOutSocket (output=0x9e1b2e0, client=0x9e1b2f4)
|||>    at ArgusUtil.c:1281
|||> #2  0x0805be78 in ArgusOutputProcess (arg=0x9e1b2e0) at ArgusOutput.c:428
|||> #3  0x0804e983 in ArgusProcessPacket (model=0x9e1a008, p=0x9e1d442 "",
|||> length=1514, tvp=0xbfb8d768, type=-1) at ArgusModeler.c:1055
|||> #4  0x0805545d in ArgusEtherPacket (user=0xb7e59008 "", h=0xbfb8d768,
|||> p=0x9e1d442 "") at ArgusSource.c:608
|||> #5  0x08063e18 in pcap_read_linux ()
|||> #6  0x0805718c in ArgusGetPackets (src=0xb7e59008) at ArgusSource.c:1477
|||> #7  0x0804b2eb in main (argc=1, argv=0xbfb8db34) at argus.c:460
|||> 
|||> 
|||> (gdb) bt full
|||> #0  0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
|||>        mem = (struct ArgusMemoryHeader *) 0xaa96df0
|||>        rec = (struct ArgusRecordStruct *) 0xaa96df0
|||> #1  0x0805a744 in ArgusWriteOutSocket (output=0x9e1b2e0, client=0x9e1b2f4)
|||>    at ArgusUtil.c:1281
|||>        asock = (struct ArgusSocketStruct *) 0xb4ffb50
|||>        list = (struct ArgusListStruct *) 0xa1e47d0
|||>        rec = (struct ArgusRecordStruct *) 0xaa96df0
|||>        retn = 276
|||>        count = 0
|||>        len = 276
|||>        ocnt = 11128989
|||>        statbuf = {st_dev = 836293388809535488, __pad1 = 39296,  __st_ino =
|||> 165781512, st_mode = 165796208, st_nlink = 3216561320,  st_uid = 165781512,
|||> st_gid = 0, st_rdev = 13815025949856902614,  __pad2 = 36845, st_size =
|||> 578914913796227081, st_blksize = 165781512,  st_blocks = 1739248179131534,
|||> st_atim = {tv_sec = 30, tv_nsec = 0},  st_mtim = {tv_sec = 1172616364,
|||> tv_nsec = 165781512}, st_ctim = {
|||>    tv_sec = -1078405756, tv_nsec = -1078405928}, st_ino = 38789285994}
|||>        ptr = (unsigned char *) 0xb4ffb9c "\020 "
|||> #2  0x0805be78 in ArgusOutputProcess (arg=0x9e1b2e0) at ArgusOutput.c:428
|||>        arguswriterecord = 1
|||>        done = 0
|||>        rec = (struct ArgusRecordStruct *) 0xaa97028
|||>        output = (struct ArgusOutputStruct *) 0x9e1b2e0
|||>        ArgusUpDate = {tv_sec = 0, tv_usec = 500000}
|||>        ArgusNextUpdate = {tv_sec = 0, tv_usec = 500000}
|||>        i = 0
|||>        val = 0
|||>        count = 0
|||>        retn = (void *) 0x0
|||> #3  0x0804e983 in ArgusProcessPacket (model=0x9e1a008, p=0x9e1d442 "",
|||> length=1514, tvp=0xbfb8d768, type=-1) at ArgusModeler.c:1055
|||>        retn = 0
|||>        tflow = (struct ArgusSystemFlow *) 0x9e1b290
|||>        flow = (struct ArgusFlowStruct *) 0x9e22b40
|||>        nflow = (struct ArgusFlowStruct *) 0xdaa8c08
|||>        ptr = 0x9e1d468 "\b\002"
|||>        value = 0
|||> #4  0x0805545d in ArgusEtherPacket (user=0xb7e59008 "", h=0xbfb8d768,
|||> p=0x9e1d442 "") at ArgusSource.c:608
|||>        ep = (struct ether_header *) 0x9e1d442
|||>        ind = 0
|||>        src = (struct ArgusSourceStruct *) 0xb7e59008
|||>        tvp = (struct timeval *) 0xbfb8d768
|||>        caplen = 160
|||>        length = 1514
|||>        statbuf = {st_dev = 0, __pad1 = 0, __st_ino = 0, st_mode = 0,
|||> st_nlink = 10354372, st_uid = 3086764936, st_gid = 0,  st_rdev =
|||> 44261669504811007, __pad2 = 18120, st_size = -4631715752896591472,
|||> st_blksize = 10255072, st_blocks = -5189186049726920576, st_atim = {
|||>    tv_sec = 1, tv_nsec = 1}, st_mtim = {tv_sec = 0, tv_nsec = 134516346},
|||> st_ctim = {tv_sec = 0, tv_nsec = 134899988}, st_ino = 10354372}
|||> #5  0x08063e18 in pcap_read_linux ()
|||> No symbol table info available.
|||> #6  0x0805718c in ArgusGetPackets (src=0xb7e59008) at ArgusSource.c:1477
|||>        ArgusReadMask = {__fds_bits = {128, 0 <repeats 31 times>}}
|||>        ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
|||>        ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
|||>        tmp = 1
|||>        i = 0
|||>        width = 7
|||>        noerror = 1
|||>        fd = 7
|||>        found = 1
|||>        up = 1
|||>        wait = {tv_sec = 0, tv_usec = 20000}
|||> #7  0x0804b2eb in main (argc=1, argv=0xbfb8db34) at argus.c:460
|||>        commandlinew = 0
|||>        doconf = 0
|||>        dodebug = 0
|||>        i = 1
|||>        pid = 0
|||>        tmparg = 0x8049f30 "[\201ï¿½ï¿½005"
|||>        filter = 0x0
|||>        statbuf = {st_dev = 64768, __pad1 = 0, __st_ino = 2688737,  st_mode =
|||> 33133, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0,  __pad2 = 0,
|||> st_size = 11071, st_blksize = 4096, st_blocks = 32, st_atim = {
|||>    tv_sec = 1172616251, tv_nsec = 0}, st_mtim = {tv_sec = 1172616251,
|||> tv_nsec = 0}, st_ctim = {tv_sec = 1172616251, tv_nsec = 0},  st_ino =
|||> 2688737}
|||>        host = (struct hostent *) 0x80a6720
|||>        commandlinei = 0
|||>        op = -1
|||>        path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
|||> 
|||> 
|||> _____________________________________________________
|||> Michael Hornung          Computing & Communications hornung at washington.edu
|||> University of Washington
|||> 
|||
|||