RC.39 (and 40) 'argus' segfault on Fedora Core 6

Wed Feb 28 15:30:58 EST 2007

Hey Michael,
Yes you are running the program correctly.  I did add ArgusOpenInterface()
just for rc.40.  You shouldn't be going into that code at all, really, 
if you're
reading a file, so let me see what's up with that.

Carter

Michael Hornung wrote:

>It appears I am cursed here.  =)
>I tried running a 'tcpdump' capture through argus to try reproducing the 
>segfaults I have been experiencing.  I got another segfault, but a 
>different one.  Is this the right way to run a tcpdump pcap file through 
>argus instead of reading through an interface?
>
>	argus -X -r /home/argus/cap.dump
>
>The tcpdump file was generated via:
>	tcpdump -li eth1 -s128 -w cap.dump ''
>
>
># ls -l /home/argus/cap.dump
>-rw-r--r-- 1 root root 4207648087 Feb 28 10:39 /home/argus/cap.dump
>
># argus -X -r /home/argus/cap.dump -w /dev/null
>argus[29483]: 28 Feb 07 10:59:19.057577 started
>Segmentation fault
>
>
>gdb shows:
>
>(gdb) run -X -r /home/argus/cap.dump -w /dev/null
>Starting program: /usr/local/sbin/argus -X -r /home/argus/cap.dump -w /dev/null
>argus[29540]: 28 Feb 07 11:05:21.421096 started
>
>Program received signal SIGSEGV, Segmentation fault.
>0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
>    at ArgusSource.c:99
>99         if ((inf->ArgusPd = pcap_open_live(device->name, 
>src->ArgusSnapLen, !src->Arguspflag, 100, errbuf)) != NULL) {
>
>
>(gdb) where
>#0  0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
>    at ArgusSource.c:99
>#1  0x080574bc in ArgusGetPackets (src=0xb7da4008) at ArgusSource.c:1547
>#2  0x0804b2eb in main (argc=6, argv=0xbfe608a4) at argus.c:460
>
>
>(gdb) bt full
>#0  0x080543f7 in ArgusOpenInterface (src=0xb7da4008, inf=0xb7da4068)
>    at ArgusSource.c:99
>        device = (struct ArgusDeviceStruct *) 0x0
>        errbuf = 
>"ï¿½002ï¿½\002ï¿½\002ï¿½b\aï¿½_\a\b\004\000\000\000ï¿½002ï¿½\004ï¿½F\235\000ï¿½216\004\bï¿½002ï¿½\230\b\bï¿½002ï¿½\214(!\000\000\000\000ï¿½\006\00028 
>Feb 07 11:05:21.421096 
>started\n\000\b\001\000\000\000ï¿½\235\000\000\000\000\020\214\005ï¿½001\000\000\000\000\020", 
>'\0' <repeats 38 times>, 
>"$\235\021\000\210ï¿½ï¿½\000\000\000\000ï¿½ï¿½ï¿½ï¿½ï¿½\235\000ï¿½\235\000ï¿½\204\004\bï¿½004ï¿½z\234\000\200H\235\000ï¿½ï¿½001\000\000\000\001\000\000\000\000\000\000\000ï¿½216\004\b\000\000\000\000ï¿½\n\b"...
>#1  0x080574bc in ArgusGetPackets (src=0xb7da4008) at ArgusSource.c:1547
>        ArgusReadMask = {__fds_bits = {0 <repeats 32 times>}}
>        ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
>        ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
>        tmp = 6
>        i = 0
>        width = 0
>        noerror = 1
>        fd = 0
>        found = 0
>        up = 0
>        wait = {tv_sec = 0, tv_usec = 20000}
>#2  0x0804b2eb in main (argc=6, argv=0xbfe608a4) at argus.c:460
>        commandlinew = 1
>        doconf = 1
>        dodebug = 0
>        i = 6
>        pid = 0
>        tmparg = 0xbfe60bb0 "/dev/null"
>        filter = 0x0
>        statbuf = {st_dev = 0, __pad1 = 0, __st_ino = 0, st_mode = 3219523608, 
>  st_nlink = 0, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, 
>  st_size = -4618995228131459072, st_blksize = 10307272, 
>  st_blocks = 134516652, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {
>    tv_sec = -163754450, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 
>0}, 
>  st_ino = 0}
>        host = (struct hostent *) 0x80a6720
>        commandlinei = 0
>        op = -1
>        path = "/usr/local/sbin/argus", '\0' <repeats 8170 times>
>
>
>(gdb) print device->name
>Cannot access memory at address 0x4
>(gdb) print src->ArgusSnapLen
>$3 = 96
>(gdb) print src->Arguspflag
>$4 = 0
>
>-Mike
>
>On Wed, 28 Feb 2007 at 10:46, Michael Hornung wrote:
>
>|I got a different segfault today:
>|
>|Program received signal SIGSEGV, Segmentation fault.
>|0x0805940f in ArgusCreateFlowKey (model=0x87af008, flow=0x87b0290, 
>|    hstruct=0x87b0200) at ArgusUtil.c:704
>|704           hstruct->hash ^= *ptr++;
>|
>|
>|(gdb) where
>|#0  0x0805940f in ArgusCreateFlowKey (model=0x87af008, flow=0x87b0290, 
>|    hstruct=0x87b0200) at ArgusUtil.c:704
>|#1  0x0804e6af in ArgusProcessPacket (model=0x87af008, p=0x87b260a "", 
>|    length=90, tvp=0xbfc03298, type=0) at ArgusModeler.c:1004
>|#2  0x0805545d in ArgusEtherPacket (user=0xb7e2b008 "", h=0xbfc03298, 
>|    p=0x87b260a "") at ArgusSource.c:608
>|#3  0x08063e18 in pcap_read_linux ()
>|#4  0x0805718c in ArgusGetPackets (src=0xb7e2b008) at ArgusSource.c:1477
>|#5  0x0804b2eb in main (argc=1, argv=0xbfc03664) at argus.c:460
>|
>|
>|(gdb) print hstruct->hash
>|$1 = 4251857491
>|(gdb) print ptr
>|$2 = (unsigned int *) 0xc023000
>|(gdb) print *ptr
>|Cannot access memory at address 0xc023000
>|
>|
>|Could this be a result of bad physical memory in this box?
>|
>|I captured traffic with tcpdump while running argus and will see if I can 
>|reproduce the crash that way.
>|
>|-Mike
>|
>|On Wed, 28 Feb 2007 at 08:04, Michael Hornung wrote:
>|
>||(gdb) print ArgusMallocList
>||$1 = (struct ArgusMemoryList *) 0x9e1d538
>||(gdb) print ArgusMallocList->end
>||$2 = (struct ArgusMemoryHeader *) 0x54a96bb8
>||(gdb) print ArgusMallocList->end->nxt
>||Cannot access memory at address 0x54a96bb8
>||
>||-Mike
>||
>||On Tue, 27 Feb 2007 at 23:30, Carter Bullard wrote:
>||
>|||This is very odd.  When you get to someplace in gdb, with a Segmentation
>|||fault, try to find out what variable is having problems by printing the actual
>|||values:
>|||
>|||  (gdb) print ArgusMallocList
>|||  (gdb) print ArgusMallocList->end
>|||  (gdb) print ArgusMallocList->end->nxt
>|||
>|||I would guess that ArgusMallocList doesn't exist, or is corrupted.
>|||This can happen for a number of reasons, but it may be useful to
>|||try to get a packet trace that generates your errors.  Maybe a
>|||bit of data, but if we can replicate the problem, we can fix it.
>|||
>|||Carter
>|||
>|||
>|||
>|||Michael Hornung wrote:
>|||
>|||> On Tue, 27 Feb 2007 at 14:04, Peter Van Epp wrote:
>|||> 
>|||> |touch .devel
>|||> |touch .debug
>|||> |./configure
>|||> |make clean |make
>|||> |
>|||> |in the top argus directory it will compile with debug symbols which will
>|||> |get interesting data if you type "where" at the gdb prompt.
>|||> 
>|||> 
>|||> # gdb /usr/local/sbin/argus
>|||> GNU gdb Red Hat Linux (6.5-15.fc6rh)
>|||> ...
>|||> (gdb) run 2>run.log
>|||> Starting program: /usr/local/sbin/argus 2>run.log
>|||> 
>|||> Program received signal SIGSEGV, Segmentation fault.
>|||> 0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
>|||> 1362                ArgusMallocList->end->nxt = mem;
>|||> 
>|||> 
>|||> (gdb) where
>|||> #0  0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
>|||> #1  0x0805a744 in ArgusWriteOutSocket (output=0x9e1b2e0, client=0x9e1b2f4)
>|||>    at ArgusUtil.c:1281
>|||> #2  0x0805be78 in ArgusOutputProcess (arg=0x9e1b2e0) at ArgusOutput.c:428
>|||> #3  0x0804e983 in ArgusProcessPacket (model=0x9e1a008, p=0x9e1d442 "",
>|||> length=1514, tvp=0xbfb8d768, type=-1) at ArgusModeler.c:1055
>|||> #4  0x0805545d in ArgusEtherPacket (user=0xb7e59008 "", h=0xbfb8d768,
>|||> p=0x9e1d442 "") at ArgusSource.c:608
>|||> #5  0x08063e18 in pcap_read_linux ()
>|||> #6  0x0805718c in ArgusGetPackets (src=0xb7e59008) at ArgusSource.c:1477
>|||> #7  0x0804b2eb in main (argc=1, argv=0xbfb8db34) at argus.c:460
>|||> 
>|||> 
>|||> (gdb) bt full
>|||> #0  0x08075df8 in ArgusFreeListRecord (buf=0xaa96df0) at argus_util.c:1362
>|||>        mem = (struct ArgusMemoryHeader *) 0xaa96df0
>|||>        rec = (struct ArgusRecordStruct *) 0xaa96df0
>|||> #1  0x0805a744 in ArgusWriteOutSocket (output=0x9e1b2e0, client=0x9e1b2f4)
>|||>    at ArgusUtil.c:1281
>|||>        asock = (struct ArgusSocketStruct *) 0xb4ffb50
>|||>        list = (struct ArgusListStruct *) 0xa1e47d0
>|||>        rec = (struct ArgusRecordStruct *) 0xaa96df0
>|||>        retn = 276
>|||>        count = 0
>|||>        len = 276
>|||>        ocnt = 11128989
>|||>        statbuf = {st_dev = 836293388809535488, __pad1 = 39296,  __st_ino =
>|||> 165781512, st_mode = 165796208, st_nlink = 3216561320,  st_uid = 165781512,
>|||> st_gid = 0, st_rdev = 13815025949856902614,  __pad2 = 36845, st_size =
>|||> 578914913796227081, st_blksize = 165781512,  st_blocks = 1739248179131534,
>|||> st_atim = {tv_sec = 30, tv_nsec = 0},  st_mtim = {tv_sec = 1172616364,
>|||> tv_nsec = 165781512}, st_ctim = {
>|||>    tv_sec = -1078405756, tv_nsec = -1078405928}, st_ino = 38789285994}
>|||>        ptr = (unsigned char *) 0xb4ffb9c "\020 "
>|||> #2  0x0805be78 in ArgusOutputProcess (arg=0x9e1b2e0) at ArgusOutput.c:428
>|||>        arguswriterecord = 1
>|||>        done = 0
>|||>        rec = (struct ArgusRecordStruct *) 0xaa97028
>|||>        output = (struct ArgusOutputStruct *) 0x9e1b2e0
>|||>        ArgusUpDate = {tv_sec = 0, tv_usec = 500000}
>|||>        ArgusNextUpdate = {tv_sec = 0, tv_usec = 500000}
>|||>        i = 0
>|||>        val = 0
>|||>        count = 0
>|||>        retn = (void *) 0x0
>|||> #3  0x0804e983 in ArgusProcessPacket (model=0x9e1a008, p=0x9e1d442 "",
>|||> length=1514, tvp=0xbfb8d768, type=-1) at ArgusModeler.c:1055
>|||>        retn = 0
>|||>        tflow = (struct ArgusSystemFlow *) 0x9e1b290
>|||>        flow = (struct ArgusFlowStruct *) 0x9e22b40
>|||>        nflow = (struct ArgusFlowStruct *) 0xdaa8c08
>|||>        ptr = 0x9e1d468 "\b\002"
>|||>        value = 0
>|||> #4  0x0805545d in ArgusEtherPacket (user=0xb7e59008 "", h=0xbfb8d768,
>|||> p=0x9e1d442 "") at ArgusSource.c:608
>|||>        ep = (struct ether_header *) 0x9e1d442
>|||>        ind = 0
>|||>        src = (struct ArgusSourceStruct *) 0xb7e59008
>|||>        tvp = (struct timeval *) 0xbfb8d768
>|||>        caplen = 160
>|||>        length = 1514
>|||>        statbuf = {st_dev = 0, __pad1 = 0, __st_ino = 0, st_mode = 0,
>|||> st_nlink = 10354372, st_uid = 3086764936, st_gid = 0,  st_rdev =
>|||> 44261669504811007, __pad2 = 18120, st_size = -4631715752896591472,
>|||> st_blksize = 10255072, st_blocks = -5189186049726920576, st_atim = {
>|||>    tv_sec = 1, tv_nsec = 1}, st_mtim = {tv_sec = 0, tv_nsec = 134516346},
>|||> st_ctim = {tv_sec = 0, tv_nsec = 134899988}, st_ino = 10354372}
>|||> #5  0x08063e18 in pcap_read_linux ()
>|||> No symbol table info available.
>|||> #6  0x0805718c in ArgusGetPackets (src=0xb7e59008) at ArgusSource.c:1477
>|||>        ArgusReadMask = {__fds_bits = {128, 0 <repeats 31 times>}}
>|||>        ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
>|||>        ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
>|||>        tmp = 1
>|||>        i = 0
>|||>        width = 7
>|||>        noerror = 1
>|||>        fd = 7
>|||>        found = 1
>|||>        up = 1
>|||>        wait = {tv_sec = 0, tv_usec = 20000}
>|||> #7  0x0804b2eb in main (argc=1, argv=0xbfb8db34) at argus.c:460
>|||>        commandlinew = 0
>|||>        doconf = 0
>|||>        dodebug = 0
>|||>        i = 1
>|||>        pid = 0
>|||>        tmparg = 0x8049f30 "[\201ï¿½ï¿½005"
>|||>        filter = 0x0
>|||>        statbuf = {st_dev = 64768, __pad1 = 0, __st_ino = 2688737,  st_mode =
>|||> 33133, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0,  __pad2 = 0,
>|||> st_size = 11071, st_blksize = 4096, st_blocks = 32, st_atim = {
>|||>    tv_sec = 1172616251, tv_nsec = 0}, st_mtim = {tv_sec = 1172616251,
>|||> tv_nsec = 0}, st_ctim = {tv_sec = 1172616251, tv_nsec = 0},  st_ino =
>|||> 2688737}
>|||>        host = (struct hostent *) 0x80a6720
>|||>        commandlinei = 0
>|||>        op = -1
>|||>        path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
>|||> 
>|||> 
>|||> _____________________________________________________
>|||> Michael Hornung          Computing & Communications hornung at washington.edu
>|||> University of Washington
>|||> 
>|||
>|||
>