argus-3.0.0 segfault (5/23 version)

carter at qosient.com carter at qosient.com
Sat Jun 2 14:54:49 EDT 2007


Hey Michael,
This is pretty common, for "sanitized" data to behave differently.  There are a lot of ways of approaching this issue.  If you want to go over the problem over the phone, that can work, or if you would like me to log onto one of your machines, I can debug there.  Or if there is a third party you trust, like the CERT, or some Internet2 partner, like CMU that would work, where you send them the data and we debug it there.

I'm pretty confident that we can do this over email, if you want to drive gdb()!!

Carter



Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

-----Original Message-----
From: Michael Hornung <hornung at cac.washington.edu>

Date: Fri, 1 Jun 2007 14:33:52 
To:carter at qosient.com
Cc:Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] argus-3.0.0 segfault (5/23 version)


Hi Carter, no it doesn't.  Sort of.  Let me explain.

I haven't sent a pcap yet because my organization wants me to sanitize it 
before sending it along.  That is not something I've done before, but I 
found an API and a tool called "anontool" 
(http://www.ics.forth.gr/dcs/Activities/Projects/anontool.html) which 
seems to work.

Now here's the rub: when I send the original pcap I captured when the 
segfault was caught, back through argus using the "-r" option, it 
segfaults at a different point than when I was capturing off a NIC.  
That's ok, because it still segfaults and that should be passed along for 
debugging.  BUT when I pass the anonymized pcap through argus using "-r" 
it completes and does not throw an exception.  See below (I set the debug 
reporting to 1):

# gdb /usr/local/sbin/argus
(gdb) set args -r /tmp/segfault.pcap
(gdb) run
Starting program: /usr/local/sbin/argus -r /tmp/segfault.pcap
argus[17283]: 01 Jun 07 14:22:55.957899 ArgusParseResourceFile: 
ArgusFilter "" 
argus[17283]: 01 Jun 07 14:22:55.958569 ArgusParseResourceFile 
(/etc/argus.conf) returning
argus[17283]: 01 Jun 07 14:22:55.958965 setArgusInterfaceStatus(1)
argus[17283]: 01 Jun 07 14:22:55.981237 ArgusInitSource() returning
argus[17283]: 01 Jun 07 14:22:55.981711 ArgusInitOutput() done
argus[17283]: 01 Jun 07 14:22:55.982090 ArgusInitModeler() done
argus[17283]: 01 Jun 07 14:22:55.982560 setArgusInterfaceStatus(0)

Program received signal SIGSEGV, Segmentation fault.
0x080780f5 in ArgusFreeListRecord (buf=0x938cc38) at argus_util.c:1410
1410                ArgusMallocList->end->nxt = mem;

(gdb) bt full
#0  0x080780f5 in ArgusFreeListRecord (buf=0x938cc38) at argus_util.c:1410
        mem = (struct ArgusMemoryHeader *) 0x938cc38
        rec = (struct ArgusRecordStruct *) 0x938cc38
#1  0x0805e0c0 in ArgusOutputProcess (arg=0x85c22e0) at ArgusOutput.c:507
        done = 0
        rec = (struct ArgusRecordStruct *) 0x938cc38
        output = (struct ArgusOutputStruct *) 0x85c22e0
        ArgusUpDate = {tv_sec = 0, tv_usec = 500000}
        ArgusNextUpdate = {tv_sec = 0, tv_usec = 0}
        i = 0
        val = 1
        count = 0
        retn = (void *) 0x0
#2  0x0804e9f3 in ArgusProcessPacket (model=0x85c1008, p=0x85c446a "", 
    length=60, tvp=0xbfe684fc, type=0) at ArgusModeler.c:1039
        retn = 0
        tflow = (struct ArgusSystemFlow *) 0x85c2290
        flow = (struct ArgusFlowStruct *) 0xb567948
        nflow = (struct ArgusFlowStruct *) 0xb4e4548
        ptr = 0x85c4478 "E"
        value = 0
#3  0x08055b61 in ArgusEtherPacket (user=0xb7e1c008 "", h=0xbfe684fc, 
    p=0x85c446a "") at ArgusSource.c:623
        ep = (struct ether_header *) 0x85c446a
        ind = 0
        src = (struct ArgusSourceStruct *) 0xb7e1c008
        tvp = (struct timeval *) 0xbfe684fc
        caplen = 60
        length = 60
        statbuf = {st_dev = 602437195686234480,__pad1 = 33864, 
__st_ino = 10722685, st_mode = 140265840, st_nlink = 892680754, 
  st_uid = 11599860, st_gid = 9, st_rdev = 45995984813458544,__pad2 = 
17514, 
  st_size = 260783057480, st_blksize = 60, st_blocks = 49821019478443114, 
  st_atim = {tv_sec = 60, tv_nsec = 140265840}, st_mtim = {
    tv_sec = -1075411836, tv_nsec = 10717480}, st_ctim = {tv_sec = 
140265840, 
    tv_nsec = 140264554}, st_ino = 13827885562977058876}
#4  0x0807123f in pcap_offline_read ()
No symbol table info available.
#5  0x08058303 in ArgusGetPackets (src=0xb7e1c008) at ArgusSource.c:1747
        ArgusReadMask = {__fds_bits = {0 <repeats 32 times>}}
        ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
        ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
        tmp = 6
        i = 0
        width = 0
        noerror = 1
        fd = 1
        found = 1
        up = 0
        wait = {tv_sec = 0, tv_usec = 20000}
#6  0x0804b333 in main (argc=3, argv=0xbfe688c4) at argus.c:464
        commandlinew = 0
        doconf = 0
        dodebug = 0
        i = 3
        pid = 0
        tmparg = 0x8049f30 "[\201�214�005"
        filter = 0x0
        statbuf = {st_dev = 64768,__pad1 = 0,__st_ino = 2688506, 
  st_mode = 33184, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0, 
__pad2 = 0, st_size = 11098, st_blksize = 4096, st_blocks = 32, st_atim 
= {
    tv_sec = 1180732964, tv_nsec = 0}, st_mtim = {tv_sec = 1180732964, 
    tv_nsec = 0}, st_ctim = {tv_sec = 1180732964, tv_nsec = 0}, 
  st_ino = 2688506}
        host = (struct hostent *) 0x80a94bc
        commandlinei = 0
        op = -1
        path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
(gdb) print ArgusMallocList->end->nxt
Cannot access memory at address 0xaa4ed9e8
(gdb) print mem
$1 = (struct ArgusMemoryHeader *) 0x94edc38


Now when I run the anonymized version of the pcap file through argus 
(generated via 'anonymize_tool -f /tmp/segfault.pcap -a MAP -c 
/tmp/segfault-anon.pcap') I get:

# gdb /usr/local/sbin/argus
(gdb) set args -r /tmp/segfault-anon.pcap
(gdb) run
Starting program: /usr/local/sbin/argus -r /tmp/segfault-anon.pcap
argus[17296]: 01 Jun 07 14:27:25.373820 ArgusParseResourceFile: 
ArgusFilter "" 
argus[17296]: 01 Jun 07 14:27:25.374457 ArgusParseResourceFile 
(/etc/argus.conf) returning
argus[17296]: 01 Jun 07 14:27:25.374845 setArgusInterfaceStatus(1)
argus[17296]: 01 Jun 07 14:27:25.403556 ArgusInitSource() returning
argus[17296]: 01 Jun 07 14:27:25.404048 ArgusInitOutput() done
argus[17296]: 01 Jun 07 14:27:25.404422 ArgusInitModeler() done
argus[17296]: 01 Jun 07 14:27:25.404897 setArgusInterfaceStatus(0)
argus[17296]: 01 Jun 07 14:28:03.225693 main() ArgusGetPackets returned: 
shuting down

argus[17296]: 01 Jun 07 14:28:03.226416 ArgusShutDown(Normal Shutdown)

argus[17296]: 01 Jun 07 14:28:03.226690 ArgusCloseSource(0xb7db1008) 
starting
argus[17296]: 01 Jun 07 14:28:03.226863 ArgusCloseSource(0xb7db1008) 
deleting source
argus[17296]: 01 Jun 07 14:28:03.399853 ArgusCloseModeler(0x97db008)
argus[17296]: 01 Jun 07 14:28:03.400353 ArgusCloseOutput() scheduling 
closure after writing records
argus[17296]: 01 Jun 07 14:28:03.400644 ArgusCloseOutput(0x97dc2e0) done
argus[17296]: 01 Jun 07 14:28:03.414111 ArgusShutDown()

Program exited normally.


So, I'm not sure how I can help at this point since the sanitized file is 
ready but does not reproduce the exception, and I can not send along the 
pcap which does produce an exception.

-Mike

On Fri, 1 Jun 2007 at 21:16, carter at qosient.com wrote:

|Hey Michael,
|I'm expecting to read the resulting packet file back through argus using the "-r file" option, and it segfaulting.
|Does it do that?
|Carter
|
|Carter Bullard
|QoSient LLC
|150 E. 57th Street Suite 12D
|New York, New York 10022
|+1 212 588-9133 Phone
|+1 212 588-9134 Fax  
|
|-----Original Message-----
|From: Michael Hornung <hornung at cac.washington.edu>
|Date: Thu, 31 May 2007 14:59:27 
|To:carter at qosient.com
|Cc:Argus <argus-info at lists.andrew.cmu.edu>
|Subject: Re: [ARGUS] argus-3.0.0 segfault (5/23 version)
|
|Are you expecting argus to reproduce the segfault by replaying it through 
|argus by setting ARGUS_PACKET_CAPTURE_FILE in argus.conf?  I tried that 
|(leaving the rest of the config the same) and the segfault does not happen 
|again.  If the pcap will still be helpful to you, let me know and I'll put 
|up the copy wiht sanitized IPs.
|
|-Mike
|
|On Thu, 24 May 2007 at 23:22, carter at qosient.com wrote:
|
||Hey Micheal,
||If you can share that would be ideal!!!
||You can use the argus write pcap file function that is turned on from the argus.conf file to try to get the packet file size down.  It causes argus to write out the paxkets it receives, and so it will stop on the packet that causes the problem!!!
||
||Thanks!!!
||
||Carter
||
||
||Carter Bullard
||QoSient LLC
||150 E. 57th Street Suite 12D
||New York, New York 10022
||+1 212 588-9133 Phone
||+1 212 588-9134 Fax  
||
||-----Original Message-----
||From: Michael Hornung <hornung at cac.washington.edu>
||Date: Thu, 24 May 2007 15:13:15 
||To:argus-info at lists.andrew.cmu.edu
||Subject: [ARGUS] argus-3.0.0 segfault (5/23 version)
||
||I've got argus running on a Fedora Core 6 x86 Linux box.  The argus daemon 
||dies *very* regularly and so needs to be monitored.  I finally got around 
||to capturing a pcap for the duration of an argus session.  Carter, let me 
||know if you want this and I'll get it to you; it is 650MB uncompressed. 
||Following is what I see when running argus in gdb:
||
||(gdb) run
||Starting program: /usr/local/sbin/argus
||argus[29762]: 24 May 07 14:56:36.593821 started
||argus[29762]: 24 May 07 14:56:36.596492 ArgusGetInterfaceStatus: interface 
||eth1 is up
||argus[29762]: 24 May 07 14:56:41.031467 connect from 128.95.135.24
||
||Program received signal SIGSEGV, Segmentation fault.
||0x0805a340 in ArgusCreateFlowKey (model=0x9491008, flow=0x9492290,
||     hstruct=0x9492200) at ArgusUtil.c:785
||785           hstruct->hash ^= *ptr++;
||
||(gdb) bt full
||#0  0x0805a340 in ArgusCreateFlowKey (model=0x9491008, flow=0x9492290,
||     hstruct=0x9492200) at ArgusUtil.c:785
||         ptr = (unsigned int *) 0xe025000
||         key = (unsigned int *) 0x9492208
||         retn = 0
||         i = 19811198
||         len = -1
||#1  0x0804e71f in ArgusProcessPacket (model=0x9491008, p=0x949460a "",
||     length=90, tvp=0xbff0d5b8, type=0) at ArgusModeler.c:988
||         retn = 0
||         tflow = (struct ArgusSystemFlow *) 0x9492290
||         flow = (struct ArgusFlowStruct *) 0x94b9d78
||         nflow = (struct ArgusFlowStruct *) 0xc6ecbc8
||         ptr = 0x949473c "\031"
||         value = 0
||#2  0x08055b61 in ArgusEtherPacket (user=0xb7e4c008 "", h=0xbff0d5b8,
||     p=0x949460a "") at ArgusSource.c:623
||         ep = (struct ether_header *) 0x949460a
||         ind = 0
||         src = (struct ArgusSourceStruct *) 0xb7e4c008
||         tvp = (struct timeval *) 0xbff0d5b8
||         caplen = 90
||         length = 90
||         statbuf = {st_dev = 0,__pad1 = 0,__st_ino = 0, st_mode = 0,
||   st_nlink = 10354372, st_uid = 3086711688, st_gid = 0,
||   st_rdev = 44261669504811007,__pad2 = 18120, st_size = 
||-4615955009626666608,
||   st_blksize = 10255072, st_blocks = -5189414748145497984, st_atim = {
||     tv_sec = 1, tv_nsec = 1}, st_mtim = {tv_sec = 0, tv_nsec = 134516346},
||   st_ctim = {tv_sec = 0, tv_nsec = 134911664}, st_ino = 10354372}
||#3  0x08066088 in pcap_read_linux ()
||No symbol table info available.
||#4  0x08057eeb in ArgusGetPackets (src=0xb7e4c008) at ArgusSource.c:1654
||         ArgusReadMask = {__fds_bits = {128, 0 <repeats 31 times>}}
||         ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
||         ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
||         tmp = 1
||         i = 0
||         width = 7
||         noerror = 1
||         fd = 7
||         found = 1
||         up = 1
||         wait = {tv_sec = 0, tv_usec = 20000}
||#5  0x0804b333 in main (argc=1, argv=0xbff0d984) at argus.c:464
||         commandlinew = 0
||         doconf = 0
||         dodebug = 0
||         i = 1
||         pid = 0
||         tmparg = 0x8049f30 "[\201�214�005"
||         filter = 0x0
||         statbuf = {st_dev = 64768,__pad1 = 0,__st_ino = 2688645,
||   st_mode = 33133, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0,
||__pad2 = 0, st_size = 11114, st_blksize = 4096, st_blocks = 32, st_atim 
||= {
||     tv_sec = 1180043663, tv_nsec = 0}, st_mtim = {tv_sec = 1180042005,
||     tv_nsec = 0}, st_ctim = {tv_sec = 1180042005, tv_nsec = 0},
||   st_ino = 2688645}
||         host = (struct hostent *) 0x80a94bc
||         commandlinei = 0
||         op = -1
||         path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
||
||(gdb) print hstruct->hash
||$1 = 2710700798
||
||So again, let me know if the pcap or anything else will be helpful.
||
||-Mike
||
|


More information about the argus mailing list