argus-3.0.0 segfault (5/23 version)
carter at qosient.com
carter at qosient.com
Sat Jun 2 14:54:49 EDT 2007
Hey Michael,
This is pretty common, for "sanitized" data to behave differently. There are a lot of ways of approaching this issue. If you want to go over the problem over the phone, that can work, or if you would like me to log onto one of your machines, I can debug there. Or if there is a third party you trust, like the CERT, or some Internet2 partner, like CMU that would work, where you send them the data and we debug it there.
I'm pretty confident that we can do this over email, if you want to drive gdb()!!
Carter
Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-----Original Message-----
From: Michael Hornung <hornung at cac.washington.edu>
Date: Fri, 1 Jun 2007 14:33:52
To:carter at qosient.com
Cc:Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] argus-3.0.0 segfault (5/23 version)
Hi Carter, no it doesn't. Sort of. Let me explain.
I haven't sent a pcap yet because my organization wants me to sanitize it
before sending it along. That is not something I've done before, but I
found an API and a tool called "anontool"
(http://www.ics.forth.gr/dcs/Activities/Projects/anontool.html) which
seems to work.
Now here's the rub: when I send the original pcap I captured when the
segfault was caught, back through argus using the "-r" option, it
segfaults at a different point than when I was capturing off a NIC.
That's ok, because it still segfaults and that should be passed along for
debugging. BUT when I pass the anonymized pcap through argus using "-r"
it completes and does not throw an exception. See below (I set the debug
reporting to 1):
# gdb /usr/local/sbin/argus
(gdb) set args -r /tmp/segfault.pcap
(gdb) run
Starting program: /usr/local/sbin/argus -r /tmp/segfault.pcap
argus[17283]: 01 Jun 07 14:22:55.957899 ArgusParseResourceFile:
ArgusFilter ""
argus[17283]: 01 Jun 07 14:22:55.958569 ArgusParseResourceFile
(/etc/argus.conf) returning
argus[17283]: 01 Jun 07 14:22:55.958965 setArgusInterfaceStatus(1)
argus[17283]: 01 Jun 07 14:22:55.981237 ArgusInitSource() returning
argus[17283]: 01 Jun 07 14:22:55.981711 ArgusInitOutput() done
argus[17283]: 01 Jun 07 14:22:55.982090 ArgusInitModeler() done
argus[17283]: 01 Jun 07 14:22:55.982560 setArgusInterfaceStatus(0)
Program received signal SIGSEGV, Segmentation fault.
0x080780f5 in ArgusFreeListRecord (buf=0x938cc38) at argus_util.c:1410
1410 ArgusMallocList->end->nxt = mem;
(gdb) bt full
#0 0x080780f5 in ArgusFreeListRecord (buf=0x938cc38) at argus_util.c:1410
mem = (struct ArgusMemoryHeader *) 0x938cc38
rec = (struct ArgusRecordStruct *) 0x938cc38
#1 0x0805e0c0 in ArgusOutputProcess (arg=0x85c22e0) at ArgusOutput.c:507
done = 0
rec = (struct ArgusRecordStruct *) 0x938cc38
output = (struct ArgusOutputStruct *) 0x85c22e0
ArgusUpDate = {tv_sec = 0, tv_usec = 500000}
ArgusNextUpdate = {tv_sec = 0, tv_usec = 0}
i = 0
val = 1
count = 0
retn = (void *) 0x0
#2 0x0804e9f3 in ArgusProcessPacket (model=0x85c1008, p=0x85c446a "",
length=60, tvp=0xbfe684fc, type=0) at ArgusModeler.c:1039
retn = 0
tflow = (struct ArgusSystemFlow *) 0x85c2290
flow = (struct ArgusFlowStruct *) 0xb567948
nflow = (struct ArgusFlowStruct *) 0xb4e4548
ptr = 0x85c4478 "E"
value = 0
#3 0x08055b61 in ArgusEtherPacket (user=0xb7e1c008 "", h=0xbfe684fc,
p=0x85c446a "") at ArgusSource.c:623
ep = (struct ether_header *) 0x85c446a
ind = 0
src = (struct ArgusSourceStruct *) 0xb7e1c008
tvp = (struct timeval *) 0xbfe684fc
caplen = 60
length = 60
statbuf = {st_dev = 602437195686234480,__pad1 = 33864,
__st_ino = 10722685, st_mode = 140265840, st_nlink = 892680754,
st_uid = 11599860, st_gid = 9, st_rdev = 45995984813458544,__pad2 =
17514,
st_size = 260783057480, st_blksize = 60, st_blocks = 49821019478443114,
st_atim = {tv_sec = 60, tv_nsec = 140265840}, st_mtim = {
tv_sec = -1075411836, tv_nsec = 10717480}, st_ctim = {tv_sec =
140265840,
tv_nsec = 140264554}, st_ino = 13827885562977058876}
#4 0x0807123f in pcap_offline_read ()
No symbol table info available.
#5 0x08058303 in ArgusGetPackets (src=0xb7e1c008) at ArgusSource.c:1747
ArgusReadMask = {__fds_bits = {0 <repeats 32 times>}}
ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
tmp = 6
i = 0
width = 0
noerror = 1
fd = 1
found = 1
up = 0
wait = {tv_sec = 0, tv_usec = 20000}
#6 0x0804b333 in main (argc=3, argv=0xbfe688c4) at argus.c:464
commandlinew = 0
doconf = 0
dodebug = 0
i = 3
pid = 0
tmparg = 0x8049f30 "[\201�214�005"
filter = 0x0
statbuf = {st_dev = 64768,__pad1 = 0,__st_ino = 2688506,
st_mode = 33184, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0,
__pad2 = 0, st_size = 11098, st_blksize = 4096, st_blocks = 32, st_atim
= {
tv_sec = 1180732964, tv_nsec = 0}, st_mtim = {tv_sec = 1180732964,
tv_nsec = 0}, st_ctim = {tv_sec = 1180732964, tv_nsec = 0},
st_ino = 2688506}
host = (struct hostent *) 0x80a94bc
commandlinei = 0
op = -1
path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
(gdb) print ArgusMallocList->end->nxt
Cannot access memory at address 0xaa4ed9e8
(gdb) print mem
$1 = (struct ArgusMemoryHeader *) 0x94edc38
Now when I run the anonymized version of the pcap file through argus
(generated via 'anonymize_tool -f /tmp/segfault.pcap -a MAP -c
/tmp/segfault-anon.pcap') I get:
# gdb /usr/local/sbin/argus
(gdb) set args -r /tmp/segfault-anon.pcap
(gdb) run
Starting program: /usr/local/sbin/argus -r /tmp/segfault-anon.pcap
argus[17296]: 01 Jun 07 14:27:25.373820 ArgusParseResourceFile:
ArgusFilter ""
argus[17296]: 01 Jun 07 14:27:25.374457 ArgusParseResourceFile
(/etc/argus.conf) returning
argus[17296]: 01 Jun 07 14:27:25.374845 setArgusInterfaceStatus(1)
argus[17296]: 01 Jun 07 14:27:25.403556 ArgusInitSource() returning
argus[17296]: 01 Jun 07 14:27:25.404048 ArgusInitOutput() done
argus[17296]: 01 Jun 07 14:27:25.404422 ArgusInitModeler() done
argus[17296]: 01 Jun 07 14:27:25.404897 setArgusInterfaceStatus(0)
argus[17296]: 01 Jun 07 14:28:03.225693 main() ArgusGetPackets returned:
shuting down
argus[17296]: 01 Jun 07 14:28:03.226416 ArgusShutDown(Normal Shutdown)
argus[17296]: 01 Jun 07 14:28:03.226690 ArgusCloseSource(0xb7db1008)
starting
argus[17296]: 01 Jun 07 14:28:03.226863 ArgusCloseSource(0xb7db1008)
deleting source
argus[17296]: 01 Jun 07 14:28:03.399853 ArgusCloseModeler(0x97db008)
argus[17296]: 01 Jun 07 14:28:03.400353 ArgusCloseOutput() scheduling
closure after writing records
argus[17296]: 01 Jun 07 14:28:03.400644 ArgusCloseOutput(0x97dc2e0) done
argus[17296]: 01 Jun 07 14:28:03.414111 ArgusShutDown()
Program exited normally.
So, I'm not sure how I can help at this point since the sanitized file is
ready but does not reproduce the exception, and I can not send along the
pcap which does produce an exception.
-Mike
On Fri, 1 Jun 2007 at 21:16, carter at qosient.com wrote:
|Hey Michael,
|I'm expecting to read the resulting packet file back through argus using the "-r file" option, and it segfaulting.
|Does it do that?
|Carter
|
|Carter Bullard
|QoSient LLC
|150 E. 57th Street Suite 12D
|New York, New York 10022
|+1 212 588-9133 Phone
|+1 212 588-9134 Fax
|
|-----Original Message-----
|From: Michael Hornung <hornung at cac.washington.edu>
|Date: Thu, 31 May 2007 14:59:27
|To:carter at qosient.com
|Cc:Argus <argus-info at lists.andrew.cmu.edu>
|Subject: Re: [ARGUS] argus-3.0.0 segfault (5/23 version)
|
|Are you expecting argus to reproduce the segfault by replaying it through
|argus by setting ARGUS_PACKET_CAPTURE_FILE in argus.conf? I tried that
|(leaving the rest of the config the same) and the segfault does not happen
|again. If the pcap will still be helpful to you, let me know and I'll put
|up the copy wiht sanitized IPs.
|
|-Mike
|
|On Thu, 24 May 2007 at 23:22, carter at qosient.com wrote:
|
||Hey Micheal,
||If you can share that would be ideal!!!
||You can use the argus write pcap file function that is turned on from the argus.conf file to try to get the packet file size down. It causes argus to write out the paxkets it receives, and so it will stop on the packet that causes the problem!!!
||
||Thanks!!!
||
||Carter
||
||
||Carter Bullard
||QoSient LLC
||150 E. 57th Street Suite 12D
||New York, New York 10022
||+1 212 588-9133 Phone
||+1 212 588-9134 Fax
||
||-----Original Message-----
||From: Michael Hornung <hornung at cac.washington.edu>
||Date: Thu, 24 May 2007 15:13:15
||To:argus-info at lists.andrew.cmu.edu
||Subject: [ARGUS] argus-3.0.0 segfault (5/23 version)
||
||I've got argus running on a Fedora Core 6 x86 Linux box. The argus daemon
||dies *very* regularly and so needs to be monitored. I finally got around
||to capturing a pcap for the duration of an argus session. Carter, let me
||know if you want this and I'll get it to you; it is 650MB uncompressed.
||Following is what I see when running argus in gdb:
||
||(gdb) run
||Starting program: /usr/local/sbin/argus
||argus[29762]: 24 May 07 14:56:36.593821 started
||argus[29762]: 24 May 07 14:56:36.596492 ArgusGetInterfaceStatus: interface
||eth1 is up
||argus[29762]: 24 May 07 14:56:41.031467 connect from 128.95.135.24
||
||Program received signal SIGSEGV, Segmentation fault.
||0x0805a340 in ArgusCreateFlowKey (model=0x9491008, flow=0x9492290,
|| hstruct=0x9492200) at ArgusUtil.c:785
||785 hstruct->hash ^= *ptr++;
||
||(gdb) bt full
||#0 0x0805a340 in ArgusCreateFlowKey (model=0x9491008, flow=0x9492290,
|| hstruct=0x9492200) at ArgusUtil.c:785
|| ptr = (unsigned int *) 0xe025000
|| key = (unsigned int *) 0x9492208
|| retn = 0
|| i = 19811198
|| len = -1
||#1 0x0804e71f in ArgusProcessPacket (model=0x9491008, p=0x949460a "",
|| length=90, tvp=0xbff0d5b8, type=0) at ArgusModeler.c:988
|| retn = 0
|| tflow = (struct ArgusSystemFlow *) 0x9492290
|| flow = (struct ArgusFlowStruct *) 0x94b9d78
|| nflow = (struct ArgusFlowStruct *) 0xc6ecbc8
|| ptr = 0x949473c "\031"
|| value = 0
||#2 0x08055b61 in ArgusEtherPacket (user=0xb7e4c008 "", h=0xbff0d5b8,
|| p=0x949460a "") at ArgusSource.c:623
|| ep = (struct ether_header *) 0x949460a
|| ind = 0
|| src = (struct ArgusSourceStruct *) 0xb7e4c008
|| tvp = (struct timeval *) 0xbff0d5b8
|| caplen = 90
|| length = 90
|| statbuf = {st_dev = 0,__pad1 = 0,__st_ino = 0, st_mode = 0,
|| st_nlink = 10354372, st_uid = 3086711688, st_gid = 0,
|| st_rdev = 44261669504811007,__pad2 = 18120, st_size =
||-4615955009626666608,
|| st_blksize = 10255072, st_blocks = -5189414748145497984, st_atim = {
|| tv_sec = 1, tv_nsec = 1}, st_mtim = {tv_sec = 0, tv_nsec = 134516346},
|| st_ctim = {tv_sec = 0, tv_nsec = 134911664}, st_ino = 10354372}
||#3 0x08066088 in pcap_read_linux ()
||No symbol table info available.
||#4 0x08057eeb in ArgusGetPackets (src=0xb7e4c008) at ArgusSource.c:1654
|| ArgusReadMask = {__fds_bits = {128, 0 <repeats 31 times>}}
|| ArgusWriteMask = {__fds_bits = {0 <repeats 32 times>}}
|| ArgusExceptMask = {__fds_bits = {0 <repeats 32 times>}}
|| tmp = 1
|| i = 0
|| width = 7
|| noerror = 1
|| fd = 7
|| found = 1
|| up = 1
|| wait = {tv_sec = 0, tv_usec = 20000}
||#5 0x0804b333 in main (argc=1, argv=0xbff0d984) at argus.c:464
|| commandlinew = 0
|| doconf = 0
|| dodebug = 0
|| i = 1
|| pid = 0
|| tmparg = 0x8049f30 "[\201�214�005"
|| filter = 0x0
|| statbuf = {st_dev = 64768,__pad1 = 0,__st_ino = 2688645,
|| st_mode = 33133, st_nlink = 1, st_uid = 500, st_gid = 500, st_rdev = 0,
||__pad2 = 0, st_size = 11114, st_blksize = 4096, st_blocks = 32, st_atim
||= {
|| tv_sec = 1180043663, tv_nsec = 0}, st_mtim = {tv_sec = 1180042005,
|| tv_nsec = 0}, st_ctim = {tv_sec = 1180042005, tv_nsec = 0},
|| st_ino = 2688645}
|| host = (struct hostent *) 0x80a94bc
|| commandlinei = 0
|| op = -1
|| path = "/etc/argus.conf\000argus", '\0' <repeats 8170 times>
||
||(gdb) print hstruct->hash
||$1 = 2710700798
||
||So again, let me know if the pcap or anything else will be helpful.
||
||-Mike
||
|
More information about the argus
mailing list