segfault at 000000000311c000 rip 000000000040fb46rsp 0000007fbffff830 error 4

Sun May 17 07:00:18 EDT 2009

My business trip left me with a real cold, so I will probably not
be at the office in a few days (I do this from home - too curious
to stay away from mail :-).

We'll see what of the sugestions will apply (I've browsed forward
so at least I'm aware of them). However, before I go back to bed:

calloc() et.al. have their own area for bookkeeping - linked lists
of what is in use and what is free etc. This is data within our own
process, so if a pointer goes haywire we can write into it and will
affect the operation of these "solid routines". I think I mentioned
such a "ticking bomb" before - the word nightmare comes to mind.

Recalling from the top of my head: I think we capture 12 bytes of
user data (which we drop almost at once - it's really sensitive to
grab anything else than headers).

The protocols I'm aware of is IPv4 and IPv6, but since it's Ethernet
(and we don't have full control over it) other "interesting" things
may well have shown up.

Finally, the idea of catching the last packet in a file. Isn't that
last packet saved in a buffer somewhere in 'core.18482'? Maybe some-
one can instruct me how to find that data (pointer name, or how to
find it) and how to have gdp print the buffer.

	Gunnar Lindberg

>From carter at qosient.com  Fri May 15 16:10:50 2009
>Cc: argus-info at lists.andrew.cmu.edu
>Message-Id: <098FF7AB-8F76-4A16-B816-D409E840B884 at qosient.com>
>From: Carter Bullard <carter at qosient.com>
>To: Gunnar Lindberg <Gunnar.Lindberg at chalmers.se>
>In-Reply-To: <200905151218.n4FCIGH2007909 at grunert.cdg.chalmers.se>
>Subject: Re: [ARGUS] segfault at 000000000311c000 rip 000000000040fb46rsp	0000007fbffff830 error 4
>Date: Fri, 15 May 2009 10:10:18 -0400
>References: <200905151218.n4FCIGH2007909 at grunert.cdg.chalmers.se>

>Hey Gunnar,
>Very odd indeed.  You are dying in routines that are solid, not  
>problems in years etc....
>When a site is having problems that other sites are not, usually its  
>caused by a new
>traffic mix that argus can't handle (packet parsing errors), or  
>hardware/OS issues.

>I'm thinking new traffic mix, so you may have encapsulation  
>combinations that we
>don't see very often?    Many memory problems in argus, come from  
>parsing packets
>we haven't seen before, or parsing and processing packet headers  
>beyond the
>packet snap length.

>Many times the new packet is a non-IP packet, or its an interesting  
>IPv6 packet
>that we haven't seen before (weird ICMP codes etc...).  One way to  
>test, is to run
>argus with the simple "- ip" filter, to see if it gets farther.

>Filtering doesn't exclude the situation where the problem is caused by  
>new
>encapsulation combinations, such as multi-level 802.11P n P (multiple  
>stacked
>VLAN tags), or IP over IP over GRE over MPLS over VLAN over IP, which we
>should deal with, but you never know.

>And when there are odd parsing problems, it can be caused by parsing  
>headers
>beyond the snap length.  The packet snap length defaults to 96  
>bytes.   Which could
>be short for some weird protocols.   If you capture user data, argus()  
>increases
>the snaplen by that amount so we can capture the data from a single  
>packet if we
>have to.

>Are you capturing user data?

>If yes, how much, if no, then ....... Could you increase the snaplen  
>on your argus to
>see if your problems go away?  Capturing something like 128 should be  
>enough, I
>suspect.

>   argus -s 128

>If you could try any combination of the above, maybe that will be a  
>workaround
>until we find the real problem?

>Carter

>On May 15, 2009, at 8:18 AM, Gunnar Lindberg wrote:

>> May 15 09:09:48 argv kernel: argus[18482] general protection rip: 
>> 34e22696bd rsp:7fbfffe840 error:0
>>
>> argus-3.0.1.beta.3/
>> -rw-rw-r--   1 root root       0 May 15 07:13 .devel
>> /bin/ls: .threads: No such file or directory
>>
>> -rwxrwxr-x  1 lindberg lindberg   829883 May 15 08:43 argus
>> -rw-r--r--  1 lindberg lindberg 41558016 May 15 09:09 core.18482
>>
>> gdb argus-g.18482 core.18482
>> #0  0x00000034e22696bd in _int_malloc () from /lib64/tls/libc.so.6
>>
>> (gdb) where
>> #0  0x00000034e22696bd in _int_malloc () from /lib64/tls/libc.so.6
>> #1  0x00000034e226b420 in calloc () from /lib64/tls/libc.so.6
>> #2  0x0000000000423994 in ArgusCalloc (nitems=5, bytes=4) at  
>> argus_util.c:1385
>> #3  0x00000000004202a2 in ArgusUpdateAppState (model=0x659010,
>>   flowstr=0x6eee20, state=16 '\020') at ArgusApp.c:278
>> #4  0x000000000040b00f in ArgusUpdateState (model=0x659010,  
>> flowstr=0x6eee20,
>>   state=16 '\020') at ArgusModeler.c:2443
>> #5  0x000000000040a0ae in ArgusUpdateFlow (model=0x659010,  
>> flow=0x6eee20,
>>   state=16 '\020') at ArgusModeler.c:2068
>> #6  0x0000000000408317 in ArgusProcessPacket (src=0x2a95786010,  
>> p=0x65bb12 "",
>>   length=171, tvp=0x7fbfffec60, type=-1) at ArgusModeler.c:1316
>> #7  0x00000000004107db in ArgusEtherPacket (user=0x2a95786010 "",
>>   h=0x7fbfffece0, p=0x65bb12 "") at ArgusSource.c:716
>> #8  0x00000034e2f04bff in ?? () from /usr/lib64/libpcap.so.0.8.3
>> #9  0x0000000000413cd2 in ArgusGetPackets (src=0x2a95786010)
>>   at ArgusSource.c:2093
>> #10 0x0000000000404c77 in main (argc=1, argv=0x7fbffff5a8) at  
>> argus.c:535
>>
>> common/argus_util.c
>>  1362 void *
>>  1363 ArgusCalloc (int nitems, int bytes)
>>  1364 {
>> ...
>>  1385       retn = calloc (1, total + offset);
>> ...
>>  1437 }
>>
>>
>> 	Gunnar Lindberg
>>
>>
>> Hardware is Dell 2950.
>>
>> OS is RHEL Linux
>> /etc/redhat-release
>> Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
>>
>> # /bin/uname -a
>> Linux argc.irt.chalmers.se 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7  
>> 17:45:52 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
>>
>> # /bin/arch
>> x86_64
>>
>> gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-10)
>>
>>

>Carter Bullard
>CEO/President
>QoSient, LLC
>150 E 57th Street Suite 12D
>New York, New York  10022

>+1 212 588-9133 Phone
>+1 212 588-9134 Fax