segfault at 000000000311c000 rip 000000000040fb46rsp 0000007fbffff830 error 4

Gunnar Lindberg Gunnar.Lindberg at chalmers.se
Mon May 11 17:10:17 EDT 2009


First of all, I re-run make with "-g" and used that with the existing
core file; I can't tell whether that should be OK but as far as I can
see it still makes some kind of sense.


[lindberg at argv ~]$ gdb argus-g core.14369
(gdb) where
#0  0x0000000000410bc2 in ArgusLoadList (l1=0x651460, l2=0x6540a0)
    at ArgusUtil.c:260
#1  0x000000000041557b in ArgusOutputProcess (arg=Variable "arg" is not available.
) at ArgusOutput.c:477
#2  0x000000000040bb6c in ArgusProcessPacket (src=Variable "src" is not available.
) at ArgusModeler.c:1324
#3  0x000000000040d006 in ArgusEtherPacket (user=0x2a95786010 "", h=Variable "h" is not available.
)
    at ArgusSource.c:716
#4  0x00000034e2f04bff in ?? () from /usr/lib64/libpcap.so.0.8.3
#5  0x0000000000410759 in ArgusGetPackets (src=0x2a95786010)
    at ArgusSource.c:2093
#6  0x0000000000404f83 in main (argc=1, argv=0x7fbffffe08) at argus.c:535


(gdb) print *l1
$3 = {start = 0x18c21f0, end = 0x17e98b0, count = 589, pushed = 3044164,
  popped = 0, loaded = 3043575, outputTime = {tv_sec = 0, tv_usec = 0},
  reportTime = {tv_sec = 0, tv_usec = 0}}

(gdb) print *l2
$5 = {start = 0x85d278d99c8b81d1, end = 0x63caa47a16f1492e,
  count = 1579875320, pushed = 1880390013, popped = 2415426777,
  loaded = 3722138485, outputTime = {tv_sec = 144115210689264557,
    tv_usec = 14930315638210660}, reportTime = {tv_sec = -7084847654803835648,
    tv_usec = -5817086215248780719}}

argus/ArgusUtil.c
    246 void
    247 ArgusLoadList(struct ArgusListStruct *l1, struct ArgusListStruct *l2)
    248 {
    249    if (l1 && l2) {
    250       int count;
    251 #if defined(ARGUS_THREADS)
    252       pthread_mutex_lock(&l1->lock);
    253       pthread_mutex_lock(&l2->lock);
    254 #endif
    255       count = l1->count;
    256
    257       if (l2->start == NULL)
    258          l2->start = l1->start;
    259       else
    260          l2->end->nxt = l1->start;
    261
    262       l2->end = l1->end;
    263       l2->count += count;
    264
    265       l1->start = NULL;
    266       l1->end = NULL;
    267       l1->loaded += count;
    268       l1->count = 0;
    269
    270 #if defined(ARGUS_THREADS)
    271       pthread_mutex_unlock(&l2->lock);
    272       pthread_mutex_unlock(&l1->lock);
    273 #endif
    274
    275 #ifdef ARGUSDEBUG
    276    ArgusDebug (5, "ArgusLoadList (0x%x, 0x%x) load %d objects\n", l1, l2        , count);
    277 #endif
    278    }
    279 }


		Gunnar Lindberg


>From SRS0=BzD3OK=BH=qosient.com=carter at srs.bis.na.blackberry.com  Mon May 11 13:18:08 2009
>Message-ID: <2044323243-1242040666-cardhu_decombobulator_blackberry.rim.net-2042564372- at bxe1165.bisx.prod.on.blackberry>
>Reply-To: carter at qosient.com
>References: <E5F8710F-522D-4579-8569-A9DD5E130A06 at qosient.com><200905110551.n4B5pV62007936 at grunert.cdg.chalmers.se>
>In-Reply-To: <200905110551.n4B5pV62007936 at grunert.cdg.chalmers.se>
>Subject: Re: [ARGUS] segfault at 000000000311c000 rip 000000000040fb46rsp	0000007fbffff830 error 4
>To: "Gunnar Lindberg" <Gunnar.Lindberg at chalmers.se>,
>        argus-info-bounces+carter=qosient.com at lists.andrew.cmu.edu,
>        "Argus" <argus-info at lists.andrew.cmu.edu>
>From: carter at qosient.com
>Date: Mon, 11 May 2009 11:19:44 +0000

>Hey Gunnar,
>The C level debugging in gdb() is very good, and gives you quick access to the symbols and stack info.
>
>I have never seen problems with ArgusLoadList(), so if you have a core file, if you could load it into gdb() and type:
>
>  (gdb) where
>  (gdb) print *l1. (assuming its in AtgusLoadList)
>  (gdb) print *l2
>
>If not, if you could run it under gdb() until it stops, and type the same, that would give me a good start.
>
>Carter
>
>Sent from my Verizon Wireless BlackBerry
>
>-----Original Message-----
>From: Gunnar Lindberg <Gunnar.Lindberg at chalmers.se>
>
>Date: Mon, 11 May 2009 07:51:31 
>To: <argus-info at lists.andrew.cmu.edu>
>Subject: Re: [ARGUS] segfault at 000000000311c000 rip 000000000040fb46
>	rsp	0000007fbffff830 error 4
>
>
>No .threads in argus-3.0.1.beta.3
>
>My gdb knowledge is limited but I've done quite some amount of
>C/machine code debugging in my early days (25 years ago and MC68000
>I'd probably been able to write the C code from the optimized
>assembler :-). But, this is *86 - "same, same, but different"...
>
>Based on that I did the "disass" trick and <<<=== indicates the
>machine code where the crash occured. What beats me on *86 is
>which register is used for which C variable, but there seems to
>have been an offset "0x8(%rsi),%r9" involved just before - that
>was variables in a C struct on MC68000 and I guess it still is.
>
>So we picked up something 8 bytes into a C struct and than tried
>to us it as a pointer "%r10,(%r9)" - and pooof.
>
>The most probable thing is that data/pointers got screwed up minutes
>ago and then the bomb goes off now because we just got to that data.
>However, before going through the linked list of data I'd like to ask
>about a line of C code:
>
>argus/ArgusUtil.c:
>
>void
>ArgusLoadList(struct ArgusListStruct *l1, struct ArgusListStruct *l2)
>{
>...
>      if (l2->start == NULL)
>         l2->start = l1->start;
>      else
>         l2->end->nxt = l1->start;		<=
>...
>}
>
>The only "nxt" I find is within a "struct ArgusListRecord",
>but "l2" and "l2->end" points at a "struct ArgusListStruct".
>Could this be it?
>
>Or is there some condition where l2->end is not correctly set?
>
>	Gunnar Lindberg
>
>May  7 16:33:30 argv kernel: argus[14369] general protection
>rip:410bc2 rsp:7fbffff308 error:0
>
>gdb argus.14369 /core.14369
>...
>#0  0x0000000000410bc2 in ArgusLoadList ()
>(gdb) where
>#0  0x0000000000410bc2 in ArgusLoadList ()
>#1  0x000000000041557b in ArgusOutputProcess ()
>#2  0x000000000040bb6c in ArgusProcessPacket ()
>#3  0x000000000040d006 in ArgusEtherPacket ()
>#4  0x00000034e2f04bff in ?? () from /usr/lib64/libpcap.so.0.8.3
>#5  0x0000000000410759 in ArgusGetPackets ()
>#6  0x0000000000404f83 in main ()
>(gdb) disass 0x0000000000410bc2
>Dump of assembler code for function ArgusLoadList:
>0x0000000000410ba0 <ArgusLoadList+0>:   test   %rdi,%rdi
>0x0000000000410ba3 <ArgusLoadList+3>:   setne  %dl
>0x0000000000410ba6 <ArgusLoadList+6>:   xor    %eax,%eax
>0x0000000000410ba8 <ArgusLoadList+8>:   test   %rsi,%rsi
>0x0000000000410bab <ArgusLoadList+11>:  setne  %al
>0x0000000000410bae <ArgusLoadList+14>:  test   %eax,%edx
>0x0000000000410bb0 <ArgusLoadList+16>:  je     0x410be9 <ArgusLoadList+73>
>0x0000000000410bb2 <ArgusLoadList+18>:  cmpq   $0x0,(%rsi)
>0x0000000000410bb6 <ArgusLoadList+22>:  mov    0x10(%rdi),%ecx
>0x0000000000410bb9 <ArgusLoadList+25>:  je     0x410bf0 <ArgusLoadList+80>
>0x0000000000410bbb <ArgusLoadList+27>:  mov    0x8(%rsi),%r9
>0x0000000000410bbf <ArgusLoadList+31>:  mov    (%rdi),%r10
>0x0000000000410bc2 <ArgusLoadList+34>:  mov    %r10,(%r9)       <<<===
>0x0000000000410bc5 <ArgusLoadList+37>:  mov    0x8(%rdi),%r11
>0x0000000000410bc9 <ArgusLoadList+41>:  add    %ecx,0x1c(%rdi)
>0x0000000000410bcc <ArgusLoadList+44>:  add    %ecx,0x10(%rsi)
>0x0000000000410bcf <ArgusLoadList+47>:  movq   $0x0,(%rdi)
>0x0000000000410bd6 <ArgusLoadList+54>:  movl   $0x0,0x10(%rdi)
>0x0000000000410bdd <ArgusLoadList+61>:  mov    %r11,0x8(%rsi)
>0x0000000000410be1 <ArgusLoadList+65>:  movq   $0x0,0x8(%rdi)
>0x0000000000410be9 <ArgusLoadList+73>:  repz retq
>0x0000000000410beb <ArgusLoadList+75>:  data16
>0x0000000000410bec <ArgusLoadList+76>:  data16
>0x0000000000410bed <ArgusLoadList+77>:  nop
>0x0000000000410bee <ArgusLoadList+78>:  data16
>0x0000000000410bef <ArgusLoadList+79>:  nop
>0x0000000000410bf0 <ArgusLoadList+80>:  mov    (%rdi),%r8
>0x0000000000410bf3 <ArgusLoadList+83>:  mov    %r8,(%rsi)
>0x0000000000410bf6 <ArgusLoadList+86>:  jmp    0x410bc5 <ArgusLoadList+37>
>0x0000000000410bf8 <ArgusLoadList+88>:  data16
>0x0000000000410bf9 <ArgusLoadList+89>:  data16
>0x0000000000410bfa <ArgusLoadList+90>:  data16
>0x0000000000410bfb <ArgusLoadList+91>:  nop
>0x0000000000410bfc <ArgusLoadList+92>:  data16
>0x0000000000410bfd <ArgusLoadList+93>:  data16
>0x0000000000410bfe <ArgusLoadList+94>:  data16
>0x0000000000410bff <ArgusLoadList+95>:  nop
>End of assembler dump.
>(gdb) info registers
>rax            0x1      1
>rbx            0x174f450        24441936
>rcx            0x24d    589
>rdx            0x4a02f101       1241706753
>rsi            0x6540a0 6635680
>rdi            0x651460 6624352
>rbp            0x6516c0 0x6516c0
>rsp            0x7fbffff308     0x7fbffff308
>r8             0x69c6d  433261
>r9             0x63caa47a16f1492e       7190740599328295214
>r10            0x18c21f0        25960944
>r11            0x41a1320        68817696
>r12            0x3      3
>r13            0x651738 6625080
>r14            0x0      0
>r15            0x7fbffff510     548682069264
>rip            0x410bc2 0x410bc2 <ArgusLoadList+34>
>eflags         0x10286  66182
>cs             0x33     51
>ss             0x2b     43
>ds             0x0      0
>es             0x0      0
>fs             0x0      0
>gs             0x0      0
>
>
>
>>From carter at qosient.com  Thu May  7 19:00:53 2009
>>Cc: argus-info at lists.andrew.cmu.edu
>>Message-Id: <E5F8710F-522D-4579-8569-A9DD5E130A06 at qosient.com>
>>From: Carter Bullard <carter at qosient.com>
>>To: Gunnar Lindberg <Gunnar.Lindberg at chalmers.se>
>>In-Reply-To: <200905071507.n47F7xeB026201 at grunert.cdg.chalmers.se>
>>Subject: Re: [ARGUS] segfault at 000000000311c000 rip 000000000040fb46 rsp	0000007fbffff830 error 4
>>Date: Thu, 7 May 2009 13:00:42 -0400
>>References: <200905071507.n47F7xeB026201 at grunert.cdg.chalmers.se>
>
>>Hey Gunnar,
>>The gdb() commands of interest are:
>
>>    (gdb) where
>
>>ArgusLoadList() is the routine that passes flow record status reports  
>>from the
>>packet processing engine to the output processor.  This definitely  
>>shouldn't
>>have a problem, so it will be interesting to figure out what the  
>>problem maybe.
>
>>Are you running with threads enabled?  (is there a ./.threads file in  
>>your root directory?)
>
>>Carter
>
>



More information about the argus mailing list