Tracking down infrequent raqglinsert() segfault (was new argus-clients-3.0.7.14 on the server)

[David Edelman]

Carter,

It looks like MAXSTRLEN is the problem. The current value is 4096 and sbuf
needs to be larger than that. I am going to experiment with 8192 but I have
no idea what else that is going to impact.
Initial results with MAXSTRLEN of 8192 are good, no SEGFAULTs and none of
those messages about malformed SQL statements.


--Dave



-----Original Message-----
From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
[mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
Behalf Of David Edelman
Sent: Sunday, August 25, 2013 10:45 PM
To: 'Carter Bullard'
Cc: 'Argus'
Subject: [ARGUS] Tracking down infrequent raqglinsert() segfault (was new
argus-clients-3.0.7.14 on the server)


Still moving right along using rasqlinsert to create two tables per day for
three months of data. I instrumented the bash for loop so I was able to tell
which source files were being processed when the segfault happened (FC 18 is
real stingy with core files)

The fault occurs at rasqlinsert.c line 3144 where the  ns-->status
ARGUS_RECORD_MODIFIED bit is being reset.
I fired up gdb and the problem is reproducible, it looks like something is
clobbering the value of ns between the test to see if the record has been
modified and the attempt to reset the ARGUS_RECORD_MODIFIED bit in
ns->status. It looks like it might be this code segment, specifically the
bcopy at line 7527. It has the feel of a buffer overrun since the new value
of the pointer as an ASCII string would be    )"0\0\1   which might make
more sense with endian adjustment  but I'm still looking for the specifics.

7523          if (ArgusSOptionRecord) {
7524             int tlen;
7525             snprintf (sbuf, slen, "INSERT INTO %s (%s,record) VALUES
(%s,\"", RaSQLCurrentTable, vbuf, fbuf);
7526             tlen = strlen(sbuf);
7527             bcopy(mbuf, &sbuf[tlen], len + 1);
7528             tlen = strlen(sbuf);
7529             snprintf (&sbuf[tlen], slen - tlen, "\")");
7530

--Dave

Reading symbols from /usr/local/bin/rasqlinsert...done.
(gdb) break 3140
Breakpoint 1 at 0x414933: file ./rasqlinsert.c, line 3140.
(gdb) run  -M time 1d -R /data/argus/2013/07/07 -w
mysql://argus:argus@localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s
ltime dur srcid saddr daddr proto bytes
Starting program: /usr/local/bin/rasqlinsert -M time 1d -R
/data/argus/2013/07/07 -w
mysql://argus:argus@localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s
ltime dur srcid saddr daddr proto bytes
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff51a2700 (LWP 31618)]
[Thread 0x7ffff51a2700 (LWP 31618) exited]
[New Thread 0x7ffff51a2700 (LWP 31619)]
[New Thread 0x7fffef7fe700 (LWP 31620)]
[New Thread 0x7fffeeffd700 (LWP 31621)]
Detaching after fork from child process 31622.
[Switching to Thread 0x7fffef7fe700 (LWP 31620)]

Breakpoint 1, ArgusDrawWindow (ws=0x7ffff00009a0) at ./rasqlinsert.c:3140
3140	                     struct ArgusRecordStruct *ns = (struct
ArgusRecordStruct *)RaCursesProcess->queue->array[i];
Missing separate debuginfos, use: debuginfo-install
flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
ncurses-libs-5.9-11.20130511.fc18.x86_64 openssl-libs-1.0.1e-4.fc18.x86_64
pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
(gdb) list
3135	               char sbuf[MAXSTRLEN];
3136	               int i;
3137	 
3138	               if (RaCursesProcess->queue->array != NULL) {
3139	                  for (i = 0; i < RaCursesProcess->queue->count;
i++) {
3140	                     struct ArgusRecordStruct *ns = (struct
ArgusRecordStruct *)RaCursesProcess->queue->array[i];
3141	 
3142	                     if (ns && (ns->status & ARGUS_RECORD_MODIFIED))
{
3143	                        ArgusScheduleSQLQuery (ArgusParser,
ArgusParser->ArgusAggregator, ns, sbuf, sizeof(sbuf), ARGUS_STATUS);
3144	                        ns->status &= ~ARGUS_RECORD_MODIFIED;
(gdb) display ns
1: ns = (struct ArgusRecordStruct *) 0x3c88c21000 <_rtld_local>
(gdb) display ns->status
2: ns->status = 127
(gdb) display i
3: i = 0
(gdb) display RaCursesProcess->queue->count
4: RaCursesProcess->queue->count = 194
(gdb) break 3144
Breakpoint 2 at 0x4149ad: file ./rasqlinsert.c, line 3144.
(gdb) continue
Continuing.

Breakpoint 2, ArgusDrawWindow (ws=0x7ffff00009a0) at ./rasqlinsert.c:3144
3144	                        ns->status &= ~ARGUS_RECORD_MODIFIED;
4: RaCursesProcess->queue->count = 194
3: i = 0
2: ns->status = <error: Cannot access memory at address 0x2922305c305c81>
1: ns = (struct ArgusRecordStruct *) 0x2922305c305c31
(gdb) 








-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6283 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130826/3365033e/attachment.bin>