rasqlinsert Segfault

[David Edelman]

It is very reproducible I have power cycled the system, I have used ldd to
verify which dynamic libraries were in use and that they were the most
recent versions.

I have reproduced this with data from several different directories but I
will grab a small directory that causes the problem and forward that. This
is all data that have been digested by argus->radium (with labels
added)->rastream. I can also try it with data that have been archived
directly after being collected by Argus matching the same time periods.

--Dave'

On 8/19/13 2:46 PM, "Carter Bullard" <carter at qosient.com> wrote:

>Hey Dave,
>This is reproducible?  Any chance you can share the data directory?
>I'm curious about all the forksŠwe shouldn't have that many ???
>
>Carter
>
>
>On Aug 18, 2013, at 4:47 PM, "David Edelman" <dedelman at iname.com> wrote:
>
>> This is happening on a clean install of the client code from the latest
>>copy
>> on the website. The source data is known good and I am using tables that
>> have never existed in the database just to be sure that there is no
>> interaction with anything that might have been hanging around. As far
>>as I
>> can tell, the two instances of rasqlinsert that I have reading their
>>input
>> from my local radium instance seem to be doing well. Debug and GDB
>>output as
>> below. I suspect that
>> [Thread 0x7fffeeffd700 (LWP 19127) exited] is not a good omen.
>> 
>> --Dave
>> 
>> The ouput using -D 3 is:
>> 
>> ----------- Snip quite a bit of output that does an update then calls
>> ArgusCloseInput  ArgusReadConnection ArgusParseInit .
>> 
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.037
>>ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764415.670",ltime="1359764633.317",record="..." WHERE
>> srcid="68.192.1.15" and saddr="50.19.85.156" and
>>smac="00:12:1e:b2:b3:b2")
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusCloseInput(0xb7aad010) closing
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusCloseInput(0xb7aad010) done
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.062
>> ArgusReadConnection() read 16 bytes from pipe
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.062
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.063
>> ArgusParseInit(0x7f90b7d7a010 0x7f90b773b010
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.063
>> ArgusReadConnection(0xb773b010, 1) returning 1
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.104
>>ArgusSQLQuery
>> (INSERT INTO XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record)
>> VALUES
>> 
>>("1359764658.613","1359765766.811","68.192.1.15","208.80.154.224","00:12:
>>1e:
>> b2:b3:b2",...))
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.104
>>ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764184.748",ltime="1359764330.540",record="..." WHERE
>> srcid="68.192.1.15" and saddr="10.1.1.22" and smac="88:53:95:71:7d:11")
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.185
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa86345b0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359845604.796","1359847381.551","68.192.1.15","17.149.32.32","00:12:1e
>>:b2
>> :b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.185
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8638410, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359846823.408","1359846939.442","68.192.1.15","173.194.43.39","00:12:1
>>e:b
>> 2:b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.186
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8636af0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359845832.956","1359845833.208","68.192.1.15","38.229.0.51","00:12:1e:
>>b2:
>> b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8637780, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359845984.422","1359845988.236","68.192.1.15","70.37.129.250","00:12:1
>>e:b
>> 2:b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa86351d0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359845700.237","1359845700.544","68.192.1.15","78.46.20.251","00:12:1e
>>:b2
>> :b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8635e60, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
>>("1359845709.127","1359845709.399","68.192.1.15","67.207.136.212","00:12:
>>1e:
>> b2:b3:b2",...), 32) done
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.280
>>ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359765141.064",ltime="1359765864.797",record="..." WHERE
>> srcid="68.192.1.15" and saddr="10.1.1.27" and smac="18:b4:30:01:86:8a")
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.326
>> ArgusCloseInput(0xb773b010) closing
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.326
>> ArgusCloseInput(0xb773b010) done
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.339
>>ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764338.670",ltime="1359764341.412",record="..." WHERE
>> srcid="68.192.1.15" and saddr="208.71.123.1" and
>>smac="00:12:1e:b2:b3:b2")
>> Segmentation fault (core dumped)
>> 
>> 
>> 
>> 
>> # gdb rasqlinsert
>> GNU gdb (GDB) Fedora (7.5.1-38.fc18)
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /usr/local/bin/rasqlinsert...done.
>> (gdb) run  -M time 1d -r /data/argus/2013/06/01/* -M rmon -w
>> mysql://argus@localhost/argus/macAddrs_2013_06_01 -m srcid saddr smac -s
>> stime ltime srcid saddr smac - ip
>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r
>> /data/argus/2013/06/01/* -M rmon -w
>> mysql://argus@localhost/argus/macAddrs_2013_06_01 -m srcid saddr smac -s
>> stime ltime srcid saddr smac - ip
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Detaching after fork from child process 19123.
>> [New Thread 0x7ffff51a2700 (LWP 19124)]
>> [Thread 0x7ffff51a2700 (LWP 19124) exited]
>> [New Thread 0x7ffff51a2700 (LWP 19125)]
>> [New Thread 0x7fffef7fe700 (LWP 19126)]
>> [New Thread 0x7fffeeffd700 (LWP 19127)]
>> Detaching after fork from child process 19128.
>> Detaching after fork from child process 19130.
>> Detaching after fork from child process 19132.
>> Detaching after fork from child process 19135.
>> Detaching after fork from child process 19137.
>> Detaching after fork from child process 19139.
>> Detaching after fork from child process 19141.
>> Detaching after fork from child process 19143.
>> Detaching after fork from child process 19145.
>> Detaching after fork from child process 19147.
>> Detaching after fork from child process 19149.
>> Detaching after fork from child process 19151.
>> Detaching after fork from child process 19153.
>> Detaching after fork from child process 19155.
>> Detaching after fork from child process 19157.
>> Detaching after fork from child process 19159.
>> Detaching after fork from child process 19161.
>> Detaching after fork from child process 19163.
>> Detaching after fork from child process 19165.
>> Detaching after fork from child process 19167.
>> Detaching after fork from child process 19170.
>> Detaching after fork from child process 19172.
>> Detaching after fork from child process 19174.
>> Detaching after fork from child process 19176.
>> [Thread 0x7fffeeffd700 (LWP 19127) exited]
>> 
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffef7fe700 (LWP 19126)]
>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>> Missing separate debuginfos, use: debuginfo-install
>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>> ncurses-libs-5.9-11.20130511.fc18.x86_64
>>openssl-libs-1.0.1e-4.fc18.x86_64
>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>> (gdb) where
>> #0  0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>> #1  0x0000000000414ea9 in ArgusCursesProcessClose () at
>>./rasqlinsert.c:3291
>> #2  0x000000000040cf5d in ArgusCursesProcess (arg=0x0) at
>> ./rasqlinsert.c:449
>> #3  0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>> #4  0x0000003c88ef253d in clone () from /lib64/libc.so.6
>> (gdb) 
>> (gdb) up
>> #1  0x0000000000414ea9 in ArgusCursesProcessClose () at
>>./rasqlinsert.c:3291
>> 3291	   pthread_join(RaCursesInputThread, NULL);
>> (gdb) up
>> #2  0x000000000040cf5d in ArgusCursesProcess (arg=0x0) at
>> ./rasqlinsert.c:449
>> 449	   ArgusCursesProcessClose();
>> (gdb) up
>> #3  0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>> (gdb) up
>> #4  0x0000003c88ef253d in clone () from /lib64/libc.so.6
>> (gdb) up
>> Initial frame selected; you cannot go up.
>> (gdb)
>> 
>> 
>> 
>> 
>