rasqlinsert Segfault

Tue Aug 20 09:59:03 EDT 2013

Hey Dave,
With these very small files, there is/was a scheduling problem,
where all the sub-threads would realize input was completed, and
exit, but there were flows that needed to be pushed into the
database.   This resulted in the db thread using structures
that were in unknown states.

I've fixed that, and will upload code this morning, but that wouldn't
explain your other scenario where you have processing for a
long while and then booom.  Take a look, and run it through
a few tests.  Please send some form of feedback so I can know
if we're on the right track.

Carter

On Aug 19, 2013, at 5:32 PM, "David Edelman" <dedelman at iname.com> wrote:

> This may come as a surprise, but an input file of a single record will cause
> the segfault. It seems that the first file is read regardless of size, and
> the fault happens either at the close of the file or at the opening of the
> next file. It doesn't seem to matter if the directory contains several files
> or only a single file. Attached is a file with a single flow record that
> consistently causes a segfault when I attempt to read it with:
> 
> rasqlinsert  -M time 1d -r /data/test/argusTestOne  -M rmon -w
> mysql://argus@localhost/argus/AAmacAddrs_%Y_%m_%d  -m srcid saddr smac -s
> stime ltime srcid saddr smac - ip
> 
> --Dave
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Monday, August 19, 2013 10:47 AM
> To: David Edelman
> Cc: Argus
> Subject: Re: rasqlinsert Segfault
> 
> Hey Dave,
> This is reproducible?  Any chance you can share the data directory?
> I'm curious about all the forks.we shouldn't have that many ???
> 
> Carter
> 
> 
> On Aug 18, 2013, at 4:47 PM, "David Edelman" <dedelman at iname.com> wrote:
> 
>> This is happening on a clean install of the client code from the latest
> copy
>> on the website. The source data is known good and I am using tables that
>> have never existed in the database just to be sure that there is no
>> interaction with anything that might have been hanging around. As far as I
>> can tell, the two instances of rasqlinsert that I have reading their input
>> from my local radium instance seem to be doing well. Debug and GDB output
> as
>> below. I suspect that 
>> [Thread 0x7fffeeffd700 (LWP 19127) exited] is not a good omen.
>> 
>> --Dave
>> 
>> The ouput using -D 3 is:
>> 
>> ----------- Snip quite a bit of output that does an update then calls
>> ArgusCloseInput  ArgusReadConnection ArgusParseInit . 
>> 
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.037 ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764415.670",ltime="1359764633.317",record="..." WHERE
>> srcid="68.192.1.15" and saddr="50.19.85.156" and smac="00:12:1e:b2:b3:b2")
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusCloseInput(0xb7aad010) closing
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusCloseInput(0xb7aad010) done
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.058
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.062
>> ArgusReadConnection() read 16 bytes from pipe
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.062
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.063
>> ArgusParseInit(0x7f90b7d7a010 0x7f90b773b010
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.063
>> ArgusReadConnection(0xb773b010, 1) returning 1
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.104 ArgusSQLQuery
>> (INSERT INTO XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record)
>> VALUES
>> 
> ("1359764658.613","1359765766.811","68.192.1.15","208.80.154.224","00:12:1e:
>> b2:b3:b2",...))
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.104 ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764184.748",ltime="1359764330.540",record="..." WHERE
>> srcid="68.192.1.15" and saddr="10.1.1.22" and smac="88:53:95:71:7d:11")
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.185
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa86345b0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359845604.796","1359847381.551","68.192.1.15","17.149.32.32","00:12:1e:b2
>> :b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.185
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8638410, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359846823.408","1359846939.442","68.192.1.15","173.194.43.39","00:12:1e:b
>> 2:b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.186
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8636af0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359845832.956","1359845833.208","68.192.1.15","38.229.0.51","00:12:1e:b2:
>> b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8637780, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359845984.422","1359845988.236","68.192.1.15","70.37.129.250","00:12:1e:b
>> 2:b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa86351d0, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359845700.237","1359845700.544","68.192.1.15","78.46.20.251","00:12:1e:b2
>> :b3:b2",...), 32) done
>> rasqlinsert[3901.002785b4907f0000]: 2013-08-18-20:35:19.188
>> ArgusScheduleSQLQuery (0xb7d7a010, 0x1eddcb0, 0xa8635e60, INSERT INTO
>> XXmacAddrs_2013_02_02 (stime,ltime,srcid,saddr,smac,record) VALUES
>> 
> ("1359845709.127","1359845709.399","68.192.1.15","67.207.136.212","00:12:1e:
>> b2:b3:b2",...), 32) done
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.280 ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359765141.064",ltime="1359765864.797",record="..." WHERE
>> srcid="68.192.1.15" and saddr="10.1.1.27" and smac="18:b4:30:01:86:8a")
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.326
>> ArgusCloseInput(0xb773b010) closing
>> rasqlinsert[3901.00f7ffaf907f0000]: 2013-08-18-20:35:19.326
>> ArgusCloseInput(0xb773b010) done
>> rasqlinsert[3901.0017a6b5907f0000]: 2013-08-18-20:35:19.339 ArgusSQLQuery
>> (UPDATE XXmacAddrs_2013_02_02 SET
>> stime="1359764338.670",ltime="1359764341.412",record="..." WHERE
>> srcid="68.192.1.15" and saddr="208.71.123.1" and smac="00:12:1e:b2:b3:b2")
>> Segmentation fault (core dumped)
>> 
>> 
>> 
>> 
>> # gdb rasqlinsert
>> GNU gdb (GDB) Fedora (7.5.1-38.fc18)
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /usr/local/bin/rasqlinsert...done.
>> (gdb) run  -M time 1d -r /data/argus/2013/06/01/* -M rmon -w
>> mysql://argus@localhost/argus/macAddrs_2013_06_01 -m srcid saddr smac -s
>> stime ltime srcid saddr smac - ip
>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r
>> /data/argus/2013/06/01/* -M rmon -w
>> mysql://argus@localhost/argus/macAddrs_2013_06_01 -m srcid saddr smac -s
>> stime ltime srcid saddr smac - ip
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Detaching after fork from child process 19123.
>> [New Thread 0x7ffff51a2700 (LWP 19124)]
>> [Thread 0x7ffff51a2700 (LWP 19124) exited]
>> [New Thread 0x7ffff51a2700 (LWP 19125)]
>> [New Thread 0x7fffef7fe700 (LWP 19126)]
>> [New Thread 0x7fffeeffd700 (LWP 19127)]
>> Detaching after fork from child process 19128.
>> Detaching after fork from child process 19130.
>> Detaching after fork from child process 19132.
>> Detaching after fork from child process 19135.
>> Detaching after fork from child process 19137.
>> Detaching after fork from child process 19139.
>> Detaching after fork from child process 19141.
>> Detaching after fork from child process 19143.
>> Detaching after fork from child process 19145.
>> Detaching after fork from child process 19147.
>> Detaching after fork from child process 19149.
>> Detaching after fork from child process 19151.
>> Detaching after fork from child process 19153.
>> Detaching after fork from child process 19155.
>> Detaching after fork from child process 19157.
>> Detaching after fork from child process 19159.
>> Detaching after fork from child process 19161.
>> Detaching after fork from child process 19163.
>> Detaching after fork from child process 19165.
>> Detaching after fork from child process 19167.
>> Detaching after fork from child process 19170.
>> Detaching after fork from child process 19172.
>> Detaching after fork from child process 19174.
>> Detaching after fork from child process 19176.
>> [Thread 0x7fffeeffd700 (LWP 19127) exited]
>> 
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffef7fe700 (LWP 19126)]
>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>> Missing separate debuginfos, use: debuginfo-install
>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>> ncurses-libs-5.9-11.20130511.fc18.x86_64 openssl-libs-1.0.1e-4.fc18.x86_64
>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>> (gdb) where
>> #0  0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>> #1  0x0000000000414ea9 in ArgusCursesProcessClose () at
> ./rasqlinsert.c:3291
>> #2  0x000000000040cf5d in ArgusCursesProcess (arg=0x0) at
>> ./rasqlinsert.c:449
>> #3  0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>> #4  0x0000003c88ef253d in clone () from /lib64/libc.so.6
>> (gdb) 
>> (gdb) up
>> #1  0x0000000000414ea9 in ArgusCursesProcessClose () at
> ./rasqlinsert.c:3291
>> 3291	   pthread_join(RaCursesInputThread, NULL);
>> (gdb) up
>> #2  0x000000000040cf5d in ArgusCursesProcess (arg=0x0) at
>> ./rasqlinsert.c:449
>> 449	   ArgusCursesProcessClose();
>> (gdb) up
>> #3  0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>> (gdb) up
>> #4  0x0000003c88ef253d in clone () from /lib64/libc.so.6
>> (gdb) up
>> Initial frame selected; you cannot go up.
>> (gdb)
>> 
>> 
>> 
>> 
> 
> <argusTestOne>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7322 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130820/8ff576af/attachment.bin>