new argus-clients-3.0.7.14 on the server

Carter Bullard carter at qosient.com
Wed Aug 21 08:10:22 EDT 2013 

Hey David,
I'm also getting the same error here, with your example.  Not sure where
this crept in, as I use rasqlinsert() on many systems and I haven't seen this,
although I have started to see rasqlinsert()s that eat a lot of CPU.

This is a mysqld problem, where it coughs blood processing packets that
are too large, or where multiple threads are making concurrent
sql calls on the same socket.  

   http://dev.mysql.com/doc/refman/5.0/en/gone-away.html

We do try to send the largest query possible by packing as many INSERTS,
or UPDATES, into the mysql ' max_allowed_packet ', and we have multiple
threads talking to mysqld.  I just now tried to cut the max_allowed_packet
buffer in 1/2, with no effect, so ..... I've got some work to do here and will
try to have something today.

Thanks for the feedback !!!!

Carter

On Aug 20, 2013, at 7:43 PM, David Edelman <dedelman at iname.com> wrote:

> I had a theory that the problem happened when there were no more records
> available to be read. To test this I moved up one level in the data source
> directory tree and used -R * figuring that I should see good tables for the
> dates prior to the most recent. The theory may be good but the test may have
> been useful for a different reason. Newest client build with .devel and
> .debug run without a -D switch. I get this almost immediately (but this is a
> big machine with lots of memory so speed is a hard thing to judge.)
> 
> 
> rasqlinsert -M time 1d -R *  -M rmon -w
> mysql://argus:argus@localhost/argus/SSmacAddrs_%Y_%m_%d -m srcid saddr smac
> -s stime ltime srcid saddr smac - ip
> *** glibc detected *** rasqlinsert: double free or corruption (!prev):
> 0x00000000020e4830 ***
> rasqlinsert[8994]: 2013-08-20-23:27:39.017 mysql_real_query error Lost
> connection to MySQL server during query
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3c88e7cb3e]
> /usr/lib64/mysql/libmysqlclient.so.18(vio_delete+0x26)[0x7f94870c5416]
> /usr/lib64/mysql/libmysqlclient.so.18(end_server+0x38)[0x7f94870a2678]
> /usr/lib64/mysql/libmysqlclient.so.18(cli_safe_read+0x30)[0x7f94870a2770]
> /usr/lib64/mysql/libmysqlclient.so.18(+0x41257)[0x7f94870a5257]
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_real_query+0x26)[0x7f94870a19a6]
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_list_tables+0x55)[0x7f948709edb5
> ]
> rasqlinsert[0x41fe3c]
> rasqlinsert[0x408c68]
> rasqlinsert[0x4086a5]
> rasqlinsert[0x438d35]
> rasqlinsert[0x43933e]
> rasqlinsert[0x47b12a]
> rasqlinsert[0x47b335]
> rasqlinsert[0x405be7]
> /lib64/libpthread.so.0(+0x3c89207d15)[0x7f9486bc4d15]
> /lib64/libc.so.6(clone+0x6d)[0x3c88ef253d]
> ======= Memory map: ========
> 00400000-004eb000 r-xp 00000000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006ea000-006eb000 r--p 000ea000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006eb000-006fb000 rw-p 000eb000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006fb000-01007000 rw-p 00000000 00:00 0 
> 020d8000-021c0000 rw-p 00000000 00:00 0
> [heap]
> 3c88a00000-3c88a20000 r-xp 00000000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c20000-3c88c21000 r--p 00020000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c21000-3c88c22000 rw-p 00021000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c22000-3c88c23000 rw-p 00000000 00:00 0 
> 3c88e00000-3c88fad000 r-xp 00000000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c88fad000-3c891ad000 ---p 001ad000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891ad000-3c891b1000 r--p 001ad000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891b1000-3c891b3000 rw-p 001b1000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891b3000-3c891b8000 rw-p 00000000 00:00 0 
> 3c89200000-3c89202000 r-xp 00000000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89202000-3c89401000 ---p 00002000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89401000-3c89402000 r--p 00001000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89402000-3c89403000 rw-p 00002000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89600000-3c89603000 r-xp 00000000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89603000-3c89802000 ---p 00003000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89802000-3c89803000 r--p 00002000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89803000-3c89804000 rw-p 00003000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89a00000-3c89a07000 r-xp 00000000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89a07000-3c89c06000 ---p 00007000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89c06000-3c89c07000 r--p 00006000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89c07000-3c89c08000 rw-p 00007000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89e00000-3c89e15000 r-xp 00000000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c89e15000-3c8a014000 ---p 00015000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a014000-3c8a015000 r--p 00014000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a015000-3c8a016000 rw-p 00015000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a200000-3c8a300000 r-xp 00000000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a300000-3c8a4ff000 ---p 00100000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a4ff000-3c8a500000 r--p 000ff000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a500000-3c8a501000 rw-p 00100000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a600000-3c8a615000 r-xp 00000000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a615000-3c8a814000 ---p 00015000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a814000-3c8a815000 r--p 00014000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a815000-3c8a816000 rw-p 00015000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8aa00000-3c8aa5c000 r-xp 00000000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8aa5c000-3c8ac5c000 ---p 0005c000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8ac5c000-3c8ac5d000 r--p 0005c000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8ac5d000-3c8ac5e000 rw-p 0005d000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8b600000-3c8b63c000 r-xp 00000000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b63c000-3c8b83b000 ---p 0003c000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b83b000-3c8b83d000 r--p 0003b000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b83d000-3c8b843000 rw-p 0003d000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b843000-3c8b845000 rw-p 00000000 00:00 0 
> 3c8ca00000-3c8cae5000 r-xp 00000000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8cae5000-3c8cce4000 ---p 000e5000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8cce4000-3c8ccec000 r--p 000e4000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8ccec000-3c8ccee000 rw-p 000ec000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8ccee000-3c8cd03000 rw-p 00000000 00:00 0 
> 3c93a00000-3c93b9b000 r-xp 00000000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93b9b000-3c93d9b000 ---p 0019b000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93d9b000-3c93db5000 r--p 0019b000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93db5000-3c93dc0000 rw-p 001b5000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93dc0000-3c93dc5000 rw-p 00000000 00:00 0 
> 3ca1200000-3ca1225000 r-xp 00000000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1225000-3ca1424000 ---p 00025000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1424000-3ca1428000 r--p 00024000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1428000-3ca1429000 rw-p 00028000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca2e00000-3ca2e23000 r-xp 00000000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca2e23000-3ca3022000 ---p 00023000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3022000-3ca3023000 r--p 00022000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3023000-3ca3024000 rw-p 00023000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3e00000-3ca3e16000 r-xp 00000000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca3e16000-3ca4015000 ---p 00016000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4015000-3ca4016000 r--p 00015000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4016000-3ca4017000 rw-p 00016000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4017000-3ca4019000 rw-p 00000000 00:00 0 
> 3ca7600000-3ca7609000 r-xp 00000000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7609000-3ca7808000 ---p 00009000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7808000-3ca7809000 r--p 00008000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7809000-3ca780a000 rw-p 00009000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca780a000-3ca780b000 rw-p 00000000 00:00 0 
> 7f9470000000-7f9470594000 rw-p 00000000 00:00 0 
> 7f9470594000-7f9474000000 ---p 00000000 00:00 0 
> 7f9474000000-7f9474021000 rw-p 00000000 00:00 0 
> 7f9474021000-7f9478000000 ---p 00000000 00:00 0 
> 7f9478000000-7f94782a5000 rw-p 00000000 00:00 0 
> 7f94782a5000-7f947c000000 ---p 00000000 00:00 0 
> 7f947c943000-7f947c944000 ---p 00000000 00:00 0 
> 7f947c944000-7f947d144000 rw-p 00000000 00:00 0
> [stack:8999]
> 7f947d144000-7f947d145000 ---p 00000000 00:00 0 
> 7f947d145000-7f947e146000 rw-p 00000000 00:00 0
> [stack:8998]
> 7f947e146000-7f947e152000 r-xp 00000000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e152000-7f947e351000 ---p 0000c000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e351000-7f947e352000 r--p 0000b000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e352000-7f947e353000 rw-p 0000c000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e353000-7f947e354000 ---p 00000000 00:00 0 
> 7f947e354000-7f9486bbd000 rw-p 00000000 00:00 0
> [stack:8997]
> 7f9486bbd000-7f9486bd3000 r-xp 00000000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486bd3000-7f9486dd3000 ---p 00016000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd3000-7f9486dd4000 r--p 00016000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd4000-7f9486dd5000 rw-p 00017000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd5000-7f9486dda000 rw-p 00000000 00:00 0 
> 7f9486dda000-7f9486e60000 r-xp 00000000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9486e60000-7f948705f000 ---p 00086000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f948705f000-7f9487061000 r--p 00085000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9487061000-7f9487064000 rw-p 00087000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9487064000-7f948735a000 r-xp 00000000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948735a000-7f948755a000 ---p 002f6000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948755a000-7f948762c000 rw-p 002f6000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948762c000-7f9487632000 rw-p 00000000 00:00 0 
> 7f948764f000-7f9487652000 rw-p 00000000 00:00 0 
> 7fff16923000-7fff16944000 rw-p 00000000 00:00 0
> [stack]
> 7fff169fe000-7fff16a00000 r-xp 00000000 00:00 0
> [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
> Aborted (core dumped)
> 
> I run ldd against the image and I get this:
> 
> ldd /usr/local/bin/rasqlinsert 
> 	linux-vdso.so.1 =>  (0x00007fffaebfe000)
> 	libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x0000003c89200000)
> 	libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c8aa00000)
> 	libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
> (0x00007f3f8c855000)
> 	libm.so.6 => /lib64/libm.so.6 (0x0000003c8a200000)
> 	libft.so.0 => /lib64/libft.so.0 (0x00007f3f8c5cb000)
> 	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8c3ae000)
> 	libz.so.1 => /lib64/libz.so.1 (0x0000003c89e00000)
> 	libncurses.so.5 => /lib64/libncurses.so.5 (0x0000003ca2e00000)
> 	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003ca1200000)
> 	libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003c8b600000)
> 	libc.so.6 => /lib64/libc.so.6 (0x0000003c88e00000)
> 	libdl.so.2 => /lib64/libdl.so.2 (0x0000003c89600000)
> 	librt.so.1 => /lib64/librt.so.1 (0x0000003c89a00000)
> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000003c8ca00000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c8a600000)
> 	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x0000003c93a00000)
> 	libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003ca3e00000)
> 	libwrap.so.0 => /lib64/libwrap.so.0 (0x0000003ca7600000)
> 	/lib64/ld-linux-x86-64.so.2 (0x0000003c88a00000)
> 
> Checking the dates of the dynamic libraries indicates that I am linking to
> the most up to date versions.
> 
> --Dave
> 
> 
> 
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Tuesday, August 20, 2013 7:17 PM
> To: David Edelman
> Cc: Argus
> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
> 
> Hmmmm, I'll take another look tonight.  It was working here with your
> file...frustrating !!!
> 
> Carter
> 
> On Aug 20, 2013, at 6:15 PM, "David Edelman" <dedelman at iname.com> wrote:
> 
>> Carter,
>> 
>> I'm having the exact same problem as before. 
>> 
>> I did a clean install after changing the string in VERSION so that I knew
>> that I was using new code. I applied the argus_label.c change which didn't
>> make any difference. I created .debug and .devel; make clobber, ./config;
>> make; make install and ran under gdb and it is the same picture.
>> 
>> The instances of rasqlinsert taking data from radium are as happy as
> clams. 
>> 
>> What additional material can I collect for you?
>> 
>> --Dave
>> 
>> -----Original Message-----
>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
>> Behalf Of Carter Bullard
>> Sent: Tuesday, August 20, 2013 10:59 AM
>> To: Argus
>> Subject: [ARGUS] new argus-clients-3.0.7.14 on the server
>> 
>> Gentle people,
>> New client code up on the server.  This release fixes all
>> known bugs that has been reported on the list, as well as
>> having major modifications to rapath().
>> 
>> New code has been added as guards around the reported
>> label problems, but I am not sure that it has fixed
>> the problem.  If we could test that, that would be great !!!
>> 
>> We've made some big changes to rapath().  rapath() extracts
>> topology information from argus data.  Basically it takes all
>> data that has ICMP TXD messages mapped to it, and tabulates path
>> information where it can.  This has the effect of capturing all
>> traceroutes() that are observed by argus, regardless of the
>> techniqu;  UDP, TCP or ICMP based, weather its vanilla or paris method,
>> or several of the proprietary strategies seen in intrusions.
>> 
>> We've changed the default output of the graph that rapath.1
>> generates (using the -A option) to include the srcid, saddr
>> and daddr, so that you can build topology from just the
>> graphs.  I'll add the stime and duration as well, but need
>> to figure out some command line options to control all these
>> new fields.  Also rapath() is going to get a realtime mode,
>> currently, its a " read a file, generate some output " type of
>> tool.
>> 
>> Please grab this code and give it a run.  I'm hoping to
>> release 3.0.7.x as 3.0.8 in the next month, so if there are
>> any gotchas, don't hold back.
>> 
>> Carter
>> 
> <rasqlinsert-Dump.txt><rasqlinsertLDD.txt>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7322 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130821/c428e807/attachment.bin>

More information about the argus mailing list