new argus-clients-3.0.7.14 on the server
David Edelman
dedelman at iname.com
Wed Aug 21 10:26:57 EDT 2013
Carter,
That's great news and it makes sense. The instances that are processing
stream data don't have enough traffic to create huge updates, and the
instances that are cruising through files surely do. I have 128GB of
physical memory and I've tuned MySQL to use as much as possible. I can
detune it and see if that makes a difference.
--Dave
-----Original Message-----
From: Carter Bullard [mailto:carter at qosient.com]
Sent: Wednesday, August 21, 2013 8:10 AM
To: David Edelman
Cc: 'Argus'
Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
Hey David,
I'm also getting the same error here, with your example. Not sure where
this crept in, as I use rasqlinsert() on many systems and I haven't seen
this,
although I have started to see rasqlinsert()s that eat a lot of CPU.
This is a mysqld problem, where it coughs blood processing packets that
are too large, or where multiple threads are making concurrent
sql calls on the same socket.
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
We do try to send the largest query possible by packing as many INSERTS,
or UPDATES, into the mysql ' max_allowed_packet ', and we have multiple
threads talking to mysqld. I just now tried to cut the max_allowed_packet
buffer in 1/2, with no effect, so ..... I've got some work to do here and
will
try to have something today.
Thanks for the feedback !!!!
Carter
On Aug 20, 2013, at 7:43 PM, David Edelman <dedelman at iname.com> wrote:
> I had a theory that the problem happened when there were no more records
> available to be read. To test this I moved up one level in the data source
> directory tree and used -R * figuring that I should see good tables for
the
> dates prior to the most recent. The theory may be good but the test may
have
> been useful for a different reason. Newest client build with .devel and
> .debug run without a -D switch. I get this almost immediately (but this is
a
> big machine with lots of memory so speed is a hard thing to judge.)
>
>
> rasqlinsert -M time 1d -R * -M rmon -w
> mysql://argus:argus@localhost/argus/SSmacAddrs_%Y_%m_%d -m srcid saddr
smac
> -s stime ltime srcid saddr smac - ip
> *** glibc detected *** rasqlinsert: double free or corruption (!prev):
> 0x00000000020e4830 ***
> rasqlinsert[8994]: 2013-08-20-23:27:39.017 mysql_real_query error Lost
> connection to MySQL server during query
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3c88e7cb3e]
> /usr/lib64/mysql/libmysqlclient.so.18(vio_delete+0x26)[0x7f94870c5416]
> /usr/lib64/mysql/libmysqlclient.so.18(end_server+0x38)[0x7f94870a2678]
> /usr/lib64/mysql/libmysqlclient.so.18(cli_safe_read+0x30)[0x7f94870a2770]
> /usr/lib64/mysql/libmysqlclient.so.18(+0x41257)[0x7f94870a5257]
>
/usr/lib64/mysql/libmysqlclient.so.18(mysql_real_query+0x26)[0x7f94870a19a6]
>
/usr/lib64/mysql/libmysqlclient.so.18(mysql_list_tables+0x55)[0x7f948709edb5
> ]
> rasqlinsert[0x41fe3c]
> rasqlinsert[0x408c68]
> rasqlinsert[0x4086a5]
> rasqlinsert[0x438d35]
> rasqlinsert[0x43933e]
> rasqlinsert[0x47b12a]
> rasqlinsert[0x47b335]
> rasqlinsert[0x405be7]
> /lib64/libpthread.so.0(+0x3c89207d15)[0x7f9486bc4d15]
> /lib64/libc.so.6(clone+0x6d)[0x3c88ef253d]
> ======= Memory map: ========
> 00400000-004eb000 r-xp 00000000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006ea000-006eb000 r--p 000ea000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006eb000-006fb000 rw-p 000eb000 fd:00 20983991
> /usr/local/bin/rasqlinsert
> 006fb000-01007000 rw-p 00000000 00:00 0
> 020d8000-021c0000 rw-p 00000000 00:00 0
> [heap]
> 3c88a00000-3c88a20000 r-xp 00000000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c20000-3c88c21000 r--p 00020000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c21000-3c88c22000 rw-p 00021000 fd:00 77463577
> /usr/lib64/ld-2.16.so
> 3c88c22000-3c88c23000 rw-p 00000000 00:00 0
> 3c88e00000-3c88fad000 r-xp 00000000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c88fad000-3c891ad000 ---p 001ad000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891ad000-3c891b1000 r--p 001ad000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891b1000-3c891b3000 rw-p 001b1000 fd:00 77463656
> /usr/lib64/libc-2.16.so
> 3c891b3000-3c891b8000 rw-p 00000000 00:00 0
> 3c89200000-3c89202000 r-xp 00000000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89202000-3c89401000 ---p 00002000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89401000-3c89402000 r--p 00001000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89402000-3c89403000 rw-p 00002000 fd:00 77464193
> /usr/lib64/libpcreposix.so.0.0.1
> 3c89600000-3c89603000 r-xp 00000000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89603000-3c89802000 ---p 00003000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89802000-3c89803000 r--p 00002000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89803000-3c89804000 rw-p 00003000 fd:00 77471354
> /usr/lib64/libdl-2.16.so
> 3c89a00000-3c89a07000 r-xp 00000000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89a07000-3c89c06000 ---p 00007000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89c06000-3c89c07000 r--p 00006000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89c07000-3c89c08000 rw-p 00007000 fd:00 77463871
> /usr/lib64/librt-2.16.so
> 3c89e00000-3c89e15000 r-xp 00000000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c89e15000-3c8a014000 ---p 00015000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a014000-3c8a015000 r--p 00014000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a015000-3c8a016000 rw-p 00015000 fd:00 77471658
> /usr/lib64/libz.so.1.2.7
> 3c8a200000-3c8a300000 r-xp 00000000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a300000-3c8a4ff000 ---p 00100000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a4ff000-3c8a500000 r--p 000ff000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a500000-3c8a501000 rw-p 00100000 fd:00 77471671
> /usr/lib64/libm-2.16.so
> 3c8a600000-3c8a615000 r-xp 00000000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a615000-3c8a814000 ---p 00015000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a814000-3c8a815000 r--p 00014000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8a815000-3c8a816000 rw-p 00015000 fd:00 77471694
> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
> 3c8aa00000-3c8aa5c000 r-xp 00000000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8aa5c000-3c8ac5c000 ---p 0005c000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8ac5c000-3c8ac5d000 r--p 0005c000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8ac5d000-3c8ac5e000 rw-p 0005d000 fd:00 77471660
> /usr/lib64/libpcre.so.1.0.1
> 3c8b600000-3c8b63c000 r-xp 00000000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b63c000-3c8b83b000 ---p 0003c000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b83b000-3c8b83d000 r--p 0003b000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b83d000-3c8b843000 rw-p 0003d000 fd:00 77465274
> /usr/lib64/libreadline.so.6.2
> 3c8b843000-3c8b845000 rw-p 00000000 00:00 0
> 3c8ca00000-3c8cae5000 r-xp 00000000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8cae5000-3c8cce4000 ---p 000e5000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8cce4000-3c8ccec000 r--p 000e4000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8ccec000-3c8ccee000 rw-p 000ec000 fd:00 77471735
> /usr/lib64/libstdc++.so.6.0.17
> 3c8ccee000-3c8cd03000 rw-p 00000000 00:00 0
> 3c93a00000-3c93b9b000 r-xp 00000000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93b9b000-3c93d9b000 ---p 0019b000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93d9b000-3c93db5000 r--p 0019b000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93db5000-3c93dc0000 rw-p 001b5000 fd:00 77471858
> /usr/lib64/libcrypto.so.1.0.1e
> 3c93dc0000-3c93dc5000 rw-p 00000000 00:00 0
> 3ca1200000-3ca1225000 r-xp 00000000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1225000-3ca1424000 ---p 00025000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1424000-3ca1428000 r--p 00024000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca1428000-3ca1429000 rw-p 00028000 fd:00 77471104
> /usr/lib64/libtinfo.so.5.9
> 3ca2e00000-3ca2e23000 r-xp 00000000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca2e23000-3ca3022000 ---p 00023000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3022000-3ca3023000 r--p 00022000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3023000-3ca3024000 rw-p 00023000 fd:00 77471813
> /usr/lib64/libncurses.so.5.9
> 3ca3e00000-3ca3e16000 r-xp 00000000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca3e16000-3ca4015000 ---p 00016000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4015000-3ca4016000 r--p 00015000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4016000-3ca4017000 rw-p 00016000 fd:00 77470820
> /usr/lib64/libnsl-2.16.so
> 3ca4017000-3ca4019000 rw-p 00000000 00:00 0
> 3ca7600000-3ca7609000 r-xp 00000000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7609000-3ca7808000 ---p 00009000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7808000-3ca7809000 r--p 00008000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca7809000-3ca780a000 rw-p 00009000 fd:00 77472018
> /usr/lib64/libwrap.so.0.7.6
> 3ca780a000-3ca780b000 rw-p 00000000 00:00 0
> 7f9470000000-7f9470594000 rw-p 00000000 00:00 0
> 7f9470594000-7f9474000000 ---p 00000000 00:00 0
> 7f9474000000-7f9474021000 rw-p 00000000 00:00 0
> 7f9474021000-7f9478000000 ---p 00000000 00:00 0
> 7f9478000000-7f94782a5000 rw-p 00000000 00:00 0
> 7f94782a5000-7f947c000000 ---p 00000000 00:00 0
> 7f947c943000-7f947c944000 ---p 00000000 00:00 0
> 7f947c944000-7f947d144000 rw-p 00000000 00:00 0
> [stack:8999]
> 7f947d144000-7f947d145000 ---p 00000000 00:00 0
> 7f947d145000-7f947e146000 rw-p 00000000 00:00 0
> [stack:8998]
> 7f947e146000-7f947e152000 r-xp 00000000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e152000-7f947e351000 ---p 0000c000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e351000-7f947e352000 r--p 0000b000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e352000-7f947e353000 rw-p 0000c000 fd:00 77471291
> /usr/lib64/libnss_files-2.16.so
> 7f947e353000-7f947e354000 ---p 00000000 00:00 0
> 7f947e354000-7f9486bbd000 rw-p 00000000 00:00 0
> [stack:8997]
> 7f9486bbd000-7f9486bd3000 r-xp 00000000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486bd3000-7f9486dd3000 ---p 00016000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd3000-7f9486dd4000 r--p 00016000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd4000-7f9486dd5000 rw-p 00017000 fd:00 77463716
> /usr/lib64/libpthread-2.16.so
> 7f9486dd5000-7f9486dda000 rw-p 00000000 00:00 0
> 7f9486dda000-7f9486e60000 r-xp 00000000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9486e60000-7f948705f000 ---p 00086000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f948705f000-7f9487061000 r--p 00085000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9487061000-7f9487064000 rw-p 00087000 fd:00 77470840
> /usr/lib64/libft.so.0.0.0
> 7f9487064000-7f948735a000 r-xp 00000000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948735a000-7f948755a000 ---p 002f6000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948755a000-7f948762c000 rw-p 002f6000 fd:00 77599545
> /usr/lib64/mysql/libmysqlclient.so.18.0.0
> 7f948762c000-7f9487632000 rw-p 00000000 00:00 0
> 7f948764f000-7f9487652000 rw-p 00000000 00:00 0
> 7fff16923000-7fff16944000 rw-p 00000000 00:00 0
> [stack]
> 7fff169fe000-7fff16a00000 r-xp 00000000 00:00 0
> [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
> Aborted (core dumped)
>
> I run ldd against the image and I get this:
>
> ldd /usr/local/bin/rasqlinsert
> linux-vdso.so.1 => (0x00007fffaebfe000)
> libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x0000003c89200000)
> libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c8aa00000)
> libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
> (0x00007f3f8c855000)
> libm.so.6 => /lib64/libm.so.6 (0x0000003c8a200000)
> libft.so.0 => /lib64/libft.so.0 (0x00007f3f8c5cb000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8c3ae000)
> libz.so.1 => /lib64/libz.so.1 (0x0000003c89e00000)
> libncurses.so.5 => /lib64/libncurses.so.5 (0x0000003ca2e00000)
> libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003ca1200000)
> libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003c8b600000)
> libc.so.6 => /lib64/libc.so.6 (0x0000003c88e00000)
> libdl.so.2 => /lib64/libdl.so.2 (0x0000003c89600000)
> librt.so.1 => /lib64/librt.so.1 (0x0000003c89a00000)
> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000003c8ca00000)
> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c8a600000)
> libcrypto.so.10 => /lib64/libcrypto.so.10 (0x0000003c93a00000)
> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003ca3e00000)
> libwrap.so.0 => /lib64/libwrap.so.0 (0x0000003ca7600000)
> /lib64/ld-linux-x86-64.so.2 (0x0000003c88a00000)
>
> Checking the dates of the dynamic libraries indicates that I am linking to
> the most up to date versions.
>
> --Dave
>
>
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Tuesday, August 20, 2013 7:17 PM
> To: David Edelman
> Cc: Argus
> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>
> Hmmmm, I'll take another look tonight. It was working here with your
> file...frustrating !!!
>
> Carter
>
> On Aug 20, 2013, at 6:15 PM, "David Edelman" <dedelman at iname.com> wrote:
>
>> Carter,
>>
>> I'm having the exact same problem as before.
>>
>> I did a clean install after changing the string in VERSION so that I knew
>> that I was using new code. I applied the argus_label.c change which
didn't
>> make any difference. I created .debug and .devel; make clobber, ./config;
>> make; make install and ran under gdb and it is the same picture.
>>
>> The instances of rasqlinsert taking data from radium are as happy as
> clams.
>>
>> What additional material can I collect for you?
>>
>> --Dave
>>
>> -----Original Message-----
>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
>> Behalf Of Carter Bullard
>> Sent: Tuesday, August 20, 2013 10:59 AM
>> To: Argus
>> Subject: [ARGUS] new argus-clients-3.0.7.14 on the server
>>
>> Gentle people,
>> New client code up on the server. This release fixes all
>> known bugs that has been reported on the list, as well as
>> having major modifications to rapath().
>>
>> New code has been added as guards around the reported
>> label problems, but I am not sure that it has fixed
>> the problem. If we could test that, that would be great !!!
>>
>> We've made some big changes to rapath(). rapath() extracts
>> topology information from argus data. Basically it takes all
>> data that has ICMP TXD messages mapped to it, and tabulates path
>> information where it can. This has the effect of capturing all
>> traceroutes() that are observed by argus, regardless of the
>> techniqu; UDP, TCP or ICMP based, weather its vanilla or paris method,
>> or several of the proprietary strategies seen in intrusions.
>>
>> We've changed the default output of the graph that rapath.1
>> generates (using the -A option) to include the srcid, saddr
>> and daddr, so that you can build topology from just the
>> graphs. I'll add the stime and duration as well, but need
>> to figure out some command line options to control all these
>> new fields. Also rapath() is going to get a realtime mode,
>> currently, its a " read a file, generate some output " type of
>> tool.
>>
>> Please grab this code and give it a run. I'm hoping to
>> release 3.0.7.x as 3.0.8 in the next month, so if there are
>> any gotchas, don't hold back.
>>
>> Carter
>>
> <rasqlinsert-Dump.txt><rasqlinsertLDD.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6283 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130821/4a58807e/attachment.bin>
More information about the argus
mailing list