new argus-clients-3.0.7.14 on the server

Carter Bullard carter at qosient.com
Wed Aug 21 11:31:24 EDT 2013 

I think I've found a knob to turn that might help.  Can you try this change?  This
controls how aggressively we will push modifications out to the database. 

==== //depot/argus/clients/examples/ramysql/raclient.c#8 - /Users/carter/argus/clients/examples/ramysql/raclient.c ====
810c810
<             RaCursesUpdateInterval.tv_sec  = 0;
---
>             RaCursesUpdateInterval.tv_sec  = 1;

You may still get a problem, but it should be better.  I'll keep looking into
a better fix that this, but any improvement means where in the right direction.

Carter

On Aug 21, 2013, at 10:26 AM, "David Edelman" <dedelman at iname.com> wrote:

> Carter,
> 
> That's great news and it makes sense. The instances that are processing
> stream data don't have enough traffic to create huge updates, and the
> instances that are cruising through files surely do. I have 128GB of
> physical memory and I've tuned MySQL to use as much as possible. I can
> detune it and see if that makes a difference.
> 
> --Dave
> 
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Wednesday, August 21, 2013 8:10 AM
> To: David Edelman
> Cc: 'Argus'
> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
> 
> Hey David,
> I'm also getting the same error here, with your example.  Not sure where
> this crept in, as I use rasqlinsert() on many systems and I haven't seen
> this,
> although I have started to see rasqlinsert()s that eat a lot of CPU.
> 
> This is a mysqld problem, where it coughs blood processing packets that
> are too large, or where multiple threads are making concurrent
> sql calls on the same socket.  
> 
>   http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
> 
> We do try to send the largest query possible by packing as many INSERTS,
> or UPDATES, into the mysql ' max_allowed_packet ', and we have multiple
> threads talking to mysqld.  I just now tried to cut the max_allowed_packet
> buffer in 1/2, with no effect, so ..... I've got some work to do here and
> will
> try to have something today.
> 
> Thanks for the feedback !!!!
> 
> Carter
> 
> On Aug 20, 2013, at 7:43 PM, David Edelman <dedelman at iname.com> wrote:
> 
>> I had a theory that the problem happened when there were no more records
>> available to be read. To test this I moved up one level in the data source
>> directory tree and used -R * figuring that I should see good tables for
> the
>> dates prior to the most recent. The theory may be good but the test may
> have
>> been useful for a different reason. Newest client build with .devel and
>> .debug run without a -D switch. I get this almost immediately (but this is
> a
>> big machine with lots of memory so speed is a hard thing to judge.)
>> 
>> 
>> rasqlinsert -M time 1d -R *  -M rmon -w
>> mysql://argus:argus@localhost/argus/SSmacAddrs_%Y_%m_%d -m srcid saddr
> smac
>> -s stime ltime srcid saddr smac - ip
>> *** glibc detected *** rasqlinsert: double free or corruption (!prev):
>> 0x00000000020e4830 ***
>> rasqlinsert[8994]: 2013-08-20-23:27:39.017 mysql_real_query error Lost
>> connection to MySQL server during query
>> ======= Backtrace: =========
>> /lib64/libc.so.6[0x3c88e7cb3e]
>> /usr/lib64/mysql/libmysqlclient.so.18(vio_delete+0x26)[0x7f94870c5416]
>> /usr/lib64/mysql/libmysqlclient.so.18(end_server+0x38)[0x7f94870a2678]
>> /usr/lib64/mysql/libmysqlclient.so.18(cli_safe_read+0x30)[0x7f94870a2770]
>> /usr/lib64/mysql/libmysqlclient.so.18(+0x41257)[0x7f94870a5257]
>> 
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_real_query+0x26)[0x7f94870a19a6]
>> 
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_list_tables+0x55)[0x7f948709edb5
>> ]
>> rasqlinsert[0x41fe3c]
>> rasqlinsert[0x408c68]
>> rasqlinsert[0x4086a5]
>> rasqlinsert[0x438d35]
>> rasqlinsert[0x43933e]
>> rasqlinsert[0x47b12a]
>> rasqlinsert[0x47b335]
>> rasqlinsert[0x405be7]
>> /lib64/libpthread.so.0(+0x3c89207d15)[0x7f9486bc4d15]
>> /lib64/libc.so.6(clone+0x6d)[0x3c88ef253d]
>> ======= Memory map: ========
>> 00400000-004eb000 r-xp 00000000 fd:00 20983991
>> /usr/local/bin/rasqlinsert
>> 006ea000-006eb000 r--p 000ea000 fd:00 20983991
>> /usr/local/bin/rasqlinsert
>> 006eb000-006fb000 rw-p 000eb000 fd:00 20983991
>> /usr/local/bin/rasqlinsert
>> 006fb000-01007000 rw-p 00000000 00:00 0 
>> 020d8000-021c0000 rw-p 00000000 00:00 0
>> [heap]
>> 3c88a00000-3c88a20000 r-xp 00000000 fd:00 77463577
>> /usr/lib64/ld-2.16.so
>> 3c88c20000-3c88c21000 r--p 00020000 fd:00 77463577
>> /usr/lib64/ld-2.16.so
>> 3c88c21000-3c88c22000 rw-p 00021000 fd:00 77463577
>> /usr/lib64/ld-2.16.so
>> 3c88c22000-3c88c23000 rw-p 00000000 00:00 0 
>> 3c88e00000-3c88fad000 r-xp 00000000 fd:00 77463656
>> /usr/lib64/libc-2.16.so
>> 3c88fad000-3c891ad000 ---p 001ad000 fd:00 77463656
>> /usr/lib64/libc-2.16.so
>> 3c891ad000-3c891b1000 r--p 001ad000 fd:00 77463656
>> /usr/lib64/libc-2.16.so
>> 3c891b1000-3c891b3000 rw-p 001b1000 fd:00 77463656
>> /usr/lib64/libc-2.16.so
>> 3c891b3000-3c891b8000 rw-p 00000000 00:00 0 
>> 3c89200000-3c89202000 r-xp 00000000 fd:00 77464193
>> /usr/lib64/libpcreposix.so.0.0.1
>> 3c89202000-3c89401000 ---p 00002000 fd:00 77464193
>> /usr/lib64/libpcreposix.so.0.0.1
>> 3c89401000-3c89402000 r--p 00001000 fd:00 77464193
>> /usr/lib64/libpcreposix.so.0.0.1
>> 3c89402000-3c89403000 rw-p 00002000 fd:00 77464193
>> /usr/lib64/libpcreposix.so.0.0.1
>> 3c89600000-3c89603000 r-xp 00000000 fd:00 77471354
>> /usr/lib64/libdl-2.16.so
>> 3c89603000-3c89802000 ---p 00003000 fd:00 77471354
>> /usr/lib64/libdl-2.16.so
>> 3c89802000-3c89803000 r--p 00002000 fd:00 77471354
>> /usr/lib64/libdl-2.16.so
>> 3c89803000-3c89804000 rw-p 00003000 fd:00 77471354
>> /usr/lib64/libdl-2.16.so
>> 3c89a00000-3c89a07000 r-xp 00000000 fd:00 77463871
>> /usr/lib64/librt-2.16.so
>> 3c89a07000-3c89c06000 ---p 00007000 fd:00 77463871
>> /usr/lib64/librt-2.16.so
>> 3c89c06000-3c89c07000 r--p 00006000 fd:00 77463871
>> /usr/lib64/librt-2.16.so
>> 3c89c07000-3c89c08000 rw-p 00007000 fd:00 77463871
>> /usr/lib64/librt-2.16.so
>> 3c89e00000-3c89e15000 r-xp 00000000 fd:00 77471658
>> /usr/lib64/libz.so.1.2.7
>> 3c89e15000-3c8a014000 ---p 00015000 fd:00 77471658
>> /usr/lib64/libz.so.1.2.7
>> 3c8a014000-3c8a015000 r--p 00014000 fd:00 77471658
>> /usr/lib64/libz.so.1.2.7
>> 3c8a015000-3c8a016000 rw-p 00015000 fd:00 77471658
>> /usr/lib64/libz.so.1.2.7
>> 3c8a200000-3c8a300000 r-xp 00000000 fd:00 77471671
>> /usr/lib64/libm-2.16.so
>> 3c8a300000-3c8a4ff000 ---p 00100000 fd:00 77471671
>> /usr/lib64/libm-2.16.so
>> 3c8a4ff000-3c8a500000 r--p 000ff000 fd:00 77471671
>> /usr/lib64/libm-2.16.so
>> 3c8a500000-3c8a501000 rw-p 00100000 fd:00 77471671
>> /usr/lib64/libm-2.16.so
>> 3c8a600000-3c8a615000 r-xp 00000000 fd:00 77471694
>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>> 3c8a615000-3c8a814000 ---p 00015000 fd:00 77471694
>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>> 3c8a814000-3c8a815000 r--p 00014000 fd:00 77471694
>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>> 3c8a815000-3c8a816000 rw-p 00015000 fd:00 77471694
>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>> 3c8aa00000-3c8aa5c000 r-xp 00000000 fd:00 77471660
>> /usr/lib64/libpcre.so.1.0.1
>> 3c8aa5c000-3c8ac5c000 ---p 0005c000 fd:00 77471660
>> /usr/lib64/libpcre.so.1.0.1
>> 3c8ac5c000-3c8ac5d000 r--p 0005c000 fd:00 77471660
>> /usr/lib64/libpcre.so.1.0.1
>> 3c8ac5d000-3c8ac5e000 rw-p 0005d000 fd:00 77471660
>> /usr/lib64/libpcre.so.1.0.1
>> 3c8b600000-3c8b63c000 r-xp 00000000 fd:00 77465274
>> /usr/lib64/libreadline.so.6.2
>> 3c8b63c000-3c8b83b000 ---p 0003c000 fd:00 77465274
>> /usr/lib64/libreadline.so.6.2
>> 3c8b83b000-3c8b83d000 r--p 0003b000 fd:00 77465274
>> /usr/lib64/libreadline.so.6.2
>> 3c8b83d000-3c8b843000 rw-p 0003d000 fd:00 77465274
>> /usr/lib64/libreadline.so.6.2
>> 3c8b843000-3c8b845000 rw-p 00000000 00:00 0 
>> 3c8ca00000-3c8cae5000 r-xp 00000000 fd:00 77471735
>> /usr/lib64/libstdc++.so.6.0.17
>> 3c8cae5000-3c8cce4000 ---p 000e5000 fd:00 77471735
>> /usr/lib64/libstdc++.so.6.0.17
>> 3c8cce4000-3c8ccec000 r--p 000e4000 fd:00 77471735
>> /usr/lib64/libstdc++.so.6.0.17
>> 3c8ccec000-3c8ccee000 rw-p 000ec000 fd:00 77471735
>> /usr/lib64/libstdc++.so.6.0.17
>> 3c8ccee000-3c8cd03000 rw-p 00000000 00:00 0 
>> 3c93a00000-3c93b9b000 r-xp 00000000 fd:00 77471858
>> /usr/lib64/libcrypto.so.1.0.1e
>> 3c93b9b000-3c93d9b000 ---p 0019b000 fd:00 77471858
>> /usr/lib64/libcrypto.so.1.0.1e
>> 3c93d9b000-3c93db5000 r--p 0019b000 fd:00 77471858
>> /usr/lib64/libcrypto.so.1.0.1e
>> 3c93db5000-3c93dc0000 rw-p 001b5000 fd:00 77471858
>> /usr/lib64/libcrypto.so.1.0.1e
>> 3c93dc0000-3c93dc5000 rw-p 00000000 00:00 0 
>> 3ca1200000-3ca1225000 r-xp 00000000 fd:00 77471104
>> /usr/lib64/libtinfo.so.5.9
>> 3ca1225000-3ca1424000 ---p 00025000 fd:00 77471104
>> /usr/lib64/libtinfo.so.5.9
>> 3ca1424000-3ca1428000 r--p 00024000 fd:00 77471104
>> /usr/lib64/libtinfo.so.5.9
>> 3ca1428000-3ca1429000 rw-p 00028000 fd:00 77471104
>> /usr/lib64/libtinfo.so.5.9
>> 3ca2e00000-3ca2e23000 r-xp 00000000 fd:00 77471813
>> /usr/lib64/libncurses.so.5.9
>> 3ca2e23000-3ca3022000 ---p 00023000 fd:00 77471813
>> /usr/lib64/libncurses.so.5.9
>> 3ca3022000-3ca3023000 r--p 00022000 fd:00 77471813
>> /usr/lib64/libncurses.so.5.9
>> 3ca3023000-3ca3024000 rw-p 00023000 fd:00 77471813
>> /usr/lib64/libncurses.so.5.9
>> 3ca3e00000-3ca3e16000 r-xp 00000000 fd:00 77470820
>> /usr/lib64/libnsl-2.16.so
>> 3ca3e16000-3ca4015000 ---p 00016000 fd:00 77470820
>> /usr/lib64/libnsl-2.16.so
>> 3ca4015000-3ca4016000 r--p 00015000 fd:00 77470820
>> /usr/lib64/libnsl-2.16.so
>> 3ca4016000-3ca4017000 rw-p 00016000 fd:00 77470820
>> /usr/lib64/libnsl-2.16.so
>> 3ca4017000-3ca4019000 rw-p 00000000 00:00 0 
>> 3ca7600000-3ca7609000 r-xp 00000000 fd:00 77472018
>> /usr/lib64/libwrap.so.0.7.6
>> 3ca7609000-3ca7808000 ---p 00009000 fd:00 77472018
>> /usr/lib64/libwrap.so.0.7.6
>> 3ca7808000-3ca7809000 r--p 00008000 fd:00 77472018
>> /usr/lib64/libwrap.so.0.7.6
>> 3ca7809000-3ca780a000 rw-p 00009000 fd:00 77472018
>> /usr/lib64/libwrap.so.0.7.6
>> 3ca780a000-3ca780b000 rw-p 00000000 00:00 0 
>> 7f9470000000-7f9470594000 rw-p 00000000 00:00 0 
>> 7f9470594000-7f9474000000 ---p 00000000 00:00 0 
>> 7f9474000000-7f9474021000 rw-p 00000000 00:00 0 
>> 7f9474021000-7f9478000000 ---p 00000000 00:00 0 
>> 7f9478000000-7f94782a5000 rw-p 00000000 00:00 0 
>> 7f94782a5000-7f947c000000 ---p 00000000 00:00 0 
>> 7f947c943000-7f947c944000 ---p 00000000 00:00 0 
>> 7f947c944000-7f947d144000 rw-p 00000000 00:00 0
>> [stack:8999]
>> 7f947d144000-7f947d145000 ---p 00000000 00:00 0 
>> 7f947d145000-7f947e146000 rw-p 00000000 00:00 0
>> [stack:8998]
>> 7f947e146000-7f947e152000 r-xp 00000000 fd:00 77471291
>> /usr/lib64/libnss_files-2.16.so
>> 7f947e152000-7f947e351000 ---p 0000c000 fd:00 77471291
>> /usr/lib64/libnss_files-2.16.so
>> 7f947e351000-7f947e352000 r--p 0000b000 fd:00 77471291
>> /usr/lib64/libnss_files-2.16.so
>> 7f947e352000-7f947e353000 rw-p 0000c000 fd:00 77471291
>> /usr/lib64/libnss_files-2.16.so
>> 7f947e353000-7f947e354000 ---p 00000000 00:00 0 
>> 7f947e354000-7f9486bbd000 rw-p 00000000 00:00 0
>> [stack:8997]
>> 7f9486bbd000-7f9486bd3000 r-xp 00000000 fd:00 77463716
>> /usr/lib64/libpthread-2.16.so
>> 7f9486bd3000-7f9486dd3000 ---p 00016000 fd:00 77463716
>> /usr/lib64/libpthread-2.16.so
>> 7f9486dd3000-7f9486dd4000 r--p 00016000 fd:00 77463716
>> /usr/lib64/libpthread-2.16.so
>> 7f9486dd4000-7f9486dd5000 rw-p 00017000 fd:00 77463716
>> /usr/lib64/libpthread-2.16.so
>> 7f9486dd5000-7f9486dda000 rw-p 00000000 00:00 0 
>> 7f9486dda000-7f9486e60000 r-xp 00000000 fd:00 77470840
>> /usr/lib64/libft.so.0.0.0
>> 7f9486e60000-7f948705f000 ---p 00086000 fd:00 77470840
>> /usr/lib64/libft.so.0.0.0
>> 7f948705f000-7f9487061000 r--p 00085000 fd:00 77470840
>> /usr/lib64/libft.so.0.0.0
>> 7f9487061000-7f9487064000 rw-p 00087000 fd:00 77470840
>> /usr/lib64/libft.so.0.0.0
>> 7f9487064000-7f948735a000 r-xp 00000000 fd:00 77599545
>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>> 7f948735a000-7f948755a000 ---p 002f6000 fd:00 77599545
>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>> 7f948755a000-7f948762c000 rw-p 002f6000 fd:00 77599545
>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>> 7f948762c000-7f9487632000 rw-p 00000000 00:00 0 
>> 7f948764f000-7f9487652000 rw-p 00000000 00:00 0 
>> 7fff16923000-7fff16944000 rw-p 00000000 00:00 0
>> [stack]
>> 7fff169fe000-7fff16a00000 r-xp 00000000 00:00 0
>> [vdso]
>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>> [vsyscall]
>> Aborted (core dumped)
>> 
>> I run ldd against the image and I get this:
>> 
>> ldd /usr/local/bin/rasqlinsert 
>> 	linux-vdso.so.1 =>  (0x00007fffaebfe000)
>> 	libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x0000003c89200000)
>> 	libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c8aa00000)
>> 	libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
>> (0x00007f3f8c855000)
>> 	libm.so.6 => /lib64/libm.so.6 (0x0000003c8a200000)
>> 	libft.so.0 => /lib64/libft.so.0 (0x00007f3f8c5cb000)
>> 	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8c3ae000)
>> 	libz.so.1 => /lib64/libz.so.1 (0x0000003c89e00000)
>> 	libncurses.so.5 => /lib64/libncurses.so.5 (0x0000003ca2e00000)
>> 	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003ca1200000)
>> 	libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003c8b600000)
>> 	libc.so.6 => /lib64/libc.so.6 (0x0000003c88e00000)
>> 	libdl.so.2 => /lib64/libdl.so.2 (0x0000003c89600000)
>> 	librt.so.1 => /lib64/librt.so.1 (0x0000003c89a00000)
>> 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000003c8ca00000)
>> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c8a600000)
>> 	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x0000003c93a00000)
>> 	libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003ca3e00000)
>> 	libwrap.so.0 => /lib64/libwrap.so.0 (0x0000003ca7600000)
>> 	/lib64/ld-linux-x86-64.so.2 (0x0000003c88a00000)
>> 
>> Checking the dates of the dynamic libraries indicates that I am linking to
>> the most up to date versions.
>> 
>> --Dave
>> 
>> 
>> 
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com] 
>> Sent: Tuesday, August 20, 2013 7:17 PM
>> To: David Edelman
>> Cc: Argus
>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>> 
>> Hmmmm, I'll take another look tonight.  It was working here with your
>> file...frustrating !!!
>> 
>> Carter
>> 
>> On Aug 20, 2013, at 6:15 PM, "David Edelman" <dedelman at iname.com> wrote:
>> 
>>> Carter,
>>> 
>>> I'm having the exact same problem as before. 
>>> 
>>> I did a clean install after changing the string in VERSION so that I knew
>>> that I was using new code. I applied the argus_label.c change which
> didn't
>>> make any difference. I created .debug and .devel; make clobber, ./config;
>>> make; make install and ran under gdb and it is the same picture.
>>> 
>>> The instances of rasqlinsert taking data from radium are as happy as
>> clams. 
>>> 
>>> What additional material can I collect for you?
>>> 
>>> --Dave
>>> 
>>> -----Original Message-----
>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
>>> Behalf Of Carter Bullard
>>> Sent: Tuesday, August 20, 2013 10:59 AM
>>> To: Argus
>>> Subject: [ARGUS] new argus-clients-3.0.7.14 on the server
>>> 
>>> Gentle people,
>>> New client code up on the server.  This release fixes all
>>> known bugs that has been reported on the list, as well as
>>> having major modifications to rapath().
>>> 
>>> New code has been added as guards around the reported
>>> label problems, but I am not sure that it has fixed
>>> the problem.  If we could test that, that would be great !!!
>>> 
>>> We've made some big changes to rapath().  rapath() extracts
>>> topology information from argus data.  Basically it takes all
>>> data that has ICMP TXD messages mapped to it, and tabulates path
>>> information where it can.  This has the effect of capturing all
>>> traceroutes() that are observed by argus, regardless of the
>>> techniqu;  UDP, TCP or ICMP based, weather its vanilla or paris method,
>>> or several of the proprietary strategies seen in intrusions.
>>> 
>>> We've changed the default output of the graph that rapath.1
>>> generates (using the -A option) to include the srcid, saddr
>>> and daddr, so that you can build topology from just the
>>> graphs.  I'll add the stime and duration as well, but need
>>> to figure out some command line options to control all these
>>> new fields.  Also rapath() is going to get a realtime mode,
>>> currently, its a " read a file, generate some output " type of
>>> tool.
>>> 
>>> Please grab this code and give it a run.  I'm hoping to
>>> release 3.0.7.x as 3.0.8 in the next month, so if there are
>>> any gotchas, don't hold back.
>>> 
>>> Carter
>>> 
>> <rasqlinsert-Dump.txt><rasqlinsertLDD.txt>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7322 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130821/1cd179f0/attachment.bin>

More information about the argus mailing list