new argus-clients-3.0.7.14 on the server
Carter Bullard
carter at qosient.com
Sun Aug 25 12:49:25 EDT 2013
Cool,
So its now completely working for you !????!
The point of the code we're tweaking is for the SQL thread to
wait until the Curses thread is done, as the Curses thread is
managing the local " view " of what is suppose to be in the
database table. Once the Curses thread is done, the SQL thread
can then process its execution queue, and be happy to know that
nothing else is coming down the pipe.
So, now that we've got all the pthread_join()s in the right place,
doesn't look like we need the pthread_cond_signal() or the
pthread_cond_wait(). Your fix basically removes the wait.
OK, so run with your fix, which should be logically equivalent to
just taking these out. I'll run it in curses mode for a while to
see if it doesn't break something.
Thanks for all the effort and patience !!!!
Carter
On Aug 24, 2013, at 11:30 PM, David Edelman <dedelman at iname.com> wrote:
> I made the following change and it seems to work:
>
> | if (ArgusParser->RaCursesMode) {
>
> pthread_mutex_lock(&ArgusSQLQueryList->lock);
> pthread_cond_wait(&ArgusSQLQueryList->cond, &ArgusSQLQueryList->lock);
> pthread_mutex_unlock(&ArgusSQLQueryList->lock);
> | }
>
> --Dave
>
> -----Original Message-----
> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
> Behalf Of David Edelman
> Sent: Saturday, August 24, 2013 10:56 PM
> To: 'Carter Bullard'
> Cc: 'Argus'
> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>
> Carter,
>
> I think that rasqlinsert is getting stuck in this condition:
>
> The process is waiting on the pthread_join(RaMySQLThread) which to me
> indicates that RaData and RaCurses have terminated and that the only threads
> that are still alive are the mainline and RaMySQL. This is confirmed by info
> threads which show the two threads and no others.
>
> if ((pthread_create(&RaCursesThread, NULL, ArgusCursesProcess, NULL))
> != 0)
> ArgusLog (LOG_ERR, "ArgusCursesProcess() pthread_create error
> %s\n", strerror(errno));
>
> if ((pthread_create(&RaDataThread, NULL, ArgusProcessData, NULL)) !=
> 0)
> ArgusLog (LOG_ERR, "main() pthread_create error %s\n",
> strerror(errno));
>
> if ((pthread_join(RaDataThread, NULL)) != 0)
> if ((pthread_join(RaCursesThread, NULL)) != 0)
> if ((pthread_join(RaMySQLThread, NULL)) != 0)
>
> When I look at the code for the RaMySQL thread, it seems to hang out in a
> loop checking the count of items on the ArgusSQLQueryList. As long as there
> are items on the list it processes them. This is confirmed by setting a
> display on the ->count element which does eventually work its way down to 0.
> So far, so good. When the count hits zero it executes this code:
>
> pthread_join(RaCursesThread, NULL);
>
> Since the RaCurseThread has exited (or the pthread_join in the mainline
> would not have been satisfied) this should not block, leading to the
> execution of this code
>
> pthread_mutex_lock(&ArgusSQLQueryList->lock);
> pthread_cond_wait(&ArgusSQLQueryList->cond, &ArgusSQLQueryList->lock);
> pthread_mutex_unlock(&ArgusSQLQueryList->lock)
>
> I just can't figure out what is going to send
> pthread_cond_signal(&ArgusSQLQueryList->cond) since that happens only in the
> RaCurses thread which is no longer running. I'm not sure I would call this a
> true deadlock (A waiting for B who is waiting for A) though that seems to be
> more of a distinction than a difference. Would it be reasonable to make this
> into a timed conditional wait?
>
> --Dave
>
>
>
>
>
>
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Friday, August 23, 2013 7:21 PM
> To: David Edelman
> Cc: 'Argus'
> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>
> Hey Dave,
> Yes, we have a bunch of threads that all want to stop at once,
> but we need them to exit in something of an order.
>
> If you ran this under gdb(), when it freezes and you hit Ctr-C,
> it should return to the debugger, and if you go through the
> threads again, we should see if we've got a deadlock, which
> it sounds like we have. A thread waiting for B thread, B thread
> waiting for C and C waiting for A, or something like it.
>
> Carter
>
> On Aug 23, 2013, at 5:45 PM, "David Edelman" <dedelman at iname.com> wrote:
>
>> I'm not sure which version of the source code you want me to modify. I
> just
>> downloaded a copy from the website and I think that this is the section in
>> question despite the differences in the line numbers:
>>
>> 3310 ArgusCloseDown = 1;
>> 3311 ArgusParser->RaParseDone = 1;
>> 3312
>> 3313 pthread_join(RaCursesInputThread, NULL);
>> 3314 pthread_cond_signal(&ArgusSQLQueryList->cond);
>> 3315 }
>>
>>
>> When I run it against a single hour of flow data it doesn't fault, but it
>> also doesn't seem to get to EOJ.
>>
>> When I run it against a file with a single file that contains a single
> flow
>> record in this is what I see:
>>
>> time rasqlinsert -M time 1d -r argusTestOne -M rmon -w
>> mysql://argus:argus@localhost/argus/YYmacAddrs_%Y_%m_%d -m srcid saddr
> smac
>> -s stime ltime srcid saddr smac -D 3 - ip
>> rasqlinsert[17007.40d7d96b757f0000]: 2013-08-23-21:31:58.054
>> Argusfinish_parse (0x2294ef8)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>> ArgusFilterCompile () done 0
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>> RaCursesNewProcess(0x6bcf8010) returns 0x2294d70
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>> RaCursesNewProcess(0x6bcf8010) returns 0x2294ea0
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>> RaCursesNewProcess(0x6bcf8010) returns 0x2294fd0
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.174 RaMySQLInit
> ()
>> RaSource (null) RaArchive (null) RaFormat (null)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.179
>> ArgusInitAddrtoname (0x7f756bcf8010, 0x0, 0x0)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.179
>> ArgusParseInit(0x7f756bcf8010, NULL)
>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.179
>> ArgusMySQLProcess() starting
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>> ArgusProcessData() starting
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>> ArgusParseInit(0x7f756bcf8010 0x7f756bc15010
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>> ArgusReadConnection(0x6bc15010, 1) returning 1
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>> RaProcessSplitOptions(YYmacAddrs_2013_08_19, 4096, 0x6bc15630): returns 0
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.187 generating
>> table CREATE table argus.YYmacAddrs_2013_08_19 (stime double(18,6)
> unsigned
>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>> record blob) ENGINE=MyISAM
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.458
>> ArgusCreateSQLSaveTable (YYmacAddrs_2013_08_19) returning
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.459
>> ArgusCloseInput(0x6bc15010) closing
>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.459
>> ArgusCloseInput(0x6bc15010) done
>> rasqlinsert[17006.00e70969757f0000]: 2013-08-23-21:31:58.480
>> ArgusScheduleSQLQuery (0x6bcf8010, 0x2294a40, 0x5c005340, INSERT INTO
>> argus.YYmacAddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376870402.927","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>> ac:31",...), 32) done
>> rasqlinsert[17006.00e70969757f0000]: 2013-08-23-21:31:58.480
>> ArgusScheduleSQLQuery (0x6bcf8010, 0x2294a40, 0x5c003fd0, INSERT INTO
>> argus.YYmacAddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376870402.927","108.50.164.35","50.93.246.64","f8:e4:fb:
>> 19:b6:84",...), 32) done
>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.480 ArgusSQLQuery
>> (INSERT INTO argus.YYmacAddrs_2013_08_19
>> (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376870402.927","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>> ac:31",...))
>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.480 ArgusSQLQuery
>> (INSERT INTO argus.YYmacAddrs_2013_08_19
>> (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376870402.927","108.50.164.35","50.93.246.64","f8:e4:fb:
>> 19:b6:84",...))
>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.517
>> ArgusMySQLProcess() done!
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusWindowClose () returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> RaParseComplete(caught signal 0)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517 ArgusShutDown
>> (0)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusWindowClose () returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> RaParseComplete(caught signal 0)
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusDeleteModeList () returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusDeleteFileList () returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
> ArgusDeleteList
>> (0x2293050, 4) returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
> ArgusDeleteList
>> (0x22930f0, 4) returning
>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusDeleteAggregator(0x7f756bcf8010, 0x2294a40) returned
>>
>> real 0m0.470s
>> user 0m0.014s
>> sys 0m0.004s
>>
>> When I add a second file which also contains a single flow record from
> later
>> that same day, I see this:
>>
>> [root at monolith 19]# time rasqlinsert -M time 1d -r /data2/test/argusTest*
>> -M rmon -w mysql://argus:argus@localhost/argus/QQmacA4ddrs_%Y_%m_%d -m
> srcid
>> saddr
>> smac -s stime ltime srcid saddr smac -D 3
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>> RaCursesNewProcess(0x2aee4010) returns 0x1854d90
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>> RaCursesNewProcess(0x2aee4010) returns 0x1854ec0
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>> RaCursesNewProcess(0x2aee4010) returns 0x1854ff0
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.707 RaMySQLInit
> ()
>> RaSource (null) RaArchive (null) RaFormat (null)
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.711
>> ArgusInitAddrtoname (0x7fe12aee4010, 0x0, 0x0)
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.711
>> ArgusParseInit(0x7fe12aee4010, NULL)
>> rasqlinsert[17130.00774329e17f0000]: 2013-08-23-21:37:41.711
>> ArgusMySQLProcess() starting
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.711
>> ArgusProcessData() starting
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>> ArgusParseInit(0x7fe12aee4010 0x7fe12ae01010
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>> ArgusReadConnection(0x2ae01010, 1) returning 1
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>> RaProcessSplitOptions(QQmacA4ddrs_2013_08_19, 4096, 0x2ae01630): returns 0
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.719 generating
>> table CREATE table argus.QQmacA4ddrs_2013_08_19 (stime double(18,6)
> unsigned
>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>> record blob) ENGINE=MyISAM
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.808
>> ArgusCreateSQLSaveTable (QQmacA4ddrs_2013_08_19) returning
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusCloseInput(0x2ae01010) closing
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusCloseInput(0x2ae01010) done
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusParseInit(0x7fe12aee4010 0x7fe12ad9f010
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusReadConnection(0x2ad9f010, 1) returning 1
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusCloseInput(0x2ad9f010) closing
>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>> ArgusCloseInput(0x2ad9f010) done
>> rasqlinsert[17130.00f7ff23e17f0000]: 2013-08-23-21:37:41.812
>> ArgusScheduleSQLQuery (0x2aee4010, 0x1854a60, 0x1c005340, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376874001.616","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>> ac:31",...), 32) done
>> rasqlinsert[17130.00f7ff23e17f0000]: 2013-08-23-21:37:41.812
>> ArgusScheduleSQLQuery (0x2aee4010, 0x1854a60, 0x1c003fd0, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376870400.000","1376874001.616","108.50.164.35","50.93.246.64","f8:e4:fb:
>> 19:b6:84",...), 32) done
>>
>> And I have to hit ^C to stop it.
>>
>> ^Crasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>> RaParseComplete(caught signal 2)
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568 ArgusShutDown
>> (2)
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>> ArgusWindowClose () returning
>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>> RaParseComplete(caught signal 2)
>>
>> A single file with 10 flow records also freezes:
>>
>> [root at monolith 19]# time rasqlinsert -M time 1d -r
>> /data2/test/argusTestThree -M rmon -w
>> mysql://argus:argus@localhost/argus/QQmacA4ddrs_%Y_%m_%d -m srcid sa
>> ddr smac -s stime ltime srcid saddr smac -D 3
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>> RaCursesNewProcess(0x4f8ce010) returns 0x2690d20
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>> RaCursesNewProcess(0x4f8ce010) returns 0x2690e50
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>> RaCursesNewProcess(0x4f8ce010) returns 0x2690f80
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.487 RaMySQLInit
> ()
>> RaSource (null) RaArchive (null) RaFormat (null)
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.492
>> ArgusInitAddrtoname (0x7f8f4f8ce010, 0x0, 0x0)
>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.492
>> ArgusParseInit(0x7f8f4f8ce010, NULL)
>> rasqlinsert[17197.0037e84d8f7f0000]: 2013-08-23-21:42:22.492
>> ArgusMySQLProcess() starting
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> ArgusProcessData() starting
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> ArgusReadConnection() read 16 bytes
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> ArgusReadConnection() read 112 bytes
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> ArgusParseInit(0x7f8f4f8ce010 0x7f8f4f7eb010
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> ArgusReadConnection(0x4f7eb010, 1) returning 1
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>> RaProcessSplitOptions(QQmacA4ddrs_2013_08_19, 4096, 0x4f7eb630): returns 0
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.500 generating
>> table CREATE table argus.QQmacA4ddrs_2013_08_19 (stime double(18,6)
> unsigned
>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>> record blob) ENGINE=MyISAM
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.589
>> ArgusCreateSQLSaveTable (QQmacA4ddrs_2013_08_19) returning
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.590
>> ArgusCloseInput(0x4f7eb010) closing
>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.590
>> ArgusCloseInput(0x4f7eb010) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40005340, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874004.586","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>> ac:31",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000d3d0, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.348","1376874004.586","108.50.164.35","10.1.1.10","00:1d:73:ff:
>> 5d:29",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40003fd0, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874001.616","108.50.164.35","50.93.246.64","f8:e4:fb:
>> 19:b6:84",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x400061f0, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874004.371","108.50.164.35","10.1.1.15","18:b4:30:01:
>> 86:8a",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40004a00, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874004.589","108.50.164.35","10.1.1.101","3c:07:54:5b
>> :be:b5",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40007090, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874004.589","108.50.164.35","216.17.8.6","f8:e4:fb:19
>> :b6:84",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40009900, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874001.014","108.50.164.35","50.16.200.78","f8:e4:fb:
>> 19:b6:84",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40003650, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874002.599","108.50.164.35","10.1.1.60","50:46:5d:6f:
>> d1:47",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000a700, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.000","1376874004.371","108.50.164.35","38.102.136.104","f8:e4:f
>> b:19:b6:84",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40007c40, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.002","1376874002.137","108.50.164.35","10.1.1.68","00:24:1d:87:
>> 53:03",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000b690, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.016","1376874000.017","108.50.164.35","00:26:f2:f2:56:bb","00:2
>> 6:f2:f2:56:bb",...), 32) done
>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x400090a0, INSERT INTO
>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>>
> ("1376874000.016","1376874000.017","108.50.164.35","01:80:c2:00:00:0e","01:8
>> 0:c2:00:00:0e",...), 32) done
>>
>> I hope that this helps, it look very close to a solution.
>>
>> --Dave
>>
>>
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Friday, August 23, 2013 10:38 AM
>> To: David Edelman
>> Cc: 'Argus'
>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>
>> Hey Dave,
>> Can you apply this patch to see if your rasqlinsert() doesn't get a little
>> better?
>>
>> osiris:ramysql carter$ diff rasqlinsert.c rasqlinsert.c.new
>> 3323c3323,3325
>> < pthread_join(RaCursesInputThread, NULL);
>> ---
>>> if (ArgusParser->RaCursesMode)
>>> pthread_join(RaCursesInputThread, NULL);
>>>
>>
>> Thanks !!!!!
>>
>> Carter
>>
>> On Aug 21, 2013, at 6:04 PM, "David Edelman" <dedelman at iname.com> wrote:
>>
>>> With the thread specific information:
>>>
>>> gdb rasqlinsert
>>> GNU gdb (GDB) Fedora (7.5.1-38.fc18)
>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>> Reading symbols from /usr/local/bin/rasqlinsert...done.
>>> (gdb) run -M time 1d -r * -M rmon -w
>>> mysql://argus:argus@localhost/argus/VVmacAddrs_%Y_%m_%d -m srcid saddr
>> smac
>>> -s stime ltime srcid saddr smac - ip
>>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r * -M rmon -w
>>> mysql://argus:argus@localhost/argus/VVmacAddrs_%Y_%m_%d -m srcid saddr
>> smac
>>> -s stime ltime srcid saddr smac - ip
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>> Detaching after fork from child process 2817.
>>> [New Thread 0x7ffff51a2700 (LWP 2820)]
>>> [Thread 0x7ffff51a2700 (LWP 2820) exited]
>>> [New Thread 0x7ffff51a2700 (LWP 2826)]
>>> [New Thread 0x7fffef7fe700 (LWP 2827)]
>>> [New Thread 0x7fffeeffd700 (LWP 2828)]
>>> Detaching after fork from child process 2829.
>>> Detaching after fork from child process 2831.
>>> Detaching after fork from child process 2833.
>>> Detaching after fork from child process 2837.
>>> Detaching after fork from child process 2839.
>>> Detaching after fork from child process 2841.
>>> Detaching after fork from child process 2843.
>>> Detaching after fork from child process 2845.
>>> Detaching after fork from child process 2847.
>>> Detaching after fork from child process 2849.
>>> Detaching after fork from child process 2851.
>>> Detaching after fork from child process 2853.
>>> Detaching after fork from child process 2855.
>>> Detaching after fork from child process 2857.
>>> Detaching after fork from child process 2859.
>>> Detaching after fork from child process 2861.
>>> Detaching after fork from child process 2863.
>>> Detaching after fork from child process 2865.
>>> Detaching after fork from child process 2867.
>>> Detaching after fork from child process 2869.
>>> Detaching after fork from child process 2872.
>>> Detaching after fork from child process 2874.
>>> Detaching after fork from child process 2876.
>>> Detaching after fork from child process 2878.
>>> [Thread 0x7fffeeffd700 (LWP 2828) exited]
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> [Switching to Thread 0x7fffef7fe700 (LWP 2827)]
>>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>> Missing separate debuginfos, use: debuginfo-install
>>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>>> ncurses-libs-5.9-11.20130511.fc18.x86_64
> openssl-libs-1.0.1e-4.fc18.x86_64
>>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>>> (gdb) where
>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>> ./rasqlinsert.c:3313
>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>> ./rasqlinsert.c:450
>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>
>>> (gdb) info threads
>>> Id Target Id Frame
>>> * 4 Thread 0x7fffef7fe700 (LWP 2827) "rasqlinsert" 0x00007ffff7570dcb
>> in
>>> pthread_join () from /lib64/libpthread.so.0
>>> 3 Thread 0x7ffff51a2700 (LWP 2826) "rasqlinsert" 0x00007ffff75764b7 in
>>> recv () from /lib64/libpthread.so.0
>>> 1 Thread 0x7ffff7560740 (LWP 2813) "rasqlinsert" 0x00007ffff7570e60 in
>>> pthread_join () from /lib64/libpthread.so.0
>>>
>>> (gdb) thread 1
>>> [Switching to thread 1 (Thread 0x7ffff7560740 (LWP 2813))]
>>> #0 0x00007ffff7570e60 in pthread_join () from /lib64/libpthread.so.0
>>> (gdb) where
>>> #0 0x00007ffff7570e60 in pthread_join () from /lib64/libpthread.so.0
>>> #1 0x000000000040cb6b in main (argc=45, argv=0x7fffffffdc38) at
>>> ./rasqlinsert.c:302
>>>
>>> (gdb) thread 3
>>> [Switching to thread 3 (Thread 0x7ffff51a2700 (LWP 2826))]
>>> #0 0x00007ffff75764b7 in recv () from /lib64/libpthread.so.0
>>> (gdb) where
>>> #0 0x00007ffff75764b7 in recv () from /lib64/libpthread.so.0
>>> #1 0x00007ffff7a71299 in inline_mysql_socket_recv (flags=<optimized
> out>,
>>> n=<optimized out>, buf=<optimized out>, mysql_socket=...,
>>> src_line=<optimized out>,
>>> src_file=<optimized out>) at /usr/include/bits/socket2.h:35
>>> #2 vio_read (vio=0x100e580, buf=0x100e770 "0", size=16384) at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/vi
>>> o/viosocket.c:114
>>> #3 0x00007ffff7a7131e in vio_read_buff (vio=0x7, buf=0x1012780 "\321\v",
>>> size=4)
>>> at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/vi
>>> o/viosocket.c:157
>>> #4 0x00007ffff7a566a7 in net_read_raw_loop (count=4, net=<optimized
> out>)
>>> at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l/net_serv.cc:663
>>> #5 net_read_packet_header (net=<optimized out>) at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l/net_serv.cc:751
>>> #6 net_read_packet (net=0x100aa60, complen=0x7ffff51a1ca0) at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l/net_serv.cc:810
>>> #7 0x00007ffff7a569af in my_net_read (net=0x7) at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l/net_serv.cc:888
>>> #8 0x00007ffff7a4d7b6 in cli_safe_read (mysql=0x7) at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l-common/client.c:610
>>> #9 0x00007ffff7a50257 in cli_read_query_result (mysql=0x7)
>>> at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l-common/client.c:4015
>>> #10 0x00007ffff7a4c9a6 in mysql_real_query (mysql=0x100aa60,
>>> query=<optimized out>, length=<optimized out>)
>>> at
>>>
>>
> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>> l-common/client.c:4103
>>> #11 0x000000000041e0ef in ArgusProcessSQLQueryList
> (parser=0x7ffff74bb010)
>>> at ./rasqlinsert.c:6556
>>> #12 0x000000000041e674 in ArgusMySQLProcess (arg=0x7ffff74bb010) at
>>> ./rasqlinsert.c:6694
>>> #13 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>> #14 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>
>>> (gdb) thread 4
>>> [Switching to thread 4 (Thread 0x7fffef7fe700 (LWP 2827))]
>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>> (gdb) where
>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>> ./rasqlinsert.c:3313
>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>> ./rasqlinsert.c:450
>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Carter Bullard [mailto:carter at qosient.com]
>>> Sent: Wednesday, August 21, 2013 2:38 PM
>>> To: David Edelman
>>> Cc: 'Argus'
>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>
>>> When you are debugging rasqlinsert(), there are a
>>> number of threads, and so after printing " where ",
>>> you can also do a " info threads ", and then for
>>> each thread, type " thread x " where x is the thread
>>> number, and do a where once in each thread.
>>>
>>> (gdb) info threads
>>> (gdb) thread 1
>>> (gdb) where
>>> (gdb) thread 2
>>> (where)
>>>
>>> etc . That can be very useful.
>>> Still working on it.
>>>
>>> Carter
>>>
>>>
>>>
>>> On Aug 21, 2013, at 2:17 PM, "David Edelman" <dedelman at iname.com> wrote:
>>>
>>>> It is getting better - that's purely subjective It still faults but I
> was
>>>> able to run through a directory tree with -R for quite a while before
> the
>>>> fault. I still had the same problem that I always had where only the
>> first
>>>> table is populated the rest are created but empty. I've been doing
> nested
>>>> bash for loops to deal with that for quite some time.
>>>>
>>>> I ran against a single flow record file under gdb and this is the
> result.
>>
>>>>
>>>> --Dave
>>>>
>>>>
>>>>
>>>> (gdb) run -M time 1d -r argus.2013.04.06.00.00.01.0.gz -M rmon -w
>>>> mysql://argus:argus@localhost/argus/TTmacAddrs_%Y_%m_%d -m srcid saddr
>>> smac
>>>> -s stime ltime srcid saddr smac - ip
>>>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r
>>>> argus.2013.04.06.00.00.01.0.gz -M rmon -w
>>>> mysql://argus:argus@localhost/argus/TTmacAddrs_%Y_%m_%d -m srcid saddr
>>> smac
>>>> -s stime ltime srcid saddr smac - ip
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>> Detaching after fork from child process 31440.
>>>> [New Thread 0x7ffff5a70700 (LWP 31441)]
>>>> [Thread 0x7ffff5a70700 (LWP 31441) exited]
>>>> [New Thread 0x7ffff5a70700 (LWP 31442)]
>>>> [New Thread 0x7ffff4861700 (LWP 31443)]
>>>> [New Thread 0x7fffeffff700 (LWP 31444)]
>>>> Detaching after fork from child process 31445.
>>>> [Thread 0x7fffeffff700 (LWP 31444) exited]
>>>>
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> [Switching to Thread 0x7ffff4861700 (LWP 31443)]
>>>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>>>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>>>> ncurses-libs-5.9-11.20130511.fc18.x86_64
>> openssl-libs-1.0.1e-4.fc18.x86_64
>>>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>>>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>>>> (gdb) where
>>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>> ./rasqlinsert.c:3313
>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>> ./rasqlinsert.c:450
>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>> (gdb) list
>>>> 230 int ArgusColorAddresses(struct ArgusParserStruct *, struct
>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>> 231 int ArgusColorFlowFields(struct ArgusParserStruct *, struct
>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>> 232 int ArgusColorGeoLocation(struct ArgusParserStruct *, struct
>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>> 233 void ArgusInitializeColorMap(struct ArgusParserStruct *, WINDOW *);
>>>> 234 #endif
>>>> 235
>>>> 236 int
>>>> 237 main(int argc, char **argv)
>>>> 238 {
>>>> 239 struct ArgusParserStruct *parser = NULL;
>>>> (gdb) up
>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>> ./rasqlinsert.c:3313
>>>> 3313 pthread_join(RaCursesInputThread, NULL);
>>>> (gdb) up
>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>> ./rasqlinsert.c:450
>>>> 450 ArgusCursesProcessClose();
>>>> (gdb) up
>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>> (gdb) up
>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>> (gdb) up
>>>> Initial frame selected; you cannot go up.
>>>> (gdb)
>>>>
>>>>
>>>> dmesg shows:
>>>>
>>>> [150781.353120] rasqlinsert[26482]: segfault at 2d0 ip 00007ff53b7e6dcb
>> sp
>>>> 00007ff533ffdda0 error 4 in libpthread-2.16.so[7ff53b7de000+16000]
>>>> [150980.758765] rasqlinsert[31192]: segfault at 2d0 ip 00007fa5bfab3dcb
>> sp
>>>> 00007fa5b7ffdda0 error 4 in libpthread-2.16.so[7fa5bfaab000+16000]
>>>> [151147.481928] rasqlinsert[31303]: segfault at 2d0 ip 00007f95d0346dcb
>> sp
>>>> 00007f95ccd67da0 error 4 in libpthread-2.16.so[7f95d033e000+16000]
>>>> [151244.434895] rasqlinsert[31402]: segfault at 2d0 ip 00007f071442fdcb
>> sp
>>>> 00007f071171eda0 error 4 in libpthread-2.16.so[7f0714427000+16000]
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>> Sent: Wednesday, August 21, 2013 11:31 AM
>>>> To: David Edelman
>>>> Cc: 'Argus'
>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>
>>>> I think I've found a knob to turn that might help. Can you try this
>>> change?
>>>> This
>>>> controls how aggressively we will push modifications out to the
> database.
>>
>>>>
>>>> ==== //depot/argus/clients/examples/ramysql/raclient.c#8 -
>>>> /Users/carter/argus/clients/examples/ramysql/raclient.c ====
>>>> 810c810
>>>> < RaCursesUpdateInterval.tv_sec = 0;
>>>> ---
>>>>> RaCursesUpdateInterval.tv_sec = 1;
>>>>
>>>> You may still get a problem, but it should be better. I'll keep looking
>>>> into
>>>> a better fix that this, but any improvement means where in the right
>>>> direction.
>>>>
>>>> Carter
>>>>
>>>> On Aug 21, 2013, at 10:26 AM, "David Edelman" <dedelman at iname.com>
> wrote:
>>>>
>>>>> Carter,
>>>>>
>>>>> That's great news and it makes sense. The instances that are processing
>>>>> stream data don't have enough traffic to create huge updates, and the
>>>>> instances that are cruising through files surely do. I have 128GB of
>>>>> physical memory and I've tuned MySQL to use as much as possible. I can
>>>>> detune it and see if that makes a difference.
>>>>>
>>>>> --Dave
>>>>>
>>>>> -----Original Message-----
>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>> Sent: Wednesday, August 21, 2013 8:10 AM
>>>>> To: David Edelman
>>>>> Cc: 'Argus'
>>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>
>>>>> Hey David,
>>>>> I'm also getting the same error here, with your example. Not sure
> where
>>>>> this crept in, as I use rasqlinsert() on many systems and I haven't
> seen
>>>>> this,
>>>>> although I have started to see rasqlinsert()s that eat a lot of CPU.
>>>>>
>>>>> This is a mysqld problem, where it coughs blood processing packets that
>>>>> are too large, or where multiple threads are making concurrent
>>>>> sql calls on the same socket.
>>>>>
>>>>> http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
>>>>>
>>>>> We do try to send the largest query possible by packing as many
> INSERTS,
>>>>> or UPDATES, into the mysql ' max_allowed_packet ', and we have multiple
>>>>> threads talking to mysqld. I just now tried to cut the
>>> max_allowed_packet
>>>>> buffer in 1/2, with no effect, so ..... I've got some work to do here
>> and
>>>>> will
>>>>> try to have something today.
>>>>>
>>>>> Thanks for the feedback !!!!
>>>>>
>>>>> Carter
>>>>>
>>>>> On Aug 20, 2013, at 7:43 PM, David Edelman <dedelman at iname.com> wrote:
>>>>>
>>>>>> I had a theory that the problem happened when there were no more
>> records
>>>>>> available to be read. To test this I moved up one level in the data
>>>> source
>>>>>> directory tree and used -R * figuring that I should see good tables
> for
>>>>> the
>>>>>> dates prior to the most recent. The theory may be good but the test
> may
>>>>> have
>>>>>> been useful for a different reason. Newest client build with .devel
> and
>>>>>> .debug run without a -D switch. I get this almost immediately (but
> this
>>>> is
>>>>> a
>>>>>> big machine with lots of memory so speed is a hard thing to judge.)
>>>>>>
>>>>>>
>>>>>> rasqlinsert -M time 1d -R * -M rmon -w
>>>>>> mysql://argus:argus@localhost/argus/SSmacAddrs_%Y_%m_%d -m srcid saddr
>>>>> smac
>>>>>> -s stime ltime srcid saddr smac - ip
>>>>>> *** glibc detected *** rasqlinsert: double free or corruption (!prev):
>>>>>> 0x00000000020e4830 ***
>>>>>> rasqlinsert[8994]: 2013-08-20-23:27:39.017 mysql_real_query error Lost
>>>>>> connection to MySQL server during query
>>>>>> ======= Backtrace: =========
>>>>>> /lib64/libc.so.6[0x3c88e7cb3e]
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(vio_delete+0x26)[0x7f94870c5416]
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(end_server+0x38)[0x7f94870a2678]
>>>>>>
>>> /usr/lib64/mysql/libmysqlclient.so.18(cli_safe_read+0x30)[0x7f94870a2770]
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(+0x41257)[0x7f94870a5257]
>>>>>>
>>>>>
>>>>
>>>
>>
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_real_query+0x26)[0x7f94870a19a6]
>>>>>>
>>>>>
>>>>
>>>
>>
> /usr/lib64/mysql/libmysqlclient.so.18(mysql_list_tables+0x55)[0x7f948709edb5
>>>>>> ]
>>>>>> rasqlinsert[0x41fe3c]
>>>>>> rasqlinsert[0x408c68]
>>>>>> rasqlinsert[0x4086a5]
>>>>>> rasqlinsert[0x438d35]
>>>>>> rasqlinsert[0x43933e]
>>>>>> rasqlinsert[0x47b12a]
>>>>>> rasqlinsert[0x47b335]
>>>>>> rasqlinsert[0x405be7]
>>>>>> /lib64/libpthread.so.0(+0x3c89207d15)[0x7f9486bc4d15]
>>>>>> /lib64/libc.so.6(clone+0x6d)[0x3c88ef253d]
>>>>>> ======= Memory map: ========
>>>>>> 00400000-004eb000 r-xp 00000000 fd:00 20983991
>>>>>> /usr/local/bin/rasqlinsert
>>>>>> 006ea000-006eb000 r--p 000ea000 fd:00 20983991
>>>>>> /usr/local/bin/rasqlinsert
>>>>>> 006eb000-006fb000 rw-p 000eb000 fd:00 20983991
>>>>>> /usr/local/bin/rasqlinsert
>>>>>> 006fb000-01007000 rw-p 00000000 00:00 0
>>>>>> 020d8000-021c0000 rw-p 00000000 00:00 0
>>>>>> [heap]
>>>>>> 3c88a00000-3c88a20000 r-xp 00000000 fd:00 77463577
>>>>>> /usr/lib64/ld-2.16.so
>>>>>> 3c88c20000-3c88c21000 r--p 00020000 fd:00 77463577
>>>>>> /usr/lib64/ld-2.16.so
>>>>>> 3c88c21000-3c88c22000 rw-p 00021000 fd:00 77463577
>>>>>> /usr/lib64/ld-2.16.so
>>>>>> 3c88c22000-3c88c23000 rw-p 00000000 00:00 0
>>>>>> 3c88e00000-3c88fad000 r-xp 00000000 fd:00 77463656
>>>>>> /usr/lib64/libc-2.16.so
>>>>>> 3c88fad000-3c891ad000 ---p 001ad000 fd:00 77463656
>>>>>> /usr/lib64/libc-2.16.so
>>>>>> 3c891ad000-3c891b1000 r--p 001ad000 fd:00 77463656
>>>>>> /usr/lib64/libc-2.16.so
>>>>>> 3c891b1000-3c891b3000 rw-p 001b1000 fd:00 77463656
>>>>>> /usr/lib64/libc-2.16.so
>>>>>> 3c891b3000-3c891b8000 rw-p 00000000 00:00 0
>>>>>> 3c89200000-3c89202000 r-xp 00000000 fd:00 77464193
>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>> 3c89202000-3c89401000 ---p 00002000 fd:00 77464193
>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>> 3c89401000-3c89402000 r--p 00001000 fd:00 77464193
>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>> 3c89402000-3c89403000 rw-p 00002000 fd:00 77464193
>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>> 3c89600000-3c89603000 r-xp 00000000 fd:00 77471354
>>>>>> /usr/lib64/libdl-2.16.so
>>>>>> 3c89603000-3c89802000 ---p 00003000 fd:00 77471354
>>>>>> /usr/lib64/libdl-2.16.so
>>>>>> 3c89802000-3c89803000 r--p 00002000 fd:00 77471354
>>>>>> /usr/lib64/libdl-2.16.so
>>>>>> 3c89803000-3c89804000 rw-p 00003000 fd:00 77471354
>>>>>> /usr/lib64/libdl-2.16.so
>>>>>> 3c89a00000-3c89a07000 r-xp 00000000 fd:00 77463871
>>>>>> /usr/lib64/librt-2.16.so
>>>>>> 3c89a07000-3c89c06000 ---p 00007000 fd:00 77463871
>>>>>> /usr/lib64/librt-2.16.so
>>>>>> 3c89c06000-3c89c07000 r--p 00006000 fd:00 77463871
>>>>>> /usr/lib64/librt-2.16.so
>>>>>> 3c89c07000-3c89c08000 rw-p 00007000 fd:00 77463871
>>>>>> /usr/lib64/librt-2.16.so
>>>>>> 3c89e00000-3c89e15000 r-xp 00000000 fd:00 77471658
>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>> 3c89e15000-3c8a014000 ---p 00015000 fd:00 77471658
>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>> 3c8a014000-3c8a015000 r--p 00014000 fd:00 77471658
>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>> 3c8a015000-3c8a016000 rw-p 00015000 fd:00 77471658
>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>> 3c8a200000-3c8a300000 r-xp 00000000 fd:00 77471671
>>>>>> /usr/lib64/libm-2.16.so
>>>>>> 3c8a300000-3c8a4ff000 ---p 00100000 fd:00 77471671
>>>>>> /usr/lib64/libm-2.16.so
>>>>>> 3c8a4ff000-3c8a500000 r--p 000ff000 fd:00 77471671
>>>>>> /usr/lib64/libm-2.16.so
>>>>>> 3c8a500000-3c8a501000 rw-p 00100000 fd:00 77471671
>>>>>> /usr/lib64/libm-2.16.so
>>>>>> 3c8a600000-3c8a615000 r-xp 00000000 fd:00 77471694
>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>> 3c8a615000-3c8a814000 ---p 00015000 fd:00 77471694
>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>> 3c8a814000-3c8a815000 r--p 00014000 fd:00 77471694
>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>> 3c8a815000-3c8a816000 rw-p 00015000 fd:00 77471694
>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>> 3c8aa00000-3c8aa5c000 r-xp 00000000 fd:00 77471660
>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>> 3c8aa5c000-3c8ac5c000 ---p 0005c000 fd:00 77471660
>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>> 3c8ac5c000-3c8ac5d000 r--p 0005c000 fd:00 77471660
>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>> 3c8ac5d000-3c8ac5e000 rw-p 0005d000 fd:00 77471660
>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>> 3c8b600000-3c8b63c000 r-xp 00000000 fd:00 77465274
>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>> 3c8b63c000-3c8b83b000 ---p 0003c000 fd:00 77465274
>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>> 3c8b83b000-3c8b83d000 r--p 0003b000 fd:00 77465274
>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>> 3c8b83d000-3c8b843000 rw-p 0003d000 fd:00 77465274
>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>> 3c8b843000-3c8b845000 rw-p 00000000 00:00 0
>>>>>> 3c8ca00000-3c8cae5000 r-xp 00000000 fd:00 77471735
>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>> 3c8cae5000-3c8cce4000 ---p 000e5000 fd:00 77471735
>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>> 3c8cce4000-3c8ccec000 r--p 000e4000 fd:00 77471735
>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>> 3c8ccec000-3c8ccee000 rw-p 000ec000 fd:00 77471735
>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>> 3c8ccee000-3c8cd03000 rw-p 00000000 00:00 0
>>>>>> 3c93a00000-3c93b9b000 r-xp 00000000 fd:00 77471858
>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>> 3c93b9b000-3c93d9b000 ---p 0019b000 fd:00 77471858
>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>> 3c93d9b000-3c93db5000 r--p 0019b000 fd:00 77471858
>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>> 3c93db5000-3c93dc0000 rw-p 001b5000 fd:00 77471858
>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>> 3c93dc0000-3c93dc5000 rw-p 00000000 00:00 0
>>>>>> 3ca1200000-3ca1225000 r-xp 00000000 fd:00 77471104
>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>> 3ca1225000-3ca1424000 ---p 00025000 fd:00 77471104
>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>> 3ca1424000-3ca1428000 r--p 00024000 fd:00 77471104
>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>> 3ca1428000-3ca1429000 rw-p 00028000 fd:00 77471104
>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>> 3ca2e00000-3ca2e23000 r-xp 00000000 fd:00 77471813
>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>> 3ca2e23000-3ca3022000 ---p 00023000 fd:00 77471813
>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>> 3ca3022000-3ca3023000 r--p 00022000 fd:00 77471813
>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>> 3ca3023000-3ca3024000 rw-p 00023000 fd:00 77471813
>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>> 3ca3e00000-3ca3e16000 r-xp 00000000 fd:00 77470820
>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>> 3ca3e16000-3ca4015000 ---p 00016000 fd:00 77470820
>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>> 3ca4015000-3ca4016000 r--p 00015000 fd:00 77470820
>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>> 3ca4016000-3ca4017000 rw-p 00016000 fd:00 77470820
>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>> 3ca4017000-3ca4019000 rw-p 00000000 00:00 0
>>>>>> 3ca7600000-3ca7609000 r-xp 00000000 fd:00 77472018
>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>> 3ca7609000-3ca7808000 ---p 00009000 fd:00 77472018
>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>> 3ca7808000-3ca7809000 r--p 00008000 fd:00 77472018
>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>> 3ca7809000-3ca780a000 rw-p 00009000 fd:00 77472018
>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>> 3ca780a000-3ca780b000 rw-p 00000000 00:00 0
>>>>>> 7f9470000000-7f9470594000 rw-p 00000000 00:00 0
>>>>>> 7f9470594000-7f9474000000 ---p 00000000 00:00 0
>>>>>> 7f9474000000-7f9474021000 rw-p 00000000 00:00 0
>>>>>> 7f9474021000-7f9478000000 ---p 00000000 00:00 0
>>>>>> 7f9478000000-7f94782a5000 rw-p 00000000 00:00 0
>>>>>> 7f94782a5000-7f947c000000 ---p 00000000 00:00 0
>>>>>> 7f947c943000-7f947c944000 ---p 00000000 00:00 0
>>>>>> 7f947c944000-7f947d144000 rw-p 00000000 00:00 0
>>>>>> [stack:8999]
>>>>>> 7f947d144000-7f947d145000 ---p 00000000 00:00 0
>>>>>> 7f947d145000-7f947e146000 rw-p 00000000 00:00 0
>>>>>> [stack:8998]
>>>>>> 7f947e146000-7f947e152000 r-xp 00000000 fd:00 77471291
>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>> 7f947e152000-7f947e351000 ---p 0000c000 fd:00 77471291
>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>> 7f947e351000-7f947e352000 r--p 0000b000 fd:00 77471291
>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>> 7f947e352000-7f947e353000 rw-p 0000c000 fd:00 77471291
>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>> 7f947e353000-7f947e354000 ---p 00000000 00:00 0
>>>>>> 7f947e354000-7f9486bbd000 rw-p 00000000 00:00 0
>>>>>> [stack:8997]
>>>>>> 7f9486bbd000-7f9486bd3000 r-xp 00000000 fd:00 77463716
>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>> 7f9486bd3000-7f9486dd3000 ---p 00016000 fd:00 77463716
>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>> 7f9486dd3000-7f9486dd4000 r--p 00016000 fd:00 77463716
>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>> 7f9486dd4000-7f9486dd5000 rw-p 00017000 fd:00 77463716
>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>> 7f9486dd5000-7f9486dda000 rw-p 00000000 00:00 0
>>>>>> 7f9486dda000-7f9486e60000 r-xp 00000000 fd:00 77470840
>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>> 7f9486e60000-7f948705f000 ---p 00086000 fd:00 77470840
>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>> 7f948705f000-7f9487061000 r--p 00085000 fd:00 77470840
>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>> 7f9487061000-7f9487064000 rw-p 00087000 fd:00 77470840
>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>> 7f9487064000-7f948735a000 r-xp 00000000 fd:00 77599545
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>> 7f948735a000-7f948755a000 ---p 002f6000 fd:00 77599545
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>> 7f948755a000-7f948762c000 rw-p 002f6000 fd:00 77599545
>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>> 7f948762c000-7f9487632000 rw-p 00000000 00:00 0
>>>>>> 7f948764f000-7f9487652000 rw-p 00000000 00:00 0
>>>>>> 7fff16923000-7fff16944000 rw-p 00000000 00:00 0
>>>>>> [stack]
>>>>>> 7fff169fe000-7fff16a00000 r-xp 00000000 00:00 0
>>>>>> [vdso]
>>>>>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>>>>>> [vsyscall]
>>>>>> Aborted (core dumped)
>>>>>>
>>>>>> I run ldd against the image and I get this:
>>>>>>
>>>>>> ldd /usr/local/bin/rasqlinsert
>>>>>> linux-vdso.so.1 => (0x00007fffaebfe000)
>>>>>> libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x0000003c89200000)
>>>>>> libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c8aa00000)
>>>>>> libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
>>>>>> (0x00007f3f8c855000)
>>>>>> libm.so.6 => /lib64/libm.so.6 (0x0000003c8a200000)
>>>>>> libft.so.0 => /lib64/libft.so.0 (0x00007f3f8c5cb000)
>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8c3ae000)
>>>>>> libz.so.1 => /lib64/libz.so.1 (0x0000003c89e00000)
>>>>>> libncurses.so.5 => /lib64/libncurses.so.5 (0x0000003ca2e00000)
>>>>>> libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003ca1200000)
>>>>>> libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003c8b600000)
>>>>>> libc.so.6 => /lib64/libc.so.6 (0x0000003c88e00000)
>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x0000003c89600000)
>>>>>> librt.so.1 => /lib64/librt.so.1 (0x0000003c89a00000)
>>>>>> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000003c8ca00000)
>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c8a600000)
>>>>>> libcrypto.so.10 => /lib64/libcrypto.so.10 (0x0000003c93a00000)
>>>>>> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003ca3e00000)
>>>>>> libwrap.so.0 => /lib64/libwrap.so.0 (0x0000003ca7600000)
>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003c88a00000)
>>>>>>
>>>>>> Checking the dates of the dynamic libraries indicates that I am
> linking
>>>> to
>>>>>> the most up to date versions.
>>>>>>
>>>>>> --Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>>> Sent: Tuesday, August 20, 2013 7:17 PM
>>>>>> To: David Edelman
>>>>>> Cc: Argus
>>>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>>
>>>>>> Hmmmm, I'll take another look tonight. It was working here with your
>>>>>> file...frustrating !!!
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>> On Aug 20, 2013, at 6:15 PM, "David Edelman" <dedelman at iname.com>
>> wrote:
>>>>>>
>>>>>>> Carter,
>>>>>>>
>>>>>>> I'm having the exact same problem as before.
>>>>>>>
>>>>>>> I did a clean install after changing the string in VERSION so that I
>>>> knew
>>>>>>> that I was using new code. I applied the argus_label.c change which
>>>>> didn't
>>>>>>> make any difference. I created .debug and .devel; make clobber,
>>>> ./config;
>>>>>>> make; make install and ran under gdb and it is the same picture.
>>>>>>>
>>>>>>> The instances of rasqlinsert taking data from radium are as happy as
>>>>>> clams.
>>>>>>>
>>>>>>> What additional material can I collect for you?
>>>>>>>
>>>>>>> --Dave
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>>>>>>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu]
> On
>>>>>>> Behalf Of Carter Bullard
>>>>>>> Sent: Tuesday, August 20, 2013 10:59 AM
>>>>>>> To: Argus
>>>>>>> Subject: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>>>
>>>>>>> Gentle people,
>>>>>>> New client code up on the server. This release fixes all
>>>>>>> known bugs that has been reported on the list, as well as
>>>>>>> having major modifications to rapath().
>>>>>>>
>>>>>>> New code has been added as guards around the reported
>>>>>>> label problems, but I am not sure that it has fixed
>>>>>>> the problem. If we could test that, that would be great !!!
>>>>>>>
>>>>>>> We've made some big changes to rapath(). rapath() extracts
>>>>>>> topology information from argus data. Basically it takes all
>>>>>>> data that has ICMP TXD messages mapped to it, and tabulates path
>>>>>>> information where it can. This has the effect of capturing all
>>>>>>> traceroutes() that are observed by argus, regardless of the
>>>>>>> techniqu; UDP, TCP or ICMP based, weather its vanilla or paris
>> method,
>>>>>>> or several of the proprietary strategies seen in intrusions.
>>>>>>>
>>>>>>> We've changed the default output of the graph that rapath.1
>>>>>>> generates (using the -A option) to include the srcid, saddr
>>>>>>> and daddr, so that you can build topology from just the
>>>>>>> graphs. I'll add the stime and duration as well, but need
>>>>>>> to figure out some command line options to control all these
>>>>>>> new fields. Also rapath() is going to get a realtime mode,
>>>>>>> currently, its a " read a file, generate some output " type of
>>>>>>> tool.
>>>>>>>
>>>>>>> Please grab this code and give it a run. I'm hoping to
>>>>>>> release 3.0.7.x as 3.0.8 in the next month, so if there are
>>>>>>> any gotchas, don't hold back.
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>> <rasqlinsert-Dump.txt><rasqlinsertLDD.txt>
>>>>>
>>>>
>>>> <rasqlinsert-gdb.txt>
>>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7322 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130825/9b77f1c6/attachment.bin>
More information about the argus
mailing list