new argus-clients-3.0.7.14 on the server
David Edelman
dedelman at iname.com
Sun Aug 25 18:46:19 EDT 2013
So far, so good. I'm processing three month's data and I've seen one segfault and a few duplicate primary keys.
Dave Edelman
On Aug 25, 2013, at 12:49, Carter Bullard <carter at qosient.com> wrote:
> Cool,
> So its now completely working for you !????!
>
> The point of the code we're tweaking is for the SQL thread to
> wait until the Curses thread is done, as the Curses thread is
> managing the local " view " of what is suppose to be in the
> database table. Once the Curses thread is done, the SQL thread
> can then process its execution queue, and be happy to know that
> nothing else is coming down the pipe.
>
> So, now that we've got all the pthread_join()s in the right place,
> doesn't look like we need the pthread_cond_signal() or the
> pthread_cond_wait(). Your fix basically removes the wait.
>
> OK, so run with your fix, which should be logically equivalent to
> just taking these out. I'll run it in curses mode for a while to
> see if it doesn't break something.
>
> Thanks for all the effort and patience !!!!
>
> Carter
>
>
>
> On Aug 24, 2013, at 11:30 PM, David Edelman <dedelman at iname.com> wrote:
>
>> I made the following change and it seems to work:
>>
>> | if (ArgusParser->RaCursesMode) {
>>
>> pthread_mutex_lock(&ArgusSQLQueryList->lock);
>> pthread_cond_wait(&ArgusSQLQueryList->cond, &ArgusSQLQueryList->lock);
>> pthread_mutex_unlock(&ArgusSQLQueryList->lock);
>> | }
>>
>> --Dave
>>
>> -----Original Message-----
>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
>> Behalf Of David Edelman
>> Sent: Saturday, August 24, 2013 10:56 PM
>> To: 'Carter Bullard'
>> Cc: 'Argus'
>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>
>> Carter,
>>
>> I think that rasqlinsert is getting stuck in this condition:
>>
>> The process is waiting on the pthread_join(RaMySQLThread) which to me
>> indicates that RaData and RaCurses have terminated and that the only threads
>> that are still alive are the mainline and RaMySQL. This is confirmed by info
>> threads which show the two threads and no others.
>>
>> if ((pthread_create(&RaCursesThread, NULL, ArgusCursesProcess, NULL))
>> != 0)
>> ArgusLog (LOG_ERR, "ArgusCursesProcess() pthread_create error
>> %s\n", strerror(errno));
>>
>> if ((pthread_create(&RaDataThread, NULL, ArgusProcessData, NULL)) !=
>> 0)
>> ArgusLog (LOG_ERR, "main() pthread_create error %s\n",
>> strerror(errno));
>>
>> if ((pthread_join(RaDataThread, NULL)) != 0)
>> if ((pthread_join(RaCursesThread, NULL)) != 0)
>> if ((pthread_join(RaMySQLThread, NULL)) != 0)
>>
>> When I look at the code for the RaMySQL thread, it seems to hang out in a
>> loop checking the count of items on the ArgusSQLQueryList. As long as there
>> are items on the list it processes them. This is confirmed by setting a
>> display on the ->count element which does eventually work its way down to 0.
>> So far, so good. When the count hits zero it executes this code:
>>
>> pthread_join(RaCursesThread, NULL);
>>
>> Since the RaCurseThread has exited (or the pthread_join in the mainline
>> would not have been satisfied) this should not block, leading to the
>> execution of this code
>>
>> pthread_mutex_lock(&ArgusSQLQueryList->lock);
>> pthread_cond_wait(&ArgusSQLQueryList->cond, &ArgusSQLQueryList->lock);
>> pthread_mutex_unlock(&ArgusSQLQueryList->lock)
>>
>> I just can't figure out what is going to send
>> pthread_cond_signal(&ArgusSQLQueryList->cond) since that happens only in the
>> RaCurses thread which is no longer running. I'm not sure I would call this a
>> true deadlock (A waiting for B who is waiting for A) though that seems to be
>> more of a distinction than a difference. Would it be reasonable to make this
>> into a timed conditional wait?
>>
>> --Dave
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Friday, August 23, 2013 7:21 PM
>> To: David Edelman
>> Cc: 'Argus'
>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>
>> Hey Dave,
>> Yes, we have a bunch of threads that all want to stop at once,
>> but we need them to exit in something of an order.
>>
>> If you ran this under gdb(), when it freezes and you hit Ctr-C,
>> it should return to the debugger, and if you go through the
>> threads again, we should see if we've got a deadlock, which
>> it sounds like we have. A thread waiting for B thread, B thread
>> waiting for C and C waiting for A, or something like it.
>>
>> Carter
>>
>> On Aug 23, 2013, at 5:45 PM, "David Edelman" <dedelman at iname.com> wrote:
>>
>>> I'm not sure which version of the source code you want me to modify. I
>> just
>>> downloaded a copy from the website and I think that this is the section in
>>> question despite the differences in the line numbers:
>>>
>>> 3310 ArgusCloseDown = 1;
>>> 3311 ArgusParser->RaParseDone = 1;
>>> 3312
>>> 3313 pthread_join(RaCursesInputThread, NULL);
>>> 3314 pthread_cond_signal(&ArgusSQLQueryList->cond);
>>> 3315 }
>>>
>>>
>>> When I run it against a single hour of flow data it doesn't fault, but it
>>> also doesn't seem to get to EOJ.
>>>
>>> When I run it against a file with a single file that contains a single
>> flow
>>> record in this is what I see:
>>>
>>> time rasqlinsert -M time 1d -r argusTestOne -M rmon -w
>>> mysql://argus:argus@localhost/argus/YYmacAddrs_%Y_%m_%d -m srcid saddr
>> smac
>>> -s stime ltime srcid saddr smac -D 3 - ip
>>> rasqlinsert[17007.40d7d96b757f0000]: 2013-08-23-21:31:58.054
>>> Argusfinish_parse (0x2294ef8)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>>> ArgusFilterCompile () done 0
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>>> RaCursesNewProcess(0x6bcf8010) returns 0x2294d70
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>>> RaCursesNewProcess(0x6bcf8010) returns 0x2294ea0
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.055
>>> RaCursesNewProcess(0x6bcf8010) returns 0x2294fd0
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.174 RaMySQLInit
>> ()
>>> RaSource (null) RaArchive (null) RaFormat (null)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.179
>>> ArgusInitAddrtoname (0x7f756bcf8010, 0x0, 0x0)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.179
>>> ArgusParseInit(0x7f756bcf8010, NULL)
>>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.179
>>> ArgusMySQLProcess() starting
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>>> ArgusProcessData() starting
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>>> ArgusReadConnection() read 16 bytes
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.179
>>> ArgusReadConnection() read 112 bytes
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>>> ArgusParseInit(0x7f756bcf8010 0x7f756bc15010
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>>> ArgusReadConnection(0x6bc15010, 1) returning 1
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.180
>>> RaProcessSplitOptions(YYmacAddrs_2013_08_19, 4096, 0x6bc15630): returns 0
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.187 generating
>>> table CREATE table argus.YYmacAddrs_2013_08_19 (stime double(18,6)
>> unsigned
>>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>>> record blob) ENGINE=MyISAM
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.458
>>> ArgusCreateSQLSaveTable (YYmacAddrs_2013_08_19) returning
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.459
>>> ArgusCloseInput(0x6bc15010) closing
>>> rasqlinsert[17006.00d78968757f0000]: 2013-08-23-21:31:58.459
>>> ArgusCloseInput(0x6bc15010) done
>>> rasqlinsert[17006.00e70969757f0000]: 2013-08-23-21:31:58.480
>>> ArgusScheduleSQLQuery (0x6bcf8010, 0x2294a40, 0x5c005340, INSERT INTO
>>> argus.YYmacAddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376870402.927","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>>> ac:31",...), 32) done
>>> rasqlinsert[17006.00e70969757f0000]: 2013-08-23-21:31:58.480
>>> ArgusScheduleSQLQuery (0x6bcf8010, 0x2294a40, 0x5c003fd0, INSERT INTO
>>> argus.YYmacAddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376870402.927","108.50.164.35","50.93.246.64","f8:e4:fb:
>>> 19:b6:84",...), 32) done
>>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.480 ArgusSQLQuery
>>> (INSERT INTO argus.YYmacAddrs_2013_08_19
>>> (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376870402.927","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>>> ac:31",...))
>>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.480 ArgusSQLQuery
>>> (INSERT INTO argus.YYmacAddrs_2013_08_19
>>> (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376870402.927","108.50.164.35","50.93.246.64","f8:e4:fb:
>>> 19:b6:84",...))
>>> rasqlinsert[17006.00d72a6a757f0000]: 2013-08-23-21:31:58.517
>>> ArgusMySQLProcess() done!
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> ArgusWindowClose () returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> RaParseComplete(caught signal 0)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517 ArgusShutDown
>>> (0)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> ArgusWindowClose () returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> RaParseComplete(caught signal 0)
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> ArgusDeleteModeList () returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> ArgusDeleteFileList () returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusDeleteList
>>> (0x2293050, 4) returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>> ArgusDeleteList
>>> (0x22930f0, 4) returning
>>> rasqlinsert[17006.40d7d96b757f0000]: 2013-08-23-21:31:58.517
>>> ArgusDeleteAggregator(0x7f756bcf8010, 0x2294a40) returned
>>>
>>> real 0m0.470s
>>> user 0m0.014s
>>> sys 0m0.004s
>>>
>>> When I add a second file which also contains a single flow record from
>> later
>>> that same day, I see this:
>>>
>>> [root at monolith 19]# time rasqlinsert -M time 1d -r /data2/test/argusTest*
>>> -M rmon -w mysql://argus:argus@localhost/argus/QQmacA4ddrs_%Y_%m_%d -m
>> srcid
>>> saddr
>>> smac -s stime ltime srcid saddr smac -D 3
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>>> RaCursesNewProcess(0x2aee4010) returns 0x1854d90
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>>> RaCursesNewProcess(0x2aee4010) returns 0x1854ec0
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.638
>>> RaCursesNewProcess(0x2aee4010) returns 0x1854ff0
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.707 RaMySQLInit
>> ()
>>> RaSource (null) RaArchive (null) RaFormat (null)
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.711
>>> ArgusInitAddrtoname (0x7fe12aee4010, 0x0, 0x0)
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:37:41.711
>>> ArgusParseInit(0x7fe12aee4010, NULL)
>>> rasqlinsert[17130.00774329e17f0000]: 2013-08-23-21:37:41.711
>>> ArgusMySQLProcess() starting
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.711
>>> ArgusProcessData() starting
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>>> ArgusReadConnection() read 16 bytes
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>>> ArgusReadConnection() read 112 bytes
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>>> ArgusParseInit(0x7fe12aee4010 0x7fe12ae01010
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>>> ArgusReadConnection(0x2ae01010, 1) returning 1
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.712
>>> RaProcessSplitOptions(QQmacA4ddrs_2013_08_19, 4096, 0x2ae01630): returns 0
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.719 generating
>>> table CREATE table argus.QQmacA4ddrs_2013_08_19 (stime double(18,6)
>> unsigned
>>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>>> record blob) ENGINE=MyISAM
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.808
>>> ArgusCreateSQLSaveTable (QQmacA4ddrs_2013_08_19) returning
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusCloseInput(0x2ae01010) closing
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusCloseInput(0x2ae01010) done
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusReadConnection() read 16 bytes
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusReadConnection() read 112 bytes
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusParseInit(0x7fe12aee4010 0x7fe12ad9f010
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusReadConnection(0x2ad9f010, 1) returning 1
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusCloseInput(0x2ad9f010) closing
>>> rasqlinsert[17130.00e77f23e17f0000]: 2013-08-23-21:37:41.809
>>> ArgusCloseInput(0x2ad9f010) done
>>> rasqlinsert[17130.00f7ff23e17f0000]: 2013-08-23-21:37:41.812
>>> ArgusScheduleSQLQuery (0x2aee4010, 0x1854a60, 0x1c005340, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376874001.616","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>>> ac:31",...), 32) done
>>> rasqlinsert[17130.00f7ff23e17f0000]: 2013-08-23-21:37:41.812
>>> ArgusScheduleSQLQuery (0x2aee4010, 0x1854a60, 0x1c003fd0, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376870400.000","1376874001.616","108.50.164.35","50.93.246.64","f8:e4:fb:
>>> 19:b6:84",...), 32) done
>>>
>>> And I have to hit ^C to stop it.
>>>
>>> ^Crasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>>> RaParseComplete(caught signal 2)
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568 ArgusShutDown
>>> (2)
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>>> ArgusWindowClose () returning
>>> rasqlinsert[17130.4097f82ae17f0000]: 2013-08-23-21:40:39.568
>>> RaParseComplete(caught signal 2)
>>>
>>> A single file with 10 flow records also freezes:
>>>
>>> [root at monolith 19]# time rasqlinsert -M time 1d -r
>>> /data2/test/argusTestThree -M rmon -w
>>> mysql://argus:argus@localhost/argus/QQmacA4ddrs_%Y_%m_%d -m srcid sa
>>> ddr smac -s stime ltime srcid saddr smac -D 3
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>>> RaCursesNewProcess(0x4f8ce010) returns 0x2690d20
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>>> RaCursesNewProcess(0x4f8ce010) returns 0x2690e50
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.410
>>> RaCursesNewProcess(0x4f8ce010) returns 0x2690f80
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.487 RaMySQLInit
>> ()
>>> RaSource (null) RaArchive (null) RaFormat (null)
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusInitAddrtoname (0x7f8f4f8ce010, 0x0, 0x0)
>>> rasqlinsert[17197.4037974f8f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusParseInit(0x7f8f4f8ce010, NULL)
>>> rasqlinsert[17197.0037e84d8f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusMySQLProcess() starting
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusProcessData() starting
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusReadConnection() read 16 bytes
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusReadConnection() read 112 bytes
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusParseInit(0x7f8f4f8ce010 0x7f8f4f7eb010
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> ArgusReadConnection(0x4f7eb010, 1) returning 1
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.492
>>> RaProcessSplitOptions(QQmacA4ddrs_2013_08_19, 4096, 0x4f7eb630): returns 0
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.500 generating
>>> table CREATE table argus.QQmacA4ddrs_2013_08_19 (stime double(18,6)
>> unsigned
>>> not null,ltime double(18,6) unsigned not null,srcid varchar(64),saddr
>>> varchar(64) not null,smac varchar(24), primary key (srcid,saddr,smac),
>>> record blob) ENGINE=MyISAM
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.589
>>> ArgusCreateSQLSaveTable (QQmacA4ddrs_2013_08_19) returning
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.590
>>> ArgusCloseInput(0x4f7eb010) closing
>>> rasqlinsert[17197.00f7ff478f7f0000]: 2013-08-23-21:42:22.590
>>> ArgusCloseInput(0x4f7eb010) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40005340, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874004.586","108.50.164.35","10.1.1.45","00:1c:c0:ee:
>>> ac:31",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000d3d0, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.348","1376874004.586","108.50.164.35","10.1.1.10","00:1d:73:ff:
>>> 5d:29",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.592
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40003fd0, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874001.616","108.50.164.35","50.93.246.64","f8:e4:fb:
>>> 19:b6:84",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x400061f0, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874004.371","108.50.164.35","10.1.1.15","18:b4:30:01:
>>> 86:8a",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40004a00, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874004.589","108.50.164.35","10.1.1.101","3c:07:54:5b
>>> :be:b5",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40007090, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874004.589","108.50.164.35","216.17.8.6","f8:e4:fb:19
>>> :b6:84",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40009900, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874001.014","108.50.164.35","50.16.200.78","f8:e4:fb:
>>> 19:b6:84",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40003650, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874002.599","108.50.164.35","10.1.1.60","50:46:5d:6f:
>>> d1:47",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000a700, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.000","1376874004.371","108.50.164.35","38.102.136.104","f8:e4:f
>>> b:19:b6:84",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x40007c40, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.002","1376874002.137","108.50.164.35","10.1.1.68","00:24:1d:87:
>>> 53:03",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x4000b690, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.016","1376874000.017","108.50.164.35","00:26:f2:f2:56:bb","00:2
>>> 6:f2:f2:56:bb",...), 32) done
>>> rasqlinsert[17197.0047c74c8f7f0000]: 2013-08-23-21:42:22.593
>>> ArgusScheduleSQLQuery (0x4f8ce010, 0x2690a40, 0x400090a0, INSERT INTO
>>> argus.QQmacA4ddrs_2013_08_19 (stime,ltime,srcid,saddr,smac,record) VALUES
>> ("1376874000.016","1376874000.017","108.50.164.35","01:80:c2:00:00:0e","01:8
>>> 0:c2:00:00:0e",...), 32) done
>>>
>>> I hope that this helps, it look very close to a solution.
>>>
>>> --Dave
>>>
>>>
>>> -----Original Message-----
>>> From: Carter Bullard [mailto:carter at qosient.com]
>>> Sent: Friday, August 23, 2013 10:38 AM
>>> To: David Edelman
>>> Cc: 'Argus'
>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>
>>> Hey Dave,
>>> Can you apply this patch to see if your rasqlinsert() doesn't get a little
>>> better?
>>>
>>> osiris:ramysql carter$ diff rasqlinsert.c rasqlinsert.c.new
>>> 3323c3323,3325
>>> < pthread_join(RaCursesInputThread, NULL);
>>> ---
>>>> if (ArgusParser->RaCursesMode)
>>>> pthread_join(RaCursesInputThread, NULL);
>>>
>>> Thanks !!!!!
>>>
>>> Carter
>>>
>>> On Aug 21, 2013, at 6:04 PM, "David Edelman" <dedelman at iname.com> wrote:
>>>
>>>> With the thread specific information:
>>>>
>>>> gdb rasqlinsert
>>>> GNU gdb (GDB) Fedora (7.5.1-38.fc18)
>>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law. Type "show
>> copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>...
>>>> Reading symbols from /usr/local/bin/rasqlinsert...done.
>>>> (gdb) run -M time 1d -r * -M rmon -w
>>>> mysql://argus:argus@localhost/argus/VVmacAddrs_%Y_%m_%d -m srcid saddr
>>> smac
>>>> -s stime ltime srcid saddr smac - ip
>>>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r * -M rmon -w
>>>> mysql://argus:argus@localhost/argus/VVmacAddrs_%Y_%m_%d -m srcid saddr
>>> smac
>>>> -s stime ltime srcid saddr smac - ip
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>> Detaching after fork from child process 2817.
>>>> [New Thread 0x7ffff51a2700 (LWP 2820)]
>>>> [Thread 0x7ffff51a2700 (LWP 2820) exited]
>>>> [New Thread 0x7ffff51a2700 (LWP 2826)]
>>>> [New Thread 0x7fffef7fe700 (LWP 2827)]
>>>> [New Thread 0x7fffeeffd700 (LWP 2828)]
>>>> Detaching after fork from child process 2829.
>>>> Detaching after fork from child process 2831.
>>>> Detaching after fork from child process 2833.
>>>> Detaching after fork from child process 2837.
>>>> Detaching after fork from child process 2839.
>>>> Detaching after fork from child process 2841.
>>>> Detaching after fork from child process 2843.
>>>> Detaching after fork from child process 2845.
>>>> Detaching after fork from child process 2847.
>>>> Detaching after fork from child process 2849.
>>>> Detaching after fork from child process 2851.
>>>> Detaching after fork from child process 2853.
>>>> Detaching after fork from child process 2855.
>>>> Detaching after fork from child process 2857.
>>>> Detaching after fork from child process 2859.
>>>> Detaching after fork from child process 2861.
>>>> Detaching after fork from child process 2863.
>>>> Detaching after fork from child process 2865.
>>>> Detaching after fork from child process 2867.
>>>> Detaching after fork from child process 2869.
>>>> Detaching after fork from child process 2872.
>>>> Detaching after fork from child process 2874.
>>>> Detaching after fork from child process 2876.
>>>> Detaching after fork from child process 2878.
>>>> [Thread 0x7fffeeffd700 (LWP 2828) exited]
>>>>
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> [Switching to Thread 0x7fffef7fe700 (LWP 2827)]
>>>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>>>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>>>> ncurses-libs-5.9-11.20130511.fc18.x86_64
>> openssl-libs-1.0.1e-4.fc18.x86_64
>>>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>>>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>>>> (gdb) where
>>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>> ./rasqlinsert.c:3313
>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>> ./rasqlinsert.c:450
>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>>
>>>> (gdb) info threads
>>>> Id Target Id Frame
>>>> * 4 Thread 0x7fffef7fe700 (LWP 2827) "rasqlinsert" 0x00007ffff7570dcb
>>> in
>>>> pthread_join () from /lib64/libpthread.so.0
>>>> 3 Thread 0x7ffff51a2700 (LWP 2826) "rasqlinsert" 0x00007ffff75764b7 in
>>>> recv () from /lib64/libpthread.so.0
>>>> 1 Thread 0x7ffff7560740 (LWP 2813) "rasqlinsert" 0x00007ffff7570e60 in
>>>> pthread_join () from /lib64/libpthread.so.0
>>>>
>>>> (gdb) thread 1
>>>> [Switching to thread 1 (Thread 0x7ffff7560740 (LWP 2813))]
>>>> #0 0x00007ffff7570e60 in pthread_join () from /lib64/libpthread.so.0
>>>> (gdb) where
>>>> #0 0x00007ffff7570e60 in pthread_join () from /lib64/libpthread.so.0
>>>> #1 0x000000000040cb6b in main (argc=45, argv=0x7fffffffdc38) at
>>>> ./rasqlinsert.c:302
>>>>
>>>> (gdb) thread 3
>>>> [Switching to thread 3 (Thread 0x7ffff51a2700 (LWP 2826))]
>>>> #0 0x00007ffff75764b7 in recv () from /lib64/libpthread.so.0
>>>> (gdb) where
>>>> #0 0x00007ffff75764b7 in recv () from /lib64/libpthread.so.0
>>>> #1 0x00007ffff7a71299 in inline_mysql_socket_recv (flags=<optimized
>> out>,
>>>> n=<optimized out>, buf=<optimized out>, mysql_socket=...,
>>>> src_line=<optimized out>,
>>>> src_file=<optimized out>) at /usr/include/bits/socket2.h:35
>>>> #2 vio_read (vio=0x100e580, buf=0x100e770 "0", size=16384) at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/vi
>>>> o/viosocket.c:114
>>>> #3 0x00007ffff7a7131e in vio_read_buff (vio=0x7, buf=0x1012780 "\321\v",
>>>> size=4)
>>>> at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/vi
>>>> o/viosocket.c:157
>>>> #4 0x00007ffff7a566a7 in net_read_raw_loop (count=4, net=<optimized
>> out>)
>>>> at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l/net_serv.cc:663
>>>> #5 net_read_packet_header (net=<optimized out>) at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l/net_serv.cc:751
>>>> #6 net_read_packet (net=0x100aa60, complen=0x7ffff51a1ca0) at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l/net_serv.cc:810
>>>> #7 0x00007ffff7a569af in my_net_read (net=0x7) at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l/net_serv.cc:888
>>>> #8 0x00007ffff7a4d7b6 in cli_safe_read (mysql=0x7) at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l-common/client.c:610
>>>> #9 0x00007ffff7a50257 in cli_read_query_result (mysql=0x7)
>>>> at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l-common/client.c:4015
>>>> #10 0x00007ffff7a4c9a6 in mysql_real_query (mysql=0x100aa60,
>>>> query=<optimized out>, length=<optimized out>)
>>>> at
>> /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sq
>>>> l-common/client.c:4103
>>>> #11 0x000000000041e0ef in ArgusProcessSQLQueryList
>> (parser=0x7ffff74bb010)
>>>> at ./rasqlinsert.c:6556
>>>> #12 0x000000000041e674 in ArgusMySQLProcess (arg=0x7ffff74bb010) at
>>>> ./rasqlinsert.c:6694
>>>> #13 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>> #14 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>>
>>>> (gdb) thread 4
>>>> [Switching to thread 4 (Thread 0x7fffef7fe700 (LWP 2827))]
>>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> (gdb) where
>>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>> ./rasqlinsert.c:3313
>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>> ./rasqlinsert.c:450
>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>> Sent: Wednesday, August 21, 2013 2:38 PM
>>>> To: David Edelman
>>>> Cc: 'Argus'
>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>
>>>> When you are debugging rasqlinsert(), there are a
>>>> number of threads, and so after printing " where ",
>>>> you can also do a " info threads ", and then for
>>>> each thread, type " thread x " where x is the thread
>>>> number, and do a where once in each thread.
>>>>
>>>> (gdb) info threads
>>>> (gdb) thread 1
>>>> (gdb) where
>>>> (gdb) thread 2
>>>> (where)
>>>>
>>>> etc . That can be very useful.
>>>> Still working on it.
>>>>
>>>> Carter
>>>>
>>>>
>>>>
>>>> On Aug 21, 2013, at 2:17 PM, "David Edelman" <dedelman at iname.com> wrote:
>>>>
>>>>> It is getting better - that's purely subjective It still faults but I
>> was
>>>>> able to run through a directory tree with -R for quite a while before
>> the
>>>>> fault. I still had the same problem that I always had where only the
>>> first
>>>>> table is populated the rest are created but empty. I've been doing
>> nested
>>>>> bash for loops to deal with that for quite some time.
>>>>>
>>>>> I ran against a single flow record file under gdb and this is the
>> result.
>>>
>>>>>
>>>>> --Dave
>>>>>
>>>>>
>>>>>
>>>>> (gdb) run -M time 1d -r argus.2013.04.06.00.00.01.0.gz -M rmon -w
>>>>> mysql://argus:argus@localhost/argus/TTmacAddrs_%Y_%m_%d -m srcid saddr
>>>> smac
>>>>> -s stime ltime srcid saddr smac - ip
>>>>> Starting program: /usr/local/bin/rasqlinsert -M time 1d -r
>>>>> argus.2013.04.06.00.00.01.0.gz -M rmon -w
>>>>> mysql://argus:argus@localhost/argus/TTmacAddrs_%Y_%m_%d -m srcid saddr
>>>> smac
>>>>> -s stime ltime srcid saddr smac - ip
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>>> Detaching after fork from child process 31440.
>>>>> [New Thread 0x7ffff5a70700 (LWP 31441)]
>>>>> [Thread 0x7ffff5a70700 (LWP 31441) exited]
>>>>> [New Thread 0x7ffff5a70700 (LWP 31442)]
>>>>> [New Thread 0x7ffff4861700 (LWP 31443)]
>>>>> [New Thread 0x7fffeffff700 (LWP 31444)]
>>>>> Detaching after fork from child process 31445.
>>>>> [Thread 0x7fffeffff700 (LWP 31444) exited]
>>>>>
>>>>> Program received signal SIGSEGV, Segmentation fault.
>>>>> [Switching to Thread 0x7ffff4861700 (LWP 31443)]
>>>>> 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>>> Missing separate debuginfos, use: debuginfo-install
>>>>> flow-tools-0.68.5.1-6.fc18.x86_64 glibc-2.16-33.fc18.x86_64
>>>>> libgcc-4.7.2-8.fc18.x86_64 libstdc++-4.7.2-8.fc18.x86_64
>>>>> ncurses-libs-5.9-11.20130511.fc18.x86_64
>>> openssl-libs-1.0.1e-4.fc18.x86_64
>>>>> pcre-8.31-5.fc18.x86_64 readline-6.2-5.fc18.x86_64
>>>>> tcp_wrappers-libs-7.6-70.fc18.x86_64 zlib-1.2.7-9.fc18.x86_64
>>>>> (gdb) where
>>>>> #0 0x00007ffff7570dcb in pthread_join () from /lib64/libpthread.so.0
>>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>>> ./rasqlinsert.c:3313
>>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>>> ./rasqlinsert.c:450
>>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>>> (gdb) list
>>>>> 230 int ArgusColorAddresses(struct ArgusParserStruct *, struct
>>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>>> 231 int ArgusColorFlowFields(struct ArgusParserStruct *, struct
>>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>>> 232 int ArgusColorGeoLocation(struct ArgusParserStruct *, struct
>>>>> ArgusRecordStruct *, struct ArgusAttributeStruct *, short, attr_t);
>>>>> 233 void ArgusInitializeColorMap(struct ArgusParserStruct *, WINDOW *);
>>>>> 234 #endif
>>>>> 235
>>>>> 236 int
>>>>> 237 main(int argc, char **argv)
>>>>> 238 {
>>>>> 239 struct ArgusParserStruct *parser = NULL;
>>>>> (gdb) up
>>>>> #1 0x0000000000414ff3 in ArgusCursesProcessClose () at
>>>> ./rasqlinsert.c:3313
>>>>> 3313 pthread_join(RaCursesInputThread, NULL);
>>>>> (gdb) up
>>>>> #2 0x000000000040cf9f in ArgusCursesProcess (arg=0x0) at
>>>>> ./rasqlinsert.c:450
>>>>> 450 ArgusCursesProcessClose();
>>>>> (gdb) up
>>>>> #3 0x00007ffff756fd15 in start_thread () from /lib64/libpthread.so.0
>>>>> (gdb) up
>>>>> #4 0x0000003c88ef253d in clone () from /lib64/libc.so.6
>>>>> (gdb) up
>>>>> Initial frame selected; you cannot go up.
>>>>> (gdb)
>>>>>
>>>>>
>>>>> dmesg shows:
>>>>>
>>>>> [150781.353120] rasqlinsert[26482]: segfault at 2d0 ip 00007ff53b7e6dcb
>>> sp
>>>>> 00007ff533ffdda0 error 4 in libpthread-2.16.so[7ff53b7de000+16000]
>>>>> [150980.758765] rasqlinsert[31192]: segfault at 2d0 ip 00007fa5bfab3dcb
>>> sp
>>>>> 00007fa5b7ffdda0 error 4 in libpthread-2.16.so[7fa5bfaab000+16000]
>>>>> [151147.481928] rasqlinsert[31303]: segfault at 2d0 ip 00007f95d0346dcb
>>> sp
>>>>> 00007f95ccd67da0 error 4 in libpthread-2.16.so[7f95d033e000+16000]
>>>>> [151244.434895] rasqlinsert[31402]: segfault at 2d0 ip 00007f071442fdcb
>>> sp
>>>>> 00007f071171eda0 error 4 in libpthread-2.16.so[7f0714427000+16000]
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>> Sent: Wednesday, August 21, 2013 11:31 AM
>>>>> To: David Edelman
>>>>> Cc: 'Argus'
>>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>
>>>>> I think I've found a knob to turn that might help. Can you try this
>>>> change?
>>>>> This
>>>>> controls how aggressively we will push modifications out to the
>> database.
>>>
>>>>>
>>>>> ==== //depot/argus/clients/examples/ramysql/raclient.c#8 -
>>>>> /Users/carter/argus/clients/examples/ramysql/raclient.c ====
>>>>> 810c810
>>>>> < RaCursesUpdateInterval.tv_sec = 0;
>>>>> ---
>>>>>> RaCursesUpdateInterval.tv_sec = 1;
>>>>>
>>>>> You may still get a problem, but it should be better. I'll keep looking
>>>>> into
>>>>> a better fix that this, but any improvement means where in the right
>>>>> direction.
>>>>>
>>>>> Carter
>>>>>
>>>>> On Aug 21, 2013, at 10:26 AM, "David Edelman" <dedelman at iname.com>
>> wrote:
>>>>>
>>>>>> Carter,
>>>>>>
>>>>>> That's great news and it makes sense. The instances that are processing
>>>>>> stream data don't have enough traffic to create huge updates, and the
>>>>>> instances that are cruising through files surely do. I have 128GB of
>>>>>> physical memory and I've tuned MySQL to use as much as possible. I can
>>>>>> detune it and see if that makes a difference.
>>>>>>
>>>>>> --Dave
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>>> Sent: Wednesday, August 21, 2013 8:10 AM
>>>>>> To: David Edelman
>>>>>> Cc: 'Argus'
>>>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>>
>>>>>> Hey David,
>>>>>> I'm also getting the same error here, with your example. Not sure
>> where
>>>>>> this crept in, as I use rasqlinsert() on many systems and I haven't
>> seen
>>>>>> this,
>>>>>> although I have started to see rasqlinsert()s that eat a lot of CPU.
>>>>>>
>>>>>> This is a mysqld problem, where it coughs blood processing packets that
>>>>>> are too large, or where multiple threads are making concurrent
>>>>>> sql calls on the same socket.
>>>>>>
>>>>>> http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
>>>>>>
>>>>>> We do try to send the largest query possible by packing as many
>> INSERTS,
>>>>>> or UPDATES, into the mysql ' max_allowed_packet ', and we have multiple
>>>>>> threads talking to mysqld. I just now tried to cut the
>>>> max_allowed_packet
>>>>>> buffer in 1/2, with no effect, so ..... I've got some work to do here
>>> and
>>>>>> will
>>>>>> try to have something today.
>>>>>>
>>>>>> Thanks for the feedback !!!!
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>> On Aug 20, 2013, at 7:43 PM, David Edelman <dedelman at iname.com> wrote:
>>>>>>
>>>>>>> I had a theory that the problem happened when there were no more
>>> records
>>>>>>> available to be read. To test this I moved up one level in the data
>>>>> source
>>>>>>> directory tree and used -R * figuring that I should see good tables
>> for
>>>>>> the
>>>>>>> dates prior to the most recent. The theory may be good but the test
>> may
>>>>>> have
>>>>>>> been useful for a different reason. Newest client build with .devel
>> and
>>>>>>> .debug run without a -D switch. I get this almost immediately (but
>> this
>>>>> is
>>>>>> a
>>>>>>> big machine with lots of memory so speed is a hard thing to judge.)
>>>>>>>
>>>>>>>
>>>>>>> rasqlinsert -M time 1d -R * -M rmon -w
>>>>>>> mysql://argus:argus@localhost/argus/SSmacAddrs_%Y_%m_%d -m srcid saddr
>>>>>> smac
>>>>>>> -s stime ltime srcid saddr smac - ip
>>>>>>> *** glibc detected *** rasqlinsert: double free or corruption (!prev):
>>>>>>> 0x00000000020e4830 ***
>>>>>>> rasqlinsert[8994]: 2013-08-20-23:27:39.017 mysql_real_query error Lost
>>>>>>> connection to MySQL server during query
>>>>>>> ======= Backtrace: =========
>>>>>>> /lib64/libc.so.6[0x3c88e7cb3e]
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(vio_delete+0x26)[0x7f94870c5416]
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(end_server+0x38)[0x7f94870a2678]
>>>> /usr/lib64/mysql/libmysqlclient.so.18(cli_safe_read+0x30)[0x7f94870a2770]
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18(+0x41257)[0x7f94870a5257]
>> /usr/lib64/mysql/libmysqlclient.so.18(mysql_real_query+0x26)[0x7f94870a19a6]
>> /usr/lib64/mysql/libmysqlclient.so.18(mysql_list_tables+0x55)[0x7f948709edb5
>>>>>>> ]
>>>>>>> rasqlinsert[0x41fe3c]
>>>>>>> rasqlinsert[0x408c68]
>>>>>>> rasqlinsert[0x4086a5]
>>>>>>> rasqlinsert[0x438d35]
>>>>>>> rasqlinsert[0x43933e]
>>>>>>> rasqlinsert[0x47b12a]
>>>>>>> rasqlinsert[0x47b335]
>>>>>>> rasqlinsert[0x405be7]
>>>>>>> /lib64/libpthread.so.0(+0x3c89207d15)[0x7f9486bc4d15]
>>>>>>> /lib64/libc.so.6(clone+0x6d)[0x3c88ef253d]
>>>>>>> ======= Memory map: ========
>>>>>>> 00400000-004eb000 r-xp 00000000 fd:00 20983991
>>>>>>> /usr/local/bin/rasqlinsert
>>>>>>> 006ea000-006eb000 r--p 000ea000 fd:00 20983991
>>>>>>> /usr/local/bin/rasqlinsert
>>>>>>> 006eb000-006fb000 rw-p 000eb000 fd:00 20983991
>>>>>>> /usr/local/bin/rasqlinsert
>>>>>>> 006fb000-01007000 rw-p 00000000 00:00 0
>>>>>>> 020d8000-021c0000 rw-p 00000000 00:00 0
>>>>>>> [heap]
>>>>>>> 3c88a00000-3c88a20000 r-xp 00000000 fd:00 77463577
>>>>>>> /usr/lib64/ld-2.16.so
>>>>>>> 3c88c20000-3c88c21000 r--p 00020000 fd:00 77463577
>>>>>>> /usr/lib64/ld-2.16.so
>>>>>>> 3c88c21000-3c88c22000 rw-p 00021000 fd:00 77463577
>>>>>>> /usr/lib64/ld-2.16.so
>>>>>>> 3c88c22000-3c88c23000 rw-p 00000000 00:00 0
>>>>>>> 3c88e00000-3c88fad000 r-xp 00000000 fd:00 77463656
>>>>>>> /usr/lib64/libc-2.16.so
>>>>>>> 3c88fad000-3c891ad000 ---p 001ad000 fd:00 77463656
>>>>>>> /usr/lib64/libc-2.16.so
>>>>>>> 3c891ad000-3c891b1000 r--p 001ad000 fd:00 77463656
>>>>>>> /usr/lib64/libc-2.16.so
>>>>>>> 3c891b1000-3c891b3000 rw-p 001b1000 fd:00 77463656
>>>>>>> /usr/lib64/libc-2.16.so
>>>>>>> 3c891b3000-3c891b8000 rw-p 00000000 00:00 0
>>>>>>> 3c89200000-3c89202000 r-xp 00000000 fd:00 77464193
>>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>>> 3c89202000-3c89401000 ---p 00002000 fd:00 77464193
>>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>>> 3c89401000-3c89402000 r--p 00001000 fd:00 77464193
>>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>>> 3c89402000-3c89403000 rw-p 00002000 fd:00 77464193
>>>>>>> /usr/lib64/libpcreposix.so.0.0.1
>>>>>>> 3c89600000-3c89603000 r-xp 00000000 fd:00 77471354
>>>>>>> /usr/lib64/libdl-2.16.so
>>>>>>> 3c89603000-3c89802000 ---p 00003000 fd:00 77471354
>>>>>>> /usr/lib64/libdl-2.16.so
>>>>>>> 3c89802000-3c89803000 r--p 00002000 fd:00 77471354
>>>>>>> /usr/lib64/libdl-2.16.so
>>>>>>> 3c89803000-3c89804000 rw-p 00003000 fd:00 77471354
>>>>>>> /usr/lib64/libdl-2.16.so
>>>>>>> 3c89a00000-3c89a07000 r-xp 00000000 fd:00 77463871
>>>>>>> /usr/lib64/librt-2.16.so
>>>>>>> 3c89a07000-3c89c06000 ---p 00007000 fd:00 77463871
>>>>>>> /usr/lib64/librt-2.16.so
>>>>>>> 3c89c06000-3c89c07000 r--p 00006000 fd:00 77463871
>>>>>>> /usr/lib64/librt-2.16.so
>>>>>>> 3c89c07000-3c89c08000 rw-p 00007000 fd:00 77463871
>>>>>>> /usr/lib64/librt-2.16.so
>>>>>>> 3c89e00000-3c89e15000 r-xp 00000000 fd:00 77471658
>>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>>> 3c89e15000-3c8a014000 ---p 00015000 fd:00 77471658
>>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>>> 3c8a014000-3c8a015000 r--p 00014000 fd:00 77471658
>>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>>> 3c8a015000-3c8a016000 rw-p 00015000 fd:00 77471658
>>>>>>> /usr/lib64/libz.so.1.2.7
>>>>>>> 3c8a200000-3c8a300000 r-xp 00000000 fd:00 77471671
>>>>>>> /usr/lib64/libm-2.16.so
>>>>>>> 3c8a300000-3c8a4ff000 ---p 00100000 fd:00 77471671
>>>>>>> /usr/lib64/libm-2.16.so
>>>>>>> 3c8a4ff000-3c8a500000 r--p 000ff000 fd:00 77471671
>>>>>>> /usr/lib64/libm-2.16.so
>>>>>>> 3c8a500000-3c8a501000 rw-p 00100000 fd:00 77471671
>>>>>>> /usr/lib64/libm-2.16.so
>>>>>>> 3c8a600000-3c8a615000 r-xp 00000000 fd:00 77471694
>>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>>> 3c8a615000-3c8a814000 ---p 00015000 fd:00 77471694
>>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>>> 3c8a814000-3c8a815000 r--p 00014000 fd:00 77471694
>>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>>> 3c8a815000-3c8a816000 rw-p 00015000 fd:00 77471694
>>>>>>> /usr/lib64/libgcc_s-4.7.2-20121109.so.1
>>>>>>> 3c8aa00000-3c8aa5c000 r-xp 00000000 fd:00 77471660
>>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>>> 3c8aa5c000-3c8ac5c000 ---p 0005c000 fd:00 77471660
>>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>>> 3c8ac5c000-3c8ac5d000 r--p 0005c000 fd:00 77471660
>>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>>> 3c8ac5d000-3c8ac5e000 rw-p 0005d000 fd:00 77471660
>>>>>>> /usr/lib64/libpcre.so.1.0.1
>>>>>>> 3c8b600000-3c8b63c000 r-xp 00000000 fd:00 77465274
>>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>>> 3c8b63c000-3c8b83b000 ---p 0003c000 fd:00 77465274
>>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>>> 3c8b83b000-3c8b83d000 r--p 0003b000 fd:00 77465274
>>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>>> 3c8b83d000-3c8b843000 rw-p 0003d000 fd:00 77465274
>>>>>>> /usr/lib64/libreadline.so.6.2
>>>>>>> 3c8b843000-3c8b845000 rw-p 00000000 00:00 0
>>>>>>> 3c8ca00000-3c8cae5000 r-xp 00000000 fd:00 77471735
>>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>>> 3c8cae5000-3c8cce4000 ---p 000e5000 fd:00 77471735
>>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>>> 3c8cce4000-3c8ccec000 r--p 000e4000 fd:00 77471735
>>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>>> 3c8ccec000-3c8ccee000 rw-p 000ec000 fd:00 77471735
>>>>>>> /usr/lib64/libstdc++.so.6.0.17
>>>>>>> 3c8ccee000-3c8cd03000 rw-p 00000000 00:00 0
>>>>>>> 3c93a00000-3c93b9b000 r-xp 00000000 fd:00 77471858
>>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>>> 3c93b9b000-3c93d9b000 ---p 0019b000 fd:00 77471858
>>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>>> 3c93d9b000-3c93db5000 r--p 0019b000 fd:00 77471858
>>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>>> 3c93db5000-3c93dc0000 rw-p 001b5000 fd:00 77471858
>>>>>>> /usr/lib64/libcrypto.so.1.0.1e
>>>>>>> 3c93dc0000-3c93dc5000 rw-p 00000000 00:00 0
>>>>>>> 3ca1200000-3ca1225000 r-xp 00000000 fd:00 77471104
>>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>>> 3ca1225000-3ca1424000 ---p 00025000 fd:00 77471104
>>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>>> 3ca1424000-3ca1428000 r--p 00024000 fd:00 77471104
>>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>>> 3ca1428000-3ca1429000 rw-p 00028000 fd:00 77471104
>>>>>>> /usr/lib64/libtinfo.so.5.9
>>>>>>> 3ca2e00000-3ca2e23000 r-xp 00000000 fd:00 77471813
>>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>>> 3ca2e23000-3ca3022000 ---p 00023000 fd:00 77471813
>>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>>> 3ca3022000-3ca3023000 r--p 00022000 fd:00 77471813
>>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>>> 3ca3023000-3ca3024000 rw-p 00023000 fd:00 77471813
>>>>>>> /usr/lib64/libncurses.so.5.9
>>>>>>> 3ca3e00000-3ca3e16000 r-xp 00000000 fd:00 77470820
>>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>>> 3ca3e16000-3ca4015000 ---p 00016000 fd:00 77470820
>>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>>> 3ca4015000-3ca4016000 r--p 00015000 fd:00 77470820
>>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>>> 3ca4016000-3ca4017000 rw-p 00016000 fd:00 77470820
>>>>>>> /usr/lib64/libnsl-2.16.so
>>>>>>> 3ca4017000-3ca4019000 rw-p 00000000 00:00 0
>>>>>>> 3ca7600000-3ca7609000 r-xp 00000000 fd:00 77472018
>>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>>> 3ca7609000-3ca7808000 ---p 00009000 fd:00 77472018
>>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>>> 3ca7808000-3ca7809000 r--p 00008000 fd:00 77472018
>>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>>> 3ca7809000-3ca780a000 rw-p 00009000 fd:00 77472018
>>>>>>> /usr/lib64/libwrap.so.0.7.6
>>>>>>> 3ca780a000-3ca780b000 rw-p 00000000 00:00 0
>>>>>>> 7f9470000000-7f9470594000 rw-p 00000000 00:00 0
>>>>>>> 7f9470594000-7f9474000000 ---p 00000000 00:00 0
>>>>>>> 7f9474000000-7f9474021000 rw-p 00000000 00:00 0
>>>>>>> 7f9474021000-7f9478000000 ---p 00000000 00:00 0
>>>>>>> 7f9478000000-7f94782a5000 rw-p 00000000 00:00 0
>>>>>>> 7f94782a5000-7f947c000000 ---p 00000000 00:00 0
>>>>>>> 7f947c943000-7f947c944000 ---p 00000000 00:00 0
>>>>>>> 7f947c944000-7f947d144000 rw-p 00000000 00:00 0
>>>>>>> [stack:8999]
>>>>>>> 7f947d144000-7f947d145000 ---p 00000000 00:00 0
>>>>>>> 7f947d145000-7f947e146000 rw-p 00000000 00:00 0
>>>>>>> [stack:8998]
>>>>>>> 7f947e146000-7f947e152000 r-xp 00000000 fd:00 77471291
>>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>>> 7f947e152000-7f947e351000 ---p 0000c000 fd:00 77471291
>>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>>> 7f947e351000-7f947e352000 r--p 0000b000 fd:00 77471291
>>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>>> 7f947e352000-7f947e353000 rw-p 0000c000 fd:00 77471291
>>>>>>> /usr/lib64/libnss_files-2.16.so
>>>>>>> 7f947e353000-7f947e354000 ---p 00000000 00:00 0
>>>>>>> 7f947e354000-7f9486bbd000 rw-p 00000000 00:00 0
>>>>>>> [stack:8997]
>>>>>>> 7f9486bbd000-7f9486bd3000 r-xp 00000000 fd:00 77463716
>>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>>> 7f9486bd3000-7f9486dd3000 ---p 00016000 fd:00 77463716
>>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>>> 7f9486dd3000-7f9486dd4000 r--p 00016000 fd:00 77463716
>>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>>> 7f9486dd4000-7f9486dd5000 rw-p 00017000 fd:00 77463716
>>>>>>> /usr/lib64/libpthread-2.16.so
>>>>>>> 7f9486dd5000-7f9486dda000 rw-p 00000000 00:00 0
>>>>>>> 7f9486dda000-7f9486e60000 r-xp 00000000 fd:00 77470840
>>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>>> 7f9486e60000-7f948705f000 ---p 00086000 fd:00 77470840
>>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>>> 7f948705f000-7f9487061000 r--p 00085000 fd:00 77470840
>>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>>> 7f9487061000-7f9487064000 rw-p 00087000 fd:00 77470840
>>>>>>> /usr/lib64/libft.so.0.0.0
>>>>>>> 7f9487064000-7f948735a000 r-xp 00000000 fd:00 77599545
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>>> 7f948735a000-7f948755a000 ---p 002f6000 fd:00 77599545
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>>> 7f948755a000-7f948762c000 rw-p 002f6000 fd:00 77599545
>>>>>>> /usr/lib64/mysql/libmysqlclient.so.18.0.0
>>>>>>> 7f948762c000-7f9487632000 rw-p 00000000 00:00 0
>>>>>>> 7f948764f000-7f9487652000 rw-p 00000000 00:00 0
>>>>>>> 7fff16923000-7fff16944000 rw-p 00000000 00:00 0
>>>>>>> [stack]
>>>>>>> 7fff169fe000-7fff16a00000 r-xp 00000000 00:00 0
>>>>>>> [vdso]
>>>>>>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>>>>>>> [vsyscall]
>>>>>>> Aborted (core dumped)
>>>>>>>
>>>>>>> I run ldd against the image and I get this:
>>>>>>>
>>>>>>> ldd /usr/local/bin/rasqlinsert
>>>>>>> linux-vdso.so.1 => (0x00007fffaebfe000)
>>>>>>> libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x0000003c89200000)
>>>>>>> libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003c8aa00000)
>>>>>>> libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
>>>>>>> (0x00007f3f8c855000)
>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x0000003c8a200000)
>>>>>>> libft.so.0 => /lib64/libft.so.0 (0x00007f3f8c5cb000)
>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8c3ae000)
>>>>>>> libz.so.1 => /lib64/libz.so.1 (0x0000003c89e00000)
>>>>>>> libncurses.so.5 => /lib64/libncurses.so.5 (0x0000003ca2e00000)
>>>>>>> libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003ca1200000)
>>>>>>> libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003c8b600000)
>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x0000003c88e00000)
>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x0000003c89600000)
>>>>>>> librt.so.1 => /lib64/librt.so.1 (0x0000003c89a00000)
>>>>>>> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000003c8ca00000)
>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c8a600000)
>>>>>>> libcrypto.so.10 => /lib64/libcrypto.so.10 (0x0000003c93a00000)
>>>>>>> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003ca3e00000)
>>>>>>> libwrap.so.0 => /lib64/libwrap.so.0 (0x0000003ca7600000)
>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003c88a00000)
>>>>>>>
>>>>>>> Checking the dates of the dynamic libraries indicates that I am
>> linking
>>>>> to
>>>>>>> the most up to date versions.
>>>>>>>
>>>>>>> --Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>>>> Sent: Tuesday, August 20, 2013 7:17 PM
>>>>>>> To: David Edelman
>>>>>>> Cc: Argus
>>>>>>> Subject: Re: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>>>
>>>>>>> Hmmmm, I'll take another look tonight. It was working here with your
>>>>>>> file...frustrating !!!
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>> On Aug 20, 2013, at 6:15 PM, "David Edelman" <dedelman at iname.com>
>>> wrote:
>>>>>>>
>>>>>>>> Carter,
>>>>>>>>
>>>>>>>> I'm having the exact same problem as before.
>>>>>>>>
>>>>>>>> I did a clean install after changing the string in VERSION so that I
>>>>> knew
>>>>>>>> that I was using new code. I applied the argus_label.c change which
>>>>>> didn't
>>>>>>>> make any difference. I created .debug and .devel; make clobber,
>>>>> ./config;
>>>>>>>> make; make install and ran under gdb and it is the same picture.
>>>>>>>>
>>>>>>>> The instances of rasqlinsert taking data from radium are as happy as
>>>>>>> clams.
>>>>>>>>
>>>>>>>> What additional material can I collect for you?
>>>>>>>>
>>>>>>>> --Dave
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>>>>>>>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu]
>> On
>>>>>>>> Behalf Of Carter Bullard
>>>>>>>> Sent: Tuesday, August 20, 2013 10:59 AM
>>>>>>>> To: Argus
>>>>>>>> Subject: [ARGUS] new argus-clients-3.0.7.14 on the server
>>>>>>>>
>>>>>>>> Gentle people,
>>>>>>>> New client code up on the server. This release fixes all
>>>>>>>> known bugs that has been reported on the list, as well as
>>>>>>>> having major modifications to rapath().
>>>>>>>>
>>>>>>>> New code has been added as guards around the reported
>>>>>>>> label problems, but I am not sure that it has fixed
>>>>>>>> the problem. If we could test that, that would be great !!!
>>>>>>>>
>>>>>>>> We've made some big changes to rapath(). rapath() extracts
>>>>>>>> topology information from argus data. Basically it takes all
>>>>>>>> data that has ICMP TXD messages mapped to it, and tabulates path
>>>>>>>> information where it can. This has the effect of capturing all
>>>>>>>> traceroutes() that are observed by argus, regardless of the
>>>>>>>> techniqu; UDP, TCP or ICMP based, weather its vanilla or paris
>>> method,
>>>>>>>> or several of the proprietary strategies seen in intrusions.
>>>>>>>>
>>>>>>>> We've changed the default output of the graph that rapath.1
>>>>>>>> generates (using the -A option) to include the srcid, saddr
>>>>>>>> and daddr, so that you can build topology from just the
>>>>>>>> graphs. I'll add the stime and duration as well, but need
>>>>>>>> to figure out some command line options to control all these
>>>>>>>> new fields. Also rapath() is going to get a realtime mode,
>>>>>>>> currently, its a " read a file, generate some output " type of
>>>>>>>> tool.
>>>>>>>>
>>>>>>>> Please grab this code and give it a run. I'm hoping to
>>>>>>>> release 3.0.7.x as 3.0.8 in the next month, so if there are
>>>>>>>> any gotchas, don't hold back.
>>>>>>>>
>>>>>>>> Carter
>>>>>>> <rasqlinsert-Dump.txt><rasqlinsertLDD.txt>
>>>>>
>>>>> <rasqlinsert-gdb.txt>
>
More information about the argus
mailing list