rasqlinsert disconnect issues
Carter Bullard
carter at qosient.com
Mon Oct 6 08:16:22 EDT 2014
is rasqlinsert done at this point or is it suppose to be continuing ??
looks like a thread deleted the queue, and someone else thinks its still needs processing ??? how many other threads ??
Carter
> On Oct 5, 2014, at 7:12 PM, David Edelman <dedelman at iname.com> wrote:
>
> A few more details:
>
> keep = 1
> cnt = 65
> x = 2
> qhdr = 0x0
> queue->array[0] = 0xb250ac28
> queue->array[1] = 0x0 as are the remaining elements of queue->array[]
>
> —Dave
>
>
> From: Carter Bullard <carter at qosient.com>
> Date: Sunday, October 5, 2014 at 9:46 PM
> To: David Edelman <dedelman at iname.com>
> Cc: Argus <argus-info at lists.andrew.cmu.edu>
> Subject: Re: [ARGUS] rasqlinsert disconnect issues
>
> Hey Dave,
> Sorry for the delayed response, just getting email and not yet near a machine. Let me look at this tonight when I get back !!!
> Thanks !!!!
> Carter
>
> Carter Bullard, QoSient, LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>> On Oct 4, 2014, at 8:27 PM, David Edelman <dedelman at iname.com> wrote:
>>
>> GDB to the rescue, I have the session hanging out in a screens instance if you need anything specific, just let me know.
>>
>> —Dave
>>
>> (gdb) run -M time 1d -M cache -S localhost:561 -w mysql://argus@localhost/argus/test2macAddrs_%Y_%m_%d -m srcid saddr smac -s stime ltime srcid saddr smac -M rmon - ipv4
>> Starting program: /layered_products/argus-clients-3.0.8/bin/rasqlinsert -M time 1d -M cache -S localhost:561 -w mysql://argus@localhost/argus/test2macAddrs_%Y_%m_%d -m srcid saddr smac -s stime ltime srcid saddr smac -M rmon - ipv4
>> [Thread debugging using libthread_db enabled]
>> [New Thread 0xb7fe68d0 (LWP 21287)]
>> Detaching after fork from child process 21288.
>> [New Thread 0xb720bb90 (LWP 21289)]
>> [Thread 0xb720bb90 (LWP 21289) exited]
>> [New Thread 0xb720bb90 (LWP 21297)]
>> [New Thread 0xb6709b90 (LWP 21298)]
>> [New Thread 0xb5d08b90 (LWP 21299)]
>> [New Thread 0xb5307b90 (LWP 21300)]
>> [New Thread 0xb4906b90 (LWP 21301)]
>> [New Thread 0xb3f05b90 (LWP 21302)]
>> [New Thread 0xb3504b90 (LWP 21303)]
>> [Thread 0xb3504b90 (LWP 21303) exited]
>> rasqlinsert[21287]: Sun 2014-10-05 00:00:14.081 mysql_real_query error Query was empty
>> rasqlinsert[21287]: Sun 2014-10-05 00:00:50.374 mysql_real_query error Query was empty
>> rasqlinsert[21287]: Sun 2014-10-05 00:00:53.109 mysql_real_query error Query was empty
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0xb5307b90 (LWP 21300)]
>> 0x08052827 in RaClientSortQueue (sorter=0x977e8b8, queue=0x977e690, type=2) at ./raclient.c:2523
>> 2523 qhdr = qhdr->nxt;
>> Missing separate debuginfos, use: debuginfo-install libgcc-4.3.2-7.i386
>> (gdb) list
>> 2518 keep = 0;
>> 2519 }
>> 2520
>> 2521 if (keep)
>> 2522 queue->array[x++] = qhdr;
>> 2523 qhdr = qhdr->nxt;
>> 2524 }
>> 2525
>> 2526 queue->array[i] = NULL;
>> 2527
>> (gdb)
>>
>>
>>
>> From: David Edelman <dedelman at iname.com>
>> Date: Thursday, October 2, 2014 at 4:50 PM
>> To: Carter Bullard <carter at qosient.com>
>> Cc: Argus <argus-info at lists.andrew.cmu.edu>
>> Subject: Re: [ARGUS] rasqlinsert disconnect issues
>>
>> Okay, I'll try that.
>>
>>
>>
>>> On Oct 2, 2014, at 00:34, Carter Bullard <carter at qosient.com> wrote:
>>>
>>> D3 should print the sql calls, which maybe all that you need to see.
>>> Carter
>>>
>>>> On Oct 1, 2014, at 9:46 PM, David Edelman <dedelman at iname.com> wrote:
>>>>
>>>> Carter,
>>>>
>>>> I’m running rasqlinsert using –S localhost:561 to process the output of radium which is doing labeling. I’ll fire up an instance of the release code with the same parameter but a different table name and see if there is anything obvious. I’ve built the release code with both .debug and .devel do you have a recommendation for a debug value?
>>>>
>>>> —Dave
>>>>
>>>> From: Carter Bullard <carter at qosient.com>
>>>> Date: Monday, September 29, 2014 at 1:51 PM
>>>> To: David Edelman <dedelman at iname.com>
>>>> Cc: "John T. Myers" <myersj0 at gmail.com>, Argus <argus-info at lists.andrew.cmu.edu>
>>>> Subject: Re: [ARGUS] rasqlinsert disconnect issues
>>>>
>>>> Hey Dave,
>>>> Well, that’s not what we’re striving for, so
>>>> if we can capture what that is all about, I’ll
>>>> fix as soon as I can.
>>>>
>>>> Carter
>>>>
>>>>> On Sep 28, 2014, at 5:47 PM, David Edelman <dedelman at iname.com> wrote:
>>>>>
>>>>> The running but seems to no update the database was a problem that I reported and you fixed in one of the very last release candidates. The current 3.0.8 does not have that problem as best I can tell but it does stop after between 8-20 hours with no log messages that I have been able to find.
>>>>>
>>>>> --Dave
>>>>>
>>>>> From: Carter Bullard [mailto:carter at qosient.com]
>>>>> Sent: Sunday, September 28, 2014 11:34 AM
>>>>> To: David Edelman
>>>>> Cc: John T. Myers; Argus
>>>>> Subject: Re: [ARGUS] rasqlinsert disconnect issues
>>>>>
>>>>> Well the earlier rasqlinsert.1 code does have a problem that the new one tried
>>>>> to fix. I’ll try to see what may be up with mysql_real_query() error messages,
>>>>> and see if we missed something there.
>>>>>
>>>>> But in the OP, the problem is that rasqlinsert.1 is running, but not updating
>>>>> the database, in this second report, rasqlinsert.1 is failing ??
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>>> On Sep 25, 2014, at 8:11 PM, David Edelman <dedelman at iname.com> wrote:
>>>>>>
>>>>>>
>>>>>> Carter,
>>>>>>
>>>>>> I am seeing something similar with Netflow data processed by Radium (labels added) and rasqlinsert reading the processed radium data from port 561. In my case the rasqlinsert process dies without any error messages (it is built with .debug and .devel) It isn’t practical for me to enable debuigger output since the failure can be many hours into the run.
>>>>>>
>>>>>> The one clue that I do have it that the release candidate set (3.0.8-rc1) worked fine.
>>>>>>
>>>>>> I haven’t had time to do much more than go back to the working release.
>>>>>>
>>>>>> --Dave
>>>>>>
>>>>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On Behalf Of Carter Bullard
>>>>>> Sent: Thursday, September 25, 2014 1:34 PM
>>>>>> To: John T. Myers
>>>>>> Cc: Argus
>>>>>> Subject: Re: [ARGUS] rasqlinsert disconnect issues
>>>>>>
>>>>>> Hey John,
>>>>>> rasqlinsert() is multi-threaded, and its possible that
>>>>>> the cache concurrency thread, the one that is managing the
>>>>>> database cache, exited if the database fails.
>>>>>>
>>>>>> rasqlinsert() should close, if that thread is done.
>>>>>>
>>>>>> So you are seeing rasqlinsert() is still running, but
>>>>>> not updating the database ?? Is rasqlinsert() getting
>>>>>> bigger ??? (should be generating INSERT and UPDATE
>>>>>> requests, but the DB thread is not processing them ???)
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>>
>>>>>> On Sep 25, 2014, at 10:35 AM, John T. Myers <myersj0 at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I was wondering if it’s normal behavior for rasqlinsert to cease inserting netflow into the database if the connection becomes interrupted? It seems if the mysql database is restarted or any part of the connection between rasqlinsert and the db is broken, it will not attempt to re-connect and continue flow insertion.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20141006/35ded549/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2443 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20141006/35ded549/attachment.bin>
More information about the argus
mailing list