Argus and rasqlinsert problems

[Leif Tishendorf]

Oh, sorry, forgot to mention there are no mysql error logs (I mean there 
is, but there isn't any errors in it), and I call rasqlinsert with the 
following:

rasqlinsert -M cache -S localhost:565 -w 
mysql://argus@localhost/argus/argus_%Y_%m_%d -M time 1d -d

-Leif

On 4/15/2011 1:07 PM, Carter Bullard wrote:
> Hey Lief,
> Well, that is disappointing.  I would recommend that you shift back,
> to get something stable going,  and I'll work with you to get things so
> you can go the database route.
>
> I am not seeing this type of instability, but that doesn't mean anything.
>
> First, things first.  We need to fix the argus seg faulting.  Did this start
> with argus-3.0.4, or with the radium() connection approach?
>
> If you can run gdb(), the best thing would be to run argus under gdb,
> after compiling the symbols in, so it will tell us where it is dying.  In
> the argus root directory:
>
>     % touch .devel .debug
>     % ./configure; make clean; make
>     % sudo gdb ./bin/argus
>     (gdb)
>
> Stop your running argus, and then run the argus under gdb.
> Assuming that your argus was running as a daemon, use the -d switch
> when running argus, so that it won't go into the background while in gdb:
>
>     (gdb) argus -d
>
> Hopefully it will cough up blood and tell us where it was.  That should
> help me to fix that.
>
> Rather than have rasqlinsert() connect to a remote radium(), you can
> radium() on the database system, connecting to the other radium(), and
> have rasqlinsert() attach to a local radium.  That may or may not help,
> but it at least leaves record distribution to radium, and lets the other
> programs have local access to data.
>
> With rasqlinsert(), there are a few possibilities.  When the CPU goes
> down, has rasqlinsert() stopped inserting records into the database?
> It may be having problems receiving records, or it could be having
> problems with mysqld.
>
> Are there any error messages in your mysqld error logs?
>
> Sometimes its hard to find where the logs are.  I use:
>     lsof -n | fgrep mysql
> to show me where the directory is. You may have to be root to see.
>
> How are you calling rasqlinsert?
>
> If you would like to take this off the email list, feel free to email me
> directly, although it is late on Friday, I'll still read some email this
> weekend.
>
> Carter
>
> On Apr 15, 2011, at 2:42 PM, Leif Tishendorf wrote:
>
>> Hey Carter,
>>
>> I've change how we're logging argus data from regular files to a MySQL
>> DB.  We used to have 3 Argus instances collected by one Radium
>> instance and then logged to disk by rasplit, and it was all working
>> fine.  Now everything is the same except instead of rasplit we use
>> rasqlinsert and instead of logging local rasqlinsert is running on
>> another system connecting to the radium instance via a private address
>> direct link.
>>
>> The first issue I noticed was every few minutes the argus instances
>> were dieing(not necessarily at the same time) with the following
>> syslog error:
>>
>> kernel: [4374754.132368] argus[28333]: segfault at 188 ip
>> 00007f27b7e61f7c sp 00007f27a63e7828 error 6 in
>> libc-2.12.1.so[7f27b7ddb000+17a000]
>>
>> Then the second issue we're having is rasqlinsert will work fine and
>> then we'll see CPU/RAM usage decline over about 30 seconds until it's
>> eventually no longer inserting new argus records.  We can get it
>> working again (without touching the running rasqlinsert instance) by
>> sometimes restarting radium and sometimes restarting the argus
>> instances and sometimes it takes both.  but after a minute or so it
>> all happens again.  The crashes don't coincide with the inserts
>> stopping, although they do sometimes fix it when my monitor scripts
>> restart the argus instances.
>>
>> I'm currently running Argus version 3.0.4 and Argus-clients 3.0.5.5
>>
>> Any ideas on where I should start troubleshooting this?
>>
>> Thanks,
>>
>> -Leif
>>
>