Argus and rasqlinsert problems

[Carter Bullard]

Hey Lief,
Well, that is disappointing.  I would recommend that you shift back,
to get something stable going,  and I'll work with you to get things so
you can go the database route.  

I am not seeing this type of instability, but that doesn't mean anything.

First, things first.  We need to fix the argus seg faulting.  Did this start
with argus-3.0.4, or with the radium() connection approach?

If you can run gdb(), the best thing would be to run argus under gdb,
after compiling the symbols in, so it will tell us where it is dying.  In
the argus root directory:

   % touch .devel .debug
   % ./configure; make clean; make
   % sudo gdb ./bin/argus
   (gdb)

Stop your running argus, and then run the argus under gdb.
Assuming that your argus was running as a daemon, use the -d switch
when running argus, so that it won't go into the background while in gdb:

   (gdb) argus -d

Hopefully it will cough up blood and tell us where it was.  That should
help me to fix that.

Rather than have rasqlinsert() connect to a remote radium(), you can
radium() on the database system, connecting to the other radium(), and
have rasqlinsert() attach to a local radium.  That may or may not help,
but it at least leaves record distribution to radium, and lets the other
programs have local access to data.

With rasqlinsert(), there are a few possibilities.  When the CPU goes
down, has rasqlinsert() stopped inserting records into the database?
It may be having problems receiving records, or it could be having
problems with mysqld.

Are there any error messages in your mysqld error logs?

Sometimes its hard to find where the logs are.  I use:
   lsof -n | fgrep mysql
to show me where the directory is. You may have to be root to see.

How are you calling rasqlinsert?

If you would like to take this off the email list, feel free to email me
directly, although it is late on Friday, I'll still read some email this
weekend.

Carter

On Apr 15, 2011, at 2:42 PM, Leif Tishendorf wrote:

> Hey Carter,
> 
> I've change how we're logging argus data from regular files to a MySQL
> DB.  We used to have 3 Argus instances collected by one Radium
> instance and then logged to disk by rasplit, and it was all working
> fine.  Now everything is the same except instead of rasplit we use
> rasqlinsert and instead of logging local rasqlinsert is running on
> another system connecting to the radium instance via a private address
> direct link.
> 
> The first issue I noticed was every few minutes the argus instances
> were dieing(not necessarily at the same time) with the following
> syslog error:
> 
> kernel: [4374754.132368] argus[28333]: segfault at 188 ip
> 00007f27b7e61f7c sp 00007f27a63e7828 error 6 in
> libc-2.12.1.so[7f27b7ddb000+17a000]
> 
> Then the second issue we're having is rasqlinsert will work fine and
> then we'll see CPU/RAM usage decline over about 30 seconds until it's
> eventually no longer inserting new argus records.  We can get it
> working again (without touching the running rasqlinsert instance) by
> sometimes restarting radium and sometimes restarting the argus
> instances and sometimes it takes both.  but after a minute or so it
> all happens again.  The crashes don't coincide with the inserts
> stopping, although they do sometimes fix it when my monitor scripts
> restart the argus instances.
> 
> I'm currently running Argus version 3.0.4 and Argus-clients 3.0.5.5
> 
> Any ideas on where I should start troubleshooting this?
> 
> Thanks,
> 
> -Leif
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110415/1046a8a4/attachment.bin>