Attaching rasqlinsert service to radium services

[Drew Dixon]

Hi Carter/All,

Is there a recommended way to attach rasqlinsert to radium? (my radium
instances are being ran with -C and collecting IPFIX directly from network
gear, in other words we aren't using the server portion of argus to
generate flow data, yet...).

Side question- is there a somewhat simple way to measure my flows per
second by attaching to my radium services?

I've just starting testing around with this and I've added a "-P 563" to my
radium service ExecStart string...so that rasqlinsert can attach there but
due to the high volume of flows coming into radium and being made available
via 0.0.0.0:563 it seems pretty shaky (or my understanding is shaky) at
least as far as the socket queue/buffer is concerned.

I've been playing around in testing using this when running rasqlinsert
intial tests:

rasqlinsert -d -D6 -M time 1d -C 127.0.0.1:563 -w
mysql://argus@localhost/binflows/umtest_%Y_%m_%d
-M cache mysql_engine=InnoDB -m none

Is this more or less a sort of race condition where you have to empty (with
rasqlinsert) the socket queue bucket (radium fills up) faster than it fills
up, more or less?  With our rate of flows this seems like it might be
not-so-stable, if the service emptying the bucket even died for a short
duration that could mean lots of issues.

I ran into this before when testing deploying radium using sockets to make
the data coming in off the wire available to other programs, but I started
getting a ton of the error below like I did a bit yesterday when testing,
which I tried to bump up the hard-coded queue size in the source code in
the past and it didn't seem to help:

Apr 23 15:55:09 argus radium: radium[177562]: 19:55:09.297016
ArgusWriteOutSocket(0xb01314e0) max queue exceeded 500122

I also saw a very large number of the syntax error below mixed in with the
other errors when testing this a bit yesterday:

******************************************
Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.803491 mysql_real_query
error You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'' at line 1
Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.803656 mysql_real_query
error You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'' at line 1
Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.805425 mysql_real_query
error Table 'binflows.umtest_2018_04_23' doesn't exist
....snip....
Apr 23 13:41:54 argus rasqlinsert[93630]: 13:41:54.024842 mysql_real_query
error Duplicate column name 'stime'
Apr 23 13:41:54 argus rasqlinsert[93630]: 13:41:54.036267 mysql_real_query
error Query was empty
******************************************

Many thanks,

-Drew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20180424/6070cc24/attachment.html>