Attaching rasqlinsert service to radium services
carter at qosient.com
Tue Apr 24 17:33:37 EDT 2018
To get flows per second ... use rabins connected to radium ...
rabins -S localhost -M time 1s nomodify hard -m srcid -s stime trans -B 5s
The trans number will be the number of flow records that were received in a second time period.
You are writing raw flow records into the database. maybe mysql can handle 5-10K of those a second ?? Don’t use the cache mode and don’t use -D6, that will really slow rasalinsert down. We don’t normally recommend writing raw data into a database.
Carter Bullard • CTO
150 E 57th Street Suite 12D
New York, New York 10022-2795
Phone +1.212.588.9133 • Mobile +1.917.497.9494
> On Apr 24, 2018, at 2:00 PM, Drew Dixon <dwdixon at umich.edu> wrote:
> Hi Carter/All,
> Is there a recommended way to attach rasqlinsert to radium? (my radium instances are being ran with -C and collecting IPFIX directly from network gear, in other words we aren't using the server portion of argus to generate flow data, yet...).
> Side question- is there a somewhat simple way to measure my flows per second by attaching to my radium services?
> I've just starting testing around with this and I've added a "-P 563" to my radium service ExecStart string...so that rasqlinsert can attach there but due to the high volume of flows coming into radium and being made available via 0.0.0.0:563 it seems pretty shaky (or my understanding is shaky) at least as far as the socket queue/buffer is concerned.
> I've been playing around in testing using this when running rasqlinsert intial tests:
> rasqlinsert -d -D6 -M time 1d -C 127.0.0.1:563 -w mysql://argus@localhost/binflows/umtest_%Y_%m_%d -M cache mysql_engine=InnoDB -m none
> Is this more or less a sort of race condition where you have to empty (with rasqlinsert) the socket queue bucket (radium fills up) faster than it fills up, more or less? With our rate of flows this seems like it might be not-so-stable, if the service emptying the bucket even died for a short duration that could mean lots of issues.
> I ran into this before when testing deploying radium using sockets to make the data coming in off the wire available to other programs, but I started getting a ton of the error below like I did a bit yesterday when testing, which I tried to bump up the hard-coded queue size in the source code in the past and it didn't seem to help:
> Apr 23 15:55:09 argus radium: radium: 19:55:09.297016 ArgusWriteOutSocket(0xb01314e0) max queue exceeded 500122
> I also saw a very large number of the syntax error below mixed in with the other errors when testing this a bit yesterday:
> Apr 23 13:39:34 argus rasqlinsert: 13:39:34.803491 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
> Apr 23 13:39:34 argus rasqlinsert: 13:39:34.803656 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
> Apr 23 13:39:34 argus rasqlinsert: 13:39:34.805425 mysql_real_query error Table 'binflows.umtest_2018_04_23' doesn't exist
> Apr 23 13:41:54 argus rasqlinsert: 13:41:54.024842 mysql_real_query error Duplicate column name 'stime'
> Apr 23 13:41:54 argus rasqlinsert: 13:41:54.036267 mysql_real_query error Query was empty
> Many thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the argus