Attaching rasqlinsert service to radium services

[David Edelman]

Since radium is ingesting the ipfix data and emitting Argus flow data in port 563. The rasqlinsert should use the -S localhost argument rather than the -C

Dave

Regards,
David Edelman 
who is responsible for the concept of this message. Unfortunately, autocorrect is responsible for the content

> On Apr 24, 2018, at 17:33, Carter Bullard <carter at qosient.com> wrote:
> 
> Hey Drew,
> To get flows per second ... use rabins connected to radium ...
>     rabins -S localhost -M time 1s nomodify hard -m srcid -s stime trans -B 5s
> 
> The trans number will be the number of flow records that were received in a second time period.
> 
> You are writing raw flow records into the database.  maybe mysql can handle 5-10K of those a second ??   Don’t use the cache mode and don’t use -D6, that will really slow rasalinsert down.  We don’t normally recommend writing raw data into a database.
> 
> Carter
> 	 	
> Carter Bullard • CTO
> 150 E 57th Street Suite 12D
> New York, New York 10022-2795
> Phone +1.212.588.9133 • Mobile +1.917.497.9494
> 
>> On Apr 24, 2018, at 2:00 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>> 
>> Hi Carter/All,
>> 
>> Is there a recommended way to attach rasqlinsert to radium? (my radium instances are being ran with -C and collecting IPFIX directly from network gear, in other words we aren't using the server portion of argus to generate flow data, yet...).  
>> 
>> Side question- is there a somewhat simple way to measure my flows per second by attaching to my radium services?
>> 
>> I've just starting testing around with this and I've added a "-P 563" to my radium service ExecStart string...so that rasqlinsert can attach there but due to the high volume of flows coming into radium and being made available via 0.0.0.0:563 it seems pretty shaky (or my understanding is shaky) at least as far as the socket queue/buffer is concerned.    
>> 
>> I've been playing around in testing using this when running rasqlinsert intial tests:
>> 
>> rasqlinsert -d -D6 -M time 1d -C 127.0.0.1:563 -w mysql://argus@localhost/binflows/umtest_%Y_%m_%d -M cache mysql_engine=InnoDB -m none
>> 
>> Is this more or less a sort of race condition where you have to empty (with rasqlinsert) the socket queue bucket (radium fills up) faster than it fills up, more or less?  With our rate of flows this seems like it might be not-so-stable, if the service emptying the bucket even died for a short duration that could mean lots of issues.
>> 
>> I ran into this before when testing deploying radium using sockets to make the data coming in off the wire available to other programs, but I started getting a ton of the error below like I did a bit yesterday when testing, which I tried to bump up the hard-coded queue size in the source code in the past and it didn't seem to help:
>> 
>> Apr 23 15:55:09 argus radium: radium[177562]: 19:55:09.297016 ArgusWriteOutSocket(0xb01314e0) max queue exceeded 500122
>> 
>> I also saw a very large number of the syntax error below mixed in with the other errors when testing this a bit yesterday:
>> 
>> ******************************************
>> Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.803491 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
>> Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.803656 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
>> Apr 23 13:39:34 argus rasqlinsert[93272]: 13:39:34.805425 mysql_real_query error Table 'binflows.umtest_2018_04_23' doesn't exist
>> ....snip....
>> Apr 23 13:41:54 argus rasqlinsert[93630]: 13:41:54.024842 mysql_real_query error Duplicate column name 'stime'
>> Apr 23 13:41:54 argus rasqlinsert[93630]: 13:41:54.036267 mysql_real_query error Query was empty
>> ******************************************
>> 
>> Many thanks,
>> 
>> -Drew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20180424/745f02c9/attachment.html>