radium threading for one dat stream

[Carter Bullard via Argus-info]

Hey Richard,
Carter

> On Mar 16, 2016, at 9:35 PM, Richard Rothwell via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
> 
> Hi Carter,
> 
> I am looking now at scaling of Argus radium for our main target router.
> A couple of questions:
> 
> 1.
> The Juniper outer is set up for sampling at a 1:10 rate on the recommendation of Juniper.
> Is router sampling an issue for creating the Argus bidirectional records?

You will need an aggregator to merge the Juniper unidirectional flow records to form argus bidirectional records.  Because you are starting with Jflow records, and will not be tracking loss, packet dynamics, etc…, you will not have any issues that would arise from sampling.  You should expect, however, a significant number of partial flows, especially from DNS, dhcp, ntp, etc…, as the Juniper will likely not report either the request or response for these small packet transactions.

> 
> 2. 
> Testing on a small traffic router repressing about 2% of total traffic I am seeing 6% average CPU load.
> For our big router representing 20% of total traffic I would expect the CPU load to be 60%.
> The server is running 8 cores so it should be able to provide 800% load.
> So can radium utilise all cores (800%) when it is processing just one stream?
> Or is radium arranged so that there is one thread per stream?

radium has input threads, a thread to manage the connections to its data sources, a non-blocking DNS lookup thread, if needed, a thread for labeling analytics, and an output thread, so it can scale up to a point.  radium is trying to take in a number of streams, generate a single output stream, and then provide that stream to any number of readers, so there is currently a single thread bottle neck to generate the single output stream.

I process 1M records per second on modest resources, but it can go much faster with proper data flow organization.  Many sites run multiple radii, sometimes one per input, especially if they are doing the same processing, and manage the multiple output streams by having readers attach to the radii of interest.

> 
> Regards from Richard
> 

Hope all is most excellent !!!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20160317/2bc9afc5/attachment.html>