Verifying flow to biflow conversion?

[Carter Bullard via Argus-info]

Hey Richard,
ra.1 doesn’t do record merging / aggregation.  To convert uni-flow data to bi-directional data you have to do aggregation.  To convert uni-flow data to bi-directional data, you have read in data, buffer it for some period of time (1-5s) so that any and all records can arrive, merge them together, and the output a single sorted stream of aggregated data.  That is a little complicated, and why rabins.1 has so many options, but you can’t do it without buffering the data.   rabins.1 is designed specifically for aggregation of streaming data, and it works great with argus data, because argus is designed specifically to support stream processing.  That’s why InSight is an argus application, not a netflow application.

As long as you are trying to drive InSight with netflow/IPFIX, you will be struggling to do what InSight was designed to do.
 
I’ve included Greg Cole, the creator of InSight, on this email, and hopefully he can fill in any gaps.  Greg designed InSight to provide visualization and reporting of end-to-end performance for his global network customers.  For those on the list, GLORIAD is one of the longest reach International Research networks and connects Russia, China, Korea, Nordic and US educational networks together.  It is a very cool network.

Greg tried for almost a decade to do something like InSight with netflow data, and was unsuccessful, but accomplished what he wanted to do with argus data, in only a matter of months.  Once he realized that there was a network data source that could provide him with the information he needed, and the tools to work with, he coupled it with ELK to get the screens and developed a great data flow architecture to give him the analytics need to do near real-time awareness of any traffic in his network.

The entire InSight effort basically demonstrated that integrated netflow data can’t support effective real-time end-to-end performance awareness.   I just want to suggest that you’re not having problems because of the argus tools.  You are having problems because of the limitations of your starting material.  Netflow data from routers is not an effective source of real-time network information.

If you want to run InSight, you really should try to generate Argus data from packets, not from netflow data.

Carter

> On Jul 21, 2016, at 8:47 PM, Richard Rothwell via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
> 
> Thanks Carter,
>  
> I was beginning to suspect that my change from using rabins to ra might be the issue.
>  
> It was never clear to me why GLORIAD InSight used rabins instead of ra.
> On the face of it rabins does splitting and aggregation .
> Since ElasticSearch does aggregation and the router does splitting of flows, the rabins functions seemed redundant.
> Oh well, looks like I change back to rabins.
>  
> Is kind of disappointing that ra can’t do the conversion to biflows, given that it can take in Netflow 9 flows.
>  
> I sort of know that using netflow / IPFIX is not ideal.
> But since I am connecting to routers that provide flows in that format, that I am stuck with it.
>  
> I might be missing something here though?
> Can Argus connect to a router? I thought its function was to connect to a network interface??
> I don’t have any network interfaces to connect to that allow me to probe the traffic.
>  
> Regards
>  
>  
> Hey Richard,
> ra.1 does not perform flow record aggregation, so you shouldn't expect any bi-directional flows from ra.1.  You will need to use racluster.1, rabins.1, ratop.1 or rasqlinsert.1.  These are flow aggregators.  By default they will merge matching records into bi-directional flows.
>  
> If you are hoping for good bi-directional flow data, netflow / IPFIX is a very poor starting point.  Argus generates bi-directional flows and I would recommend using argus ... If you have to use nprobe, you can run argus and nprobe on the same box, that does work pretty well in many situations.
>  
> Carter
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20160722/2327cb95/attachment.html>