Correlation rules

[Carter Bullard]

Hey Jaime,
Pretty large topic, and pretty open ended question.

In the argus project we have a lot of correlation capability, which I can speak to,
but generally you really do have to have a problem to solve, a hypothesis, if you will.
You really do need to know what you are trying to detect..

Argus, itself is a huge correlation engine, tracking potentially millions of
flows concurrently, using complex flow models, with a structured data
model, etc….  We call that micro flow correlation.  The rest of the tool
set performs flow correlation and macro flow correlation.

The primary purpose of micro flow correlation is to capture network activity,
in a timely fashion, with a maximum of semantic preservation and data reduction,
to support network ops, performance and security management.  Flow and
macroflow correlation are used to enhance the utility of the data for accounting
and billing, and network optimization, which is where the ops, performance and
security is being done.

There are two correlation strategies that the argus project focuses on, and each
enables different types of analytics.   The strategies are grouped into streaming,
or real-time correlation, and non-streaming, or non-realtime, data correlation.
The non-streaming is akin to all the forms of data mining against large data,
and so is pretty generic, like stream-block processing, meta-data
processing and aggregation.

In streaming correlation, the problem is all about trying to analyze multi-probe
observations of the same network activity.  A and B are talking to each other,
and argus sensors in A and B as well as network deployed argo that are along
the path, all see the connection and report on its activity.   Argus enables
real-time streaming data analysis through its radium() system, which can bring the
flow records from all the sensors into a common argus data stream.  This 
merged data source enables a single ra* program to align and process the
records all together, enabling differential metrics like:
   1) one-way network latency
   2) reachability and connectivity assessments
   3) loss measurements
   4) virtual path utilization metrics
   4) NAT object translations
   5) and more

These of course can provide a lot of insight if you’re worried about security,
and need to respond quickly.

The non-streaming correlation methods are focused on data organization,
data enhancement (labeling), aggregation and processing, to do behavioral
baselining and trend analysis.   The project does get bogged down in
making the infrastructure and data sources work, but we have quite a bit
of the rest.

The one that has gotten the most attention is labeling and flexible
aggregation strategies using tools like racluster(), rabins() and rahisto().
These tools support generalized aggregation with complex data preservation
methods that allow for discovery of persistent objects and trends.
rabins() is a time oriented tool, and rahisto() is a frequency distribution
oriented tool.

In the non-realtime, non-streaming world, there are a number of groups that
are using R with large argus data, and or importing argus data to drive some
machine learning things.  And the ra* tools help those guys.

So, what do you think you want to detect ???  That may help us have a
conversation about how you might want to do it.

Carter

On Nov 15, 2013, at 8:24 AM, Jaime Nebrera <jnebrera at eneotecnologia.com> wrote:

>  Hi all,
> 
>  Let me introduce myself. We are developing a correlation engine for redBorder, our open source security management platform.
> We are looking for sources of ideas for correlation rules in the security and netflow areas. Craig Merchant has suggested us to
> query Argus list as its quite active and has a lot of knowledge inside.
> 
>  Yes, I have already looked at http://mbrownnyc.wordpress.com/2013/05/21/anomaly-detection-creating-baselines-and-determining-statistical-outliers-in-argus-data/ 
> as well as the source suggested in that area.
> 
>  We have also looked at Sec, OSSIM, Sagan, and others.
> 
>  Any ideas or suggestions?
> 
>  In particular, to start with we are interested in netflow specific rules.
> 
>  Kind regards
> 
> -- 
> Jaime Nebrera - jnebrera at eneotecnologia.com
> Consultor TI - ENEO Tecnologia SL
> C/ Manufactura 2, Edificio Euro, Oficina 3N
> Mairena del Aljarafe - 41927 - Sevilla
> Telf.- 955 60 11 60 / 619 04 55 18
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7322 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131115/6108497d/attachment.bin>