Correlation rules

[Jaime Nebrera]

Hi Carter,

Thanks from your prompt reply. At this moment we will focus on what you
called macro level correlation.

Here are some ideas:

* Link saturation / loss
* Connection to reputation IP, domain, url
* Connection to unprivileged ports
* Activation of privileged ports in our network
* Outlier detection in time series
* Excess of SIN packets
* Surge in error packets
* IP sweep
* Port scan
* Changes of entropy in certain variables both globally and per IP
* Change in ip profile, from consumer to producer or the other way around
* Direct connection to outside dns, Web, mail when existing inside
* Change in gateway IP
* Weird usage of light protocols like dns, icmp or voip
* Change in usual traffic ratios like syn to syn/ack or tcp/total,
successful sessions
* Botnet C&C
* Worm propagation
* Abuse of dns and authentication protocols (LDAP,...)

So you see some are already specific, some just name the goal

Of course here we miss a lot :)

Regards

Jaime Nebrera - ENEO Tecnología
Sent with mobile, sorry for typos
El 15/11/2013 17:03, "Carter Bullard" <carter at qosient.com> escribió:

> Hey Jaime,
> Pretty large topic, and pretty open ended question.
>
> In the argus project we have a lot of correlation capability, which I can
> speak to,
> but generally you really do have to have a problem to solve, a hypothesis,
> if you will.
> You really do need to know what you are trying to detect..
>
> Argus, itself is a huge correlation engine, tracking potentially millions
> of
> flows concurrently, using complex flow models, with a structured data
> model, etc….  We call that micro flow correlation.  The rest of the tool
> set performs flow correlation and macro flow correlation.
>
> The primary purpose of micro flow correlation is to capture network
> activity,
> in a timely fashion, with a maximum of semantic preservation and data
> reduction,
> to support network ops, performance and security management.  Flow and
> macroflow correlation are used to enhance the utility of the data for
> accounting
> and billing, and network optimization, which is where the ops, performance
> and
> security is being done.
>
> There are two correlation strategies that the argus project focuses on,
> and each
> enables different types of analytics.   The strategies are grouped into
> streaming,
> or real-time correlation, and non-streaming, or non-realtime, data
> correlation.
> The non-streaming is akin to all the forms of data mining against large
> data,
> and so is pretty generic, like stream-block processing, meta-data
> processing and aggregation.
>
> In streaming correlation, the problem is all about trying to analyze
> multi-probe
> observations of the same network activity.  A and B are talking to each
> other,
> and argus sensors in A and B as well as network deployed argo that are
> along
> the path, all see the connection and report on its activity.   Argus
> enables
> real-time streaming data analysis through its radium() system, which can
> bring the
> flow records from all the sensors into a common argus data stream.  This
> merged data source enables a single ra* program to align and process the
> records all together, enabling differential metrics like:
>    1) one-way network latency
>    2) reachability and connectivity assessments
>    3) loss measurements
>    4) virtual path utilization metrics
>    4) NAT object translations
>    5) and more
>
> These of course can provide a lot of insight if you’re worried about
> security,
> and need to respond quickly.
>
> The non-streaming correlation methods are focused on data organization,
> data enhancement (labeling), aggregation and processing, to do behavioral
> baselining and trend analysis.   The project does get bogged down in
> making the infrastructure and data sources work, but we have quite a bit
> of the rest.
>
> The one that has gotten the most attention is labeling and flexible
> aggregation strategies using tools like racluster(), rabins() and
> rahisto().
> These tools support generalized aggregation with complex data preservation
> methods that allow for discovery of persistent objects and trends.
> rabins() is a time oriented tool, and rahisto() is a frequency distribution
> oriented tool.
>
> In the non-realtime, non-streaming world, there are a number of groups that
> are using R with large argus data, and or importing argus data to drive
> some
> machine learning things.  And the ra* tools help those guys.
>
> So, what do you think you want to detect ???  That may help us have a
> conversation about how you might want to do it.
>
> Carter
>
> On Nov 15, 2013, at 8:24 AM, Jaime Nebrera <jnebrera at eneotecnologia.com>
> wrote:
>
> >  Hi all,
> >
> >  Let me introduce myself. We are developing a correlation engine for
> redBorder, our open source security management platform.
> > We are looking for sources of ideas for correlation rules in the
> security and netflow areas. Craig Merchant has suggested us to
> > query Argus list as its quite active and has a lot of knowledge inside.
> >
> >  Yes, I have already looked at
> http://mbrownnyc.wordpress.com/2013/05/21/anomaly-detection-creating-baselines-and-determining-statistical-outliers-in-argus-data/
> > as well as the source suggested in that area.
> >
> >  We have also looked at Sec, OSSIM, Sagan, and others.
> >
> >  Any ideas or suggestions?
> >
> >  In particular, to start with we are interested in netflow specific
> rules.
> >
> >  Kind regards
> >
> > --
> > Jaime Nebrera - jnebrera at eneotecnologia.com
> > Consultor TI - ENEO Tecnologia SL
> > C/ Manufactura 2, Edificio Euro, Oficina 3N
> > Mairena del Aljarafe - 41927 - Sevilla
> > Telf.- 955 60 11 60 / 619 04 55 18
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131115/02c8876e/attachment.html>