Argus and "Big Data" solutions

[Carter Bullard]

So I have never been a fan of putting primitive flow data into a database or Splunk or ArcSight, regardless of the application.  There were a number of presentations at the last FloCon, one from Markus De Shon, a Google researcher, about MapReduce and flow data, and there was a paper about Scalable Netflow Analysis with Hadoop. Both indicated that Hadoop had its pluses and minuses for flow data queries.

I don’t think I ever said, or that I would ever say, that aggregating argus for security analysis was problematic.  On the contrary, its pretty straight forward and very useful.
The hard part is knowing what to look for, and building a data model that will make it fast and easy to find what you think is important.

Carter


On Oct 28, 2013, at 10:37 PM, Craig Merchant <cmerchant at responsys.com> wrote:

> Hey, Carter…  I wanted to follow up on a conversation we had a few weeks back online…
>  
> As you know, we’re using Splunk as our primary tool for processing and visualizing our Argus data.  In their license model, you pay for the amount of data you index per day.  They are also coming out with a connector in February or so that will allow Splunk to query data in hadoop using its native search language.
>  
> Our unaggregated flow data would probably be in the neighborhood of 80 GB per day, which could cost as much $100K if we bought the license.  To make Argus affordable to index, we are considering taking a look at getting Argus data into something like Cloudera and/or using racluster to aggregate the flows and reduce the volume.
>  
> In our offline conversation, you said that Hadoop wasn’t an appropriate file system for Argus data.  I looked around for any presentations you did at FloCon about it, but I couldn’t find anything.  What do you see as the limitations on a Hadoop-like system for analyzing Argus data?
>  
> You also said that aggregating your data for security analysis was problematic.  I was just curious what your reasoning for that was.
>  
> Thanks!
> 
> Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131030/562ab6a3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131030/562ab6a3/attachment.bin>