Argus and "Big Data" solutions

[Craig Merchant]

Hey, Carter...  I wanted to follow up on a conversation we had a few weeks back online...

As you know, we're using Splunk as our primary tool for processing and visualizing our Argus data.  In their license model, you pay for the amount of data you index per day.  They are also coming out with a connector in February or so that will allow Splunk to query data in hadoop using its native search language.

Our unaggregated flow data would probably be in the neighborhood of 80 GB per day, which could cost as much $100K if we bought the license.  To make Argus affordable to index, we are considering taking a look at getting Argus data into something like Cloudera and/or using racluster to aggregate the flows and reduce the volume.

In our offline conversation, you said that Hadoop wasn't an appropriate file system for Argus data.  I looked around for any presentations you did at FloCon about it, but I couldn't find anything.  What do you see as the limitations on a Hadoop-like system for analyzing Argus data?

You also said that aggregating your data for security analysis was problematic.  I was just curious what your reasoning for that was.

Thanks!

Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131029/db4e412c/attachment.html>