Large scale Argus implementation

[Craig Merchant]

Hey, Carter.

 

Now that Responsys has become a part of the Oracle Cloud, I have been tasked with implementing a network flow solution for all of the cloud infrastructure.  Initially, we would only monitor ingress/egress traffic, but eventually we want to monitor internal traffic as well.

 

We currently have about 25 data centers and approximately 100,000 hosts.  The Responsys data centers are small in comparison to some of the other cloud data centers and our internal traffic volume is around 10-15 Gbps.  So, I would imagine that the rest of data centers would generate something like 10x to 15x that traffic.

 

After reading a bit about scalability and the rasql tools, I'm pretty sure that the volume of traffic would overwhelm a relational database.  We can get Oracle Big Data Appliances fairly cheap (all things considered).  I'm thinking that the best approach may be to write the ASCII output of ra tools to HDFS and then use something like Splunk's product for Hadoop (Hunk) to search and visualize that data.

 

With a sufficiently powerful box, can a single instance of radium collect and aggregate that volume of traffic?  Because a lot of Oracle's growth in the cloud has been through acquisitions, we have a number of overlapping IP subnets in different locations.  If that could potentially interfere with radium's ability to dedup flows from multiple argi, is it possible to configure nested radium clients?

 

I was just hoping to get your thoughts on what gotchas might exist trying to implement argus at that kind of scale.

 

Thanks!

 

Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140729/fa6b8d80/attachment.html>