Large scale Argus implementation

[Carter Bullard]

Hey Craig,
This is the kind of thing we do professionally, so if you would like to work with QoSient on your project, contact me directly.

Hope all is most excellent,

Carter

Carter Bullard, QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

> On Jul 29, 2014, at 5:41 PM, Craig Merchant <craig.merchant at oracle.com> wrote:
> 
> Hey, Carter…
>  
> Now that Responsys has become a part of the Oracle Cloud, I have been tasked with implementing a network flow solution for all of the cloud infrastructure.  Initially, we would only monitor ingress/egress traffic, but eventually we want to monitor internal traffic as well.
>  
> We currently have about 25 data centers and approximately 100,000 hosts.  The Responsys data centers are small in comparison to some of the other cloud data centers and our internal traffic volume is around 10-15 Gbps.  So, I would imagine that the rest of data centers would generate something like 10x to 15x that traffic.
>  
> After reading a bit about scalability and the rasql tools, I’m pretty sure that the volume of traffic would overwhelm a relational database.  We can get Oracle Big Data Appliances fairly cheap (all things considered).  I’m thinking that the best approach may be to write the ASCII output of ra tools to HDFS and then use something like Splunk’s product for Hadoop (Hunk) to search and visualize that data.
>  
> With a sufficiently powerful box, can a single instance of radium collect and aggregate that volume of traffic?  Because a lot of Oracle’s growth in the cloud has been through acquisitions, we have a number of overlapping IP subnets in different locations.  If that could potentially interfere with radium’s ability to dedup flows from multiple argi, is it possible to configure nested radium clients?
>  
> I was just hoping to get your thoughts on what gotchas might exist trying to implement argus at that kind of scale.
>  
> Thanks!
>  
> Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140730/f346931e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2443 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140730/f346931e/attachment.bin>