200GB a day

[Eric Gustafson]

Hey all,
Just weighing in, and a quick question:
Not only can Argus itself handle 200GB a day, or 1TB a day, like was
mentioned, but we are pushing roughly ten times that.  This thing can
scale, given the right hardware! (Bivio 7000 series) No dropped
packets, no signs of memory issues, running for months and months
straight.

This leads to the question of how one manages the massive amount of
data such a setup generates.  How do those of you with larger argus
installs manage your data? Right now, our in-house Perl wizard has
prepared some scripts to attempt to wrangle (search / compute stats
on) the trees of datestamped bzips that make up our data, but this
seems far from ideal, but given the size of the data being processed,
and the number of records, I don't know of a better one. I briefly
thought about SQL, but even taking a smaller file of ours and running
it through made a test SQL instance cry and beg for mercy, obviously
due to the number of records involved.

Is a linear search with ra the best I can do?

(Thankfully, we don't need to do searches and stuff too often!)

Cheers,
- Eric

On Fri, Aug 5, 2011 at 9:36 AM, Carter Bullard <carter at qosient.com> wrote:
> Hey Jonathan,
> Seriously, I mean that if your monitoring a 100 Mbps link today, and its currently presenting 12 Mbps,
> you should not lose any sleep using port mirroring to monitor that 100 Mbps link.
>
> Port mirroring breaks down differently with different vendors products, but the rule of thumb I use is,
> if the total mirrored traffic sent to an output port is over 80% of that output ports speed, then you are
> probably losing packets some where, depending on how much money you spent on the switch.
>
> For me, its not the speed of the link you are monitoring, its the speed of the output port you are sending your
> copied packets to.  If you are monitoring a full-duplex 100 Mbps link, and writing the output to a 1Gbps output
> port, you will be fine.  If however, you are sending the potential 200 Mbps load to a 100 Mbps output port,
> then you can have collision / contention problems if the total instantaneous load exceeds 70-80 Mbps, and
> you'll have monitored packet loss.  If you're just seeing 12 Mbps, you probably (the keyword is probable)
> won't have any problems.
>
> That is just a rule of thumb that has worked for me, but its not a fact or truth or the law.
>
> Port mirroring is done in hardware, now a days, and so its not additional "work" for the switch.
> It will generate more heat, but it shouldn't negatively impact the functionality of the device.
>
> But Peter and the others are absolutely correct saying that there are limits based on the nature of the
> network device, link, traffic etc….
>
> Carter
>
> On Aug 5, 2011, at 7:38 AM, Jonathan Tripathy wrote:
>
>>
>> On 05/08/2011 00:18, Carter Bullard wrote:
>>> Hey Jonathan,
>>> Port mirroring works pretty good up to about 80% of port speed.  Really a vendor and statistical thing. You should be fine monitoring 100 Mbps links using port mirroring with modern switches routers.
>>>
>>> Carter
>> Hi Carter,
>>
>> Did you mean 100Mbps each way, or 100Mbps in total?
>>
>> Thanks
>>
>
>