200GB a day

[Peter Van Epp]

On Wed, Aug 03, 2011 at 03:46:00PM +0100, Jonathan Tripathy wrote:
> Hi Everyone,
> 
> Do you think argus is able to handle a setup using standard hardware which pushes 200GB a day? I intend to run argus on the actual servers, and have a dedicated server for reading the stream and writing it out to a database. No switch port mirroring involved.
> 
> Do this sound reasonable?
> 
> Thanks

	Personally I prefer to run the argus sensor on its own box behind a
network tap so argus can not affect the production network, however as long 
as there is CPU and memory capacity / bandwidth available running argus on the
server should work. Note that argus will copy a second copy of all incoming 
packets which creates a load on the machine. It would be wise to enable MAN
records in the argus data (I believe they are off by default) and have a look
at them during your busiest time. One of the fields is pcap reported packet 
loss and if that isn't close to 0 (or at least quite low compared to total 
packets received) then argus is losing packets and probably overall machine
performance is suffering too and it may be time to consider a separate argus
sensor.
	Some old (version 2.0.6, the 3 series stores more and  thus won't 
be as good): 1.3 terabytes across 24 hours (200 megabit link saturated most
of the day) generated about 1.3 gigabytes of argus data (as noted 2.0.6, the
3 series number would be higher due to more data stored). This was running 
Intel server pro NICs, DAGs aren't needed for that slow a link. Note that 
this assumes the argus data is not being stored to disk on the sensor machine.
On old (~10 years now :-)) hardware storing to disk on the sensor machine 
without DAGs starts to lose packets due to bus contention at 30 to 50 megabits
per second. You are always best to test your setup (using for instance 
tcpreplay) with a know workload to assess whether your packet loss is 
acceptable or you need to improve performance.

Peter Van Epp