Argus on Bivio 7500

[Peter Van Epp]

On Fri, Aug 21, 2009 at 10:08:57AM -0700, Eric Gustafson wrote:
> Hey everyone,
> We've been doing some further experiments with Argus on our Bivios in our
> environment, and have noticed some partial flows, and magically disappearing
> packets when compared to our currently-in-production Dell-based sensor.
> We figured out, however, that the problem is not with Argus or Snort (the
> other app running on our Bivios) specifically, but only happens when we
> combine the two.

	Noting that I know very little about Bivios :-) the first thing I'd 
suspect is memory and/or bus bandwidth. Assuming that argus and snort are 
looking at the same interface, off the top you have an extra copy (and the 
full packet in Snort's case I believe) to create the two libpcap streams 
(unless Bivio is managing to map the same memory to the two streams which 
is possible). RAM bandwidth claims about 16 gigabytes per second on the
latest fastest DDR3, DDR2 is around 6.8 gigabits per second (calculated from 
a DIMM data sheet using the cycle time and bus width) which is also a common 
limiting value in 10 gig implementations from the honest :-)). A copy of a 
full duplex gigabit of traffic eats 2 gigabits of that just for the pcap copy. 
In addition your CPU is filling cache out of that same memory pool to run your 
program (which hopefully is mostly in CPU internal cache to slow that down) so
this is a fairly likely bottleneck. As well most of the interface busses peak 
out at a couple of gigabits per second per second as well so you could be 
seeing a bottleneck as the traffic has to all go through a speed limited bus 
(given that Bivio is a multiprocessor system they may have addressed this 
though too). 
	One interesting test if you can arrange it would be to run several 
(4 minumum assuming 1 gig streams, if that doesn't die then more :-)) of 
netperf or iperf from a test box (or multiple test boxes with on netperf each 
feeding a switch) to your Bivio. Netperf is a much lower load that argus or 
snort but will pump and record the speed of traffic through an interface. 
Assuming you are using 1 gig netperf boxes you need two copies to simulate an 
fdx 1 gig load (and thus the count of 4 netperf instances). This should tell 
you how much raw bandwidth you can pump through the system. The speed that 
argus or snort can process will be much less than this because they do a lot 
more than just read the packet but this is the first potential bottleneck (and 
one is all you need :-)). Its also possible that you need to boost the tcp 
buffers in your kernel if thats not already done. A number of years ago I was
involved in a project on a 1 gig lightpath. Luckily we had production HPC
tuned machines on a parallel link that could achieve 995 megabits per second
on netperf because the initial cut (with a stock Linux kernel on the far end)
could only get 35 megabits per second on netperf until the kernel was modified
to match the machine on our end (there are lots of places to have problems :-)).
	If you have a box with a 10 gig interface running a single netperf 
stream in to the Bivio will do the trick, but I'm guessing it may be easier to 
come up with multiple 1 gig boxes and an aggregating switch. You don't want to 
do this on a production network though as netperf/iperf will happily take all 
the bandwidth they can get (thats what they are for) and freeze everyone else
out (which generally makes everyone else unhappy with you :-)). 

Peter Van Epp