hardware for argus with 10GB link

[Peter Van Epp]

On Fri, Apr 23, 2010 at 12:14:02PM -0700, Michael Sanderson wrote:
> We've recently had our campus network connection upgraded to 10Gb and  
> are now looking at our options for tapping that connection and getting  
> argus collecting data again.  We'll be tapping our fibre links with most  
> likely with NetOptics taps, but we're looking for suggestions of  
> appropriate 10Gb NICs for our argus "sensor" box.  We definitely won't  
> be pushing the 10Gb link initially, but I expect that we'll see periods  
> where we approach 2Gb sustained.
>
> I recall discussion on the list regarding interrupt coalescing on the  
> network cards generating potential issues due to the way that time  
> stamps are assigned to packets.  I know that Endance's DAG cards solve  
> this particular issue by timestamping every packet, but they are  
> expensive and eat a big portion of my hardware budget.

	Unfortunatly you only get what you pay for :-). The biggest win on DAGs
is a large (4 meg on gig cards) internal SRAM buffer. That (against 64 kbytes
on a typical Gig NIC) gives you a small amount of breathing room for bus and
interrupt latency in to the CPU before packet loss occurs. My comments on 
a bunch of this (including interrupt coalescing) are available at 

http://www.qosient.com/argus/sensorPerformance.shtml

	As Mr Plonka noted the www.ntop.org site is a good place to go to learn
more about the issues of high speed capture as well (you really want a zero
copy interface such as pf-ring for instance, the memory copy is expensive). 

>
> Does anyone recall the specific issues with interrupt coalescing and  
> argus?  There will definitely be potential for out of order TCP  
> connection startup/shutdown.  What is the impact on the flow reporting?

	Mostly timing inaccuracies as multiple packets appear at the same
apparant time (when they didn't in reality). Not much else happens as Carter
put in some changes for me when I was running 2 NICs on an FDX link that 
(as long as the individual time order is correct which it should be) allowed
for things like a syn-ack to arrive before the syn was sent due to interrupt
scheduling. As far as I know thats still there and works.

>
> Are there other options to Endance's cards that solve the timestamp  
> problem, either other vendors or system/kernel configuration?
>

	I haven't heard of any, but since I retired I haven't been paying much
attention either :-). There is an Intel 10 Gig NIC that has internal time 
stamping for the HPC guys (I came across it looking at ntp implementations 
since thats one of the things you need for a DAG clone) that may be a good bet,
but it is brand new and perhaps not very available yet and drivers will likely 
be an issue. The problem is emulating a DAG is low volume and non standard (I'm 
poking at doing so more cheaply at gig with FPGAs) in that you need to tap in 
to the Ethernet MAC to get the start of frame signal, you need an ntp type 
time source on card and you want large amounts of SRAM buffer which are 
expensive. I'll note one other thing to consider on a DAG is that it has an 
internal CPU and thus could do the network to host order header translation
on card (perhaps ...) which sould save you a very valuable %20 or so memory
bandwidth on the sensor side (but you would also need to lie to argus about 
machine endian). This assumes that your sensor is little endian (i.e. Intel),
if you have a big endian machine you are better off (except probably in price
:-). 
	BCnet's network research group has a 10 Gig capture applience (an 
Endace Ninja) fed from a Netoptics Network Director (more to aggregate multiple
1 Gig segments on to the 10 gig for capture than the other way but ether will
work, but it was the same price as a 10 gig DAG) although I don't know how its
working as I retired just as it was being installed (I did a spec and run :-)),
so it would probably be a good bet to have a chat with the NMC folks your way
about what its doing.
	Its also worth noting that argus runs perfectly well half duplex, so 
(budget allowing) two sensors are a good bet one for each half of the FDX 
connection. Both sensors feed out via ethernet to a collection box where argus
merges the streams and writes to disk (you don't want to be doing disk I/O on
your sensor box). This halves the required performance on each sensor (and
DAGs at 10gig are HDX only or used to be anyway). You also want to take care
to get the fastest RAM you can afford (preferably the fastest DDR3) in a 
high performance CPU (your HPC folks are a good bet for advice here) for the
sensors as memory bandwidth is the most usual limiting factor at high link 
speeds.

Peter Van Epp