[ARGUS] A concrete packet loss example :-)

[Peter Van Epp]

	While I intended this more as an example of the "here's where you can
lose packets at high speed" it has also produced some useful pointers off list
 :-). This paper from SANE was pointed out (and is available in PDF on the 
author's home page (which has some more interesting looking stuff too):

http://www.nluug.nl/events/sane2004/abstracts/ab.html?id=100

	and someone else contributed this experience (although our HPC folks
have been having troubles with their SK drivers on Linux, thats part of why
my box is there because one card was reported broken when they were borrowing
it, but it looks to be working fine under FreeBSD) so perhaps there is Linux
in my future after all :-).

"BTW, the box running argus/snort is now Linux 2.6 kernel with Phil Wood's      
ring buffered libpcap and it has improved performance greatly - about half      
the kernel cpu load as same config without the ring buffer (50MB of kernel      
memory!) and no apparent packet loss."

	Our HPC folks are happy to have anything at all looking at their 
network and thus aren't pressuring and can easily fill the pipe with jumbo
frames (all the packets when the failure occurred were indeed 9K jumbo frames
and I do suspect the SK driver as the cuplprit :-)) so I can work on this as
there is time, for fun (for some value of fun :-)). Production isn't (at least
yet) seeing this problem so it isn't so far an issue.

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Mon, Oct 18, 2004 at 09:15:50AM -0400, Carter Bullard wrote:
> Hey Peter,
> If you are monitoring jumbo frames, I'd focus on the SK card and
> how the snaplen is being implemented.  If you can change the snaplen
> to some low level (48-64 bytes) in argus and not see a better measure,
> then there may not be anything you can do  as you could be hitting
> PCI bus limitations moving jumbo frames to memory.  What do you think
> is the biggest packet you're seeing?
> 
> The Endace cards really do a good job with bigger packets, as the
> snaplen is enforced in the card itself, at least with the OC-48
> and OC-192 cards that I have used.
> 
> Carter
> 
> 
> > From: Peter Van Epp <vanepp at sfu.ca>
> > Date: Sat, 16 Oct 2004 15:58:45 -0700
> > To: <argus-info at lists.andrew.cmu.edu>
> > Subject: Re: [ARGUS] A concrete packet loss example :-)
> > 
> > Well a look at the kernel source is illuminating :-). I suspect the
> > problem is in the SK driver. It implements local buffers for jumbo frames
> > (which I'm using) and issues ENOBUFS in a few places (a few too many places
> > for my comfort in fact) and I expect those ENOBUFS are one cause of an rx
> > error
> > incrementing somewhere in there. If I can figure out how to make it log a hit
> > on those somewhere I may have found my driver packet loss (which is the first
> > step, other than the "10 x the number of buffers and see what breaks" which I
> > also may try). As well it allocates 512 tx buffers and 256 rx buffers in card
> > memory, since I don't do no stinkin tx, I see 768 (or maybe 760 and 8) rx
> > buffers in my future :-) It may also be profitable (possible may be something
> > else :-)) to chop the jumbo frame down to the 128 or 196 bytes argus wants on
> > its way to the bpf interface (or ideally on the card before DMA) if that isn't
> > already done. This would leave a very strange but probably very fast driver if
> > you only want to sniff and may be useful as an option beside the current real
> > sk driver available by kernel config for sniffing machines.
> > I've got the feeling this whole operation may take a while because its
> > getting complex (and in to an area where I haven't played before) but we will
> > see how it goes, and in any case it should be fun :-).
> > 
> > Peter Van Epp / Operations and Technical Support
> > Simon Fraser University, Burnaby, B.C. Canada
> > 
>