[ARGUS] Best Hardware

[Peter Van Epp]

	You are right that is a test I should do. However I won't be suprised
if the results are counter intuitive :-). The slow part of the operation is
likely the fetch from disk, given that, compressed may be better (less data 
comes off disk and gets expanded by even cheaper CPU time in fast memory in
to something larger which may turn out to be faster over all). I expect the 
real win would be moving to a 3ware raid controller with multiple striped 
disks to get much higher disk bandwith and a faster CPU with more memory. 
	I've been playing around with post processing changing memory usage by 
spooling to disk, and instrumenting to see what is being costly and a lot of 
it is processing the information long after it has come out of ra to pick out
the interesting (to me anyway) patterns. At the moment (at least during times
of non extreme port scans) my setup is running acceptably as is and the money
that could have gone to new hardware is instead buying some 4 port regen taps
from netoptics for even better playing around on my production links, one port
for argus, three more for sniffers and other such toys, maybe snort :-). I 
think although I'm not there yet, a split of port scanning from traffic 
accumulation should fix that problem for now (which is primarily one of memory).
A port scan (or scans) Monday took hourly post processing from 5 minutes to 
around 7 hours as the machine swapped with a non serious ripple effect which 
pointed out some poorly thought out engineering that needs fixing.
	If I get some time I'll see what difference running an uncompressed 
file makes. And once the transient from buying the netoptics dies away perhaps 
I'll go for a 3ware and a bunch of SATA disks as the next step. 3ware is smart 
enough to be advertising FreeBSD support for their cards which immediately got 
my vote, although I was shopping by driver support from the FreeBSD kernel 
lint file anyway :-).

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Thu, Oct 14, 2004 at 11:32:22AM +1000, Andrew Pollock wrote:
> On Mon, Oct 11, 2004 at 07:37:15PM -0700, Peter Van Epp wrote:
> 
> [snip]
> 
> > 
> > [test4:~] vanepp% time /usr/local/bin/ra  -r com_argus.2004.10.10.22.00.00.0.gz -nn >/dev/null
> > 156.000u 150.690s 5:46.06 88.6% 0+0k 0+2io 0pf+0w
> > 
> > 	Which is a fair bit slower at post processing that the PCs (a new G5
> > may do better). This is similar to previous times I've done this, a 600 Meg
> > PC with less memory seems to do better on post processing (capture may be a
> > different matter though).
> 
> Disk is cheap. I've always been of the opinion that it was faster for
> processing to keep the logs around uncompressed for processing, to avoid the
> overhead of having to run them through gzip. Come archiving time, compress
> 'em sure...
> 
> regards
> 
> Andrew