argusarchive (rasort) segfaulting

[Peter Van Epp]

On Wed, Aug 10, 2005 at 06:45:29PM -0600, Richard Johnson wrote:
> At 12:05 -0700 on 2005-08-10, Peter Van Epp wrote:
> > 	Check for a per task memory rlimit. On FreeBSD you need to reconfig the
> > kernel to allow a process to use more than (I think) 512K of memory which is
> > usually what bites rasort. Of course you can also usefully fix this problem
> > (and save some time) by commenting out rasort in argusarchive and only sort
> > if you really need to. I know we had to boost a kernel memory limit to give
> > the ring buffer code a couple hundred meg buffer on Linux, but my post
> > processing is still on FreeBSD (where the limit above has bitten me before).
> 
> 
> Same on OpenBSD.  Adding the following types of commands to argusarchive
> might help (modulo the size of your RAM, of course):
> 	ulimit -d 1048576
> 	ulimit -m 3316384
> 
> Of course, 4GB total on the machine isn't enough for us when we're trying
> to sort a 30 minute slot in which we've been hit by a serious port scan.
> (Traffic levels are peak 350Mbps and about 90kpps without the scans.)  Thus
> I now muck about with building amd64 RAID kernels for OpenBSD 3.8-current,
> which will run on a box with more than 4GB of RAM.
> 
> 
> Richard

	Sounds like time to change tactics to me :-). I hit a similar problem
over a longer interval (24 hours in my case) and in perl (because as noted
I don't bother sorting the records). I've come to the conclusion the answer is
to take several sweeps through the data. On the first one identify port 
scanning source hosts (which I already do) and mark them (ignoring further 
records from that host once it is identified as scanning). On the second pass
dump the records relating to a scanning host in to a disk file (to avoid the
memory blowup from combitorial explosion) and process the better behaved hosts
as usual. On a third pass look over the scanning hosts mostly to see what of
my machines responded which is usually a fairly small number or which of my
machines is compromised, skyped or otherwise p2ped and needs whacking :-). 
Thats where the real combitorial explosion takes place, scans of our address
space are usually tolorable. I currently find this by seeing that the overnight
processing has blown up, run a quick real time scan to catch and whack the 
cuprit and then filter the host out of the overnight data. One day (maybe even
soon) that is going to motivate me to fix this :-).

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada