What to do in this situation

[Carter Bullard]

Gentle people,
   The error that has slowed us down has a component 
that needs to be engineered for, so I'd like your
opinion.

   A part of the problem is that on machines that are
exhibiting the problem, flow records aren't being
processed fast enough, and the output queue to the process
that is writing records to the disk is getting too big.
At some point, Argus has to make the decision that the
load is too much for the file writing process, and gracefully
shut it down.  Currently the watermark is 8096 records in the
outbound queue.

   The problem situation occurs because of how we deal with
this condition, which has been to just shutdown the queue.
This causes the disk process to finish, but that's it.  Because
it no longer has an input queue, it just won't get any more
records.  Thus the "argus just stops writing output records,
but everything else seems OK".

   Why were getting to this point, I'm not quite sure, but one
of the machines is having difficulty probably is either
underpowered or the scheduling is terrible, as its reporting
that its dropping ~5% of the packets on the floor.

   In the fix, I've added kill the process and wait,
to deal with the resulting zombie, and then continue on,
writing some strong language to syslog().

   Now the question.  Should argus() exit if the problem
output process is the one writing to the file?  If the output
process is a network based remote access, I'd say keep going,
but if its the file, we should exit so that we can start again
as soon as possible.  Does this sound reasonable?

Carter


Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 18K
New York, New York  10022

carter at qosient.com
Phone +1 212 588-9133
Fax   +1 212 588-9134
http://qosient.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20010314/a27b8b1a/attachment.html>