Queue Exceeded Maximum Limit, Terminating process (was "Re: DoS woes....")

[Dave Plonka]

Hello argus users,

One of our argus instances is periodically terminating one of its
processes with these messages " Queue Exceeded Maximum Limit",
"ArgusWriteOutSocket failed Input/output error", then "Terminating process".

It's argus-2.0.5 on this platform:

   # cat /etc/redhat-release
   Red Hat Linux release 8.0 (Psyche)
   # uname -r               
   2.4.18-14smp

This argus is watching all IP traffic for a relatively quiescent
class-A network (5,000-10,000 packets-per-second).

When the problem occurs, two other argus processes continue to run, but
no more content is written to the argus output file.  Once we discover
the problem, we typically kill the argus parent at this point, and
restart it, and it runs for another week or two.

Here is a typical series of argus syslog messages leading up to the
termination:

   Jun  9 03:52:26 localhost argus[19729]: ArgusWriteOutSocket(0x8148ae0) Queue Count 249238 
   Jun  9 03:52:56 localhost argus[19729]: ArgusWriteOutSocket(0x8148ae0) Queue Count 254228 
   Jun  9 03:53:26 localhost argus[19729]: ArgusWriteOutSocket(0x8148ae0) Queue Count 259423 
   Jun  9 03:53:51 localhost argus[19729]: ArgusWriteOutSocket(0x8148ae0) Queue Exceeded Maximum Limit 
   Jun  9 03:53:51 localhost argus[19729]: ArgusHandleData: ArgusWriteOutSocket failed Input/output error 
   Jun  9 03:53:51 localhost argus[19729]: ArgusHandleData: Terminating process 19730

I see that this is quite similar to what Russell reported below.  Any
hints on what's going on?  What does ""Queue Exceeded Maximum Limit" mean?
Any code changes/workarounds?

Thanks,
Dave

On Wed, Jan 30, 2002 at 05:26:39PM +1300, Russell Fulton wrote:
> Hi All,
> 	We are currently suffering a wave of Syn flood attacks, they last about
> 15-20 minutes and argus dies a few minutes after the attack starts:
> This one started at 16:19:30
> 
> Jan 30 16:20:23 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Count 50001 
> Jan 30 16:20:53 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Count 96277 
> Jan 30 16:21:06 hihi argus_linux[8525]: client(/home/argus/data/current)
> done. 
> Jan 30 16:21:23 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Count 141188 
> Jan 30 16:21:53 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Count 211530 
> Jan 30 16:22:23 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Count 255422 
> Jan 30 16:22:26 hihi argus_linux[22625]: ArgusWriteOutSocket(0x81ec988)
> Queue Exceeded Maximum Limit 
> Jan 30 16:22:26 hihi argus_linux[22625]: ArgusHandleClientData:
> ArgusWriteOutSocket failed Resource temporarily unavailable 
> Jan 30 16:23:09 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 50001 
> Jan 30 16:23:39 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 90661 
> Jan 30 16:24:10 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 153387 
> Jan 30 16:24:40 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 152026 
> Jan 30 16:25:10 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 150928 
> Jan 30 16:25:40 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 150064 
> Jan 30 16:26:10 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 148050 
> Jan 30 16:26:40 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 145983 
> Jan 30 16:27:10 hihi argus_linux[24730]: ArgusWriteOutSocket(0x83d50f0)
> Queue Count 145536 
> Jan 30 16:27:30 hihi argus_linux[8523]: ArgusProcessPacket ()
> ArgusWriteOutSocket Failed to Multiplexor. Shuting Down 
> 
> I am guessing that the problem is that process that is feeding my
> watcher script which connects to the server by a socket.  Unfortunately
> the whole argus server dies.  
> 
> Is there any way to make argus more robust in this situation?  If it is
> the network socket that is the problem then killing this off while
> keeping the disk logging  would be great.
> 
> -- 
> Russell Fulton, Computer and Network Security Officer
> The University of Auckland,  New Zealand

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka  ARS:N9HZF  Madison, WI