Argus server exits with "maximum errors exceeded 200000"

[Peter Van Epp]

On Mon, Nov 23, 2009 at 10:39:36AM -0500, Guy Dickinson wrote:
> [my apologies, this should have gone to the list]
> 
> Peter Van Epp wrote:
> 
> > > 	It may be profitable to change the 6 to a 1 at line 1443 in
> > > argus/ArgusUtil.c i.e.:
> > >
> > > ArgusDebug (6,"ArgusWriteSocket: write returned %d, errno
> %d\n",retn, errno);
> > >
> > > to
> > >
> > > ArgusDebug (1,"ArgusWriteSocket: write returned %d, errno
> %d\n",retn, errno);
> > >
> 
> I've now done so, and after another crash, the debug output looks like this:
> 
> argus[1265]: 22 Nov 09 01:22:21.797208 main() ArgusGetPackets returned:
> shuting down
> 
> argus[1265]: 22 Nov 09 01:22:21.797787 ArgusShutDown(Normal Shutdown)
> 
> argus[1265]: 22 Nov 09 01:22:21.797855 ArgusCloseSource(0xb7e69008) starting
> argus[1265]: 22 Nov 09 01:22:21.797917 ArgusCloseSource(0xb7e69008)
> deleting source
> argus[1265]: 22 Nov 09 01:22:21.798007 ArgusCloseModeler(0x9470008)
> pushing close record 0x3243f27c
> argus[1265]: 22 Nov 09 01:22:21.798064 ArgusCloseModeler(0x9470008)
> argus[1265]: 22 Nov 09 01:22:21.798125 ArgusCloseOutput() scheduling
> closure after writing records
> argus[1265]: 22 Nov 09 01:22:21.798226 ArgusOutputProcess() received
> stop record 0 records on the list
> argus[1265]: 22 Nov 09 01:22:22.199980 ArgusWriteOutSocket(0xad15e008)
> maximum errors exceeded 200000
> argus[1265]: 22 Nov 09 01:22:23.050160 ArgusCloseOutput(0x94704e8) done
> argus[1265]: 22 Nov 09 01:22:23.050243 ArgusShutDown()
> 
> Any further insight would be much appreciated.
> 
> Thanks!
> Guy
> 
> 
> -- ------------------ Guy Dickinson, Network Security Analyst NYU ITS
> Technology Security Services guy.dickinson at nyu.edu (212) 998-3052

	Looks like the socket errors may be a red herring. They seem to happen
during shutdown (which could mean the socket has been closed when something
thinks it should write, which may be a bug but probably not whats biting us 
right now :-)), but something previous has caused the input stream to return 
and cause an argus shutdown:


argus[1265]: 22 Nov 09 01:22:21.797208 main() ArgusGetPackets returned:
shuting down
 
we need to figure out why that happened (usually a libpcap error of some kind).
Does /var/log/messages have anything interesting in it around this time? If
I'm remembering the correct thread you are on a Ninja with RHEL so thats where
syslog should be going by default. There hopefully is a syslog message of the
form "ArgusGetInterfaceStatus: something bad happened" in syslog to point us
to what is unhappy. 
	There was a similar problem a few months ago on a Bivio and at that 
time I found a case where argus would silently close the pcap interface (and 
then shut down due to no interface :-)) when it got an error and suggested we 
needed to add a syslog message in that case for the next time this happened 
:-).  A quick look at the code (argus/ArgusSource.c in ArgusGetInterfaceStatus)
indicates the code has changed substantially and so may need a closer look to 
see if a similar silent path exists if there isn't anything in syslog.

Peter Van Epp