argus fails to allocate memory...

[Michael Sanderson]

Hi Russell, I have seen this behaviour with 4.4 (and 4.3)  (Jan 26, 09 
posting to the list).  Argus appears to be getting packets with time 
stamps that are significantly out to lunch.  The daemon then selectively 
pushes out flow data and accumulates lots of data in memory.  If you 
kill -HUP it, it will happily flush all the flows out and exit.  In my 
experience, looking at that data will show one flow with a time stamp 
that makes no sense.  I've implemented some things to watch the daemon 
and kill -HUP/restart when it gets "too big".  I haven't found the time 
to tcpdump the system and deal with rotating those logs so that I don't 
run out of space before I catch the packet and find out if this is an 
OBSD problem or some weird argus problem.

I believe that a similar problem had been seen by Martijn van Oosterhout 
back in Dec 08/Jan 09, but on Linux.  The root cause there appears to 
have been a Linux kernel bug.

     Michael Sanderson

Russell Fulton wrote:
> Hi Folks,
> 
> I'm afraid I've been totally snowed under for that last 18 months and 
> not really following what been happening in the argus world. I *have* 
> registered that things have happened that I need to catch up on.  Carter 
> has clearly not been idle :)
> 
> This post is prompted by problems I am having on a new OBSD 4.4 sensor.  
> The box has 1GB of memory, it is running snort and pf (part of the 
> kernel) and without argus running shows free memory at around 600MB.  
> After starting argus and letting it run for a while free memory drops to 
> 590MB and top shows argus using 10MB.  I will keep an eye on this now I 
> have some idea about what is happening and post again if behaviour changes.
> 
> I start argus and it runs fine for a few hours and then stops.  I could 
> not find any errors in the logs so I removed daemon from the config and 
> started it sending both stdout and stderr to a file.  Here is the tail 
> of the file after it stopped:
> 
> -bash-3.2$ head argus.out
>   ArgusWarning: argus[15833]: 11 Apr 09 20:34:51.271087 started
>   ArgusWarning: argus[15833]: 11 Apr 09 20:34:51.271471 
> ArgusGetInterfaceStatus: interface em1 is up
>   ArgusWarning: argus[15833]: 11 Apr 09 23:58:39.068381 ArgusInterface 
> timestamps wayyy out of order: now -293216183 then 1239451118
>   ArgusWarning: argus[15833]: 12 Apr 09 00:48:28.318861 ArgusInterface 
> timestamps wayyy out of order: now -1668161463 then 1239454108
>   ArgusWarning: argus[15833]: 12 Apr 09 00:48:28.602530 ArgusInterface 
> timestamps wayyy out of order: now -1668161463 then 1239454108
>   ArgusWarning: argus[15833]: 12 Apr 09 00:48:28.602602 ArgusInterface 
> timestamps wayyy out of order: now -1668161463 then 1239454108
>   ArgusWarning: argus[15833]: 12 Apr 09 07:33:24.759899 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
> 
> -bash-3.2$ tail argus.out
> ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850584 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850661 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850753 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850820 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850886 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.850951 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.851016 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.851099 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>   ArgusWarning: argus[15833]: 12 Apr 09 09:58:19.851168 ArgusNewFlow() 
> ArgusMalloc error Cannot allocate memory.
>     ArgusError: argus[15833]: 12 Apr 09 09:58:19.851195 
> ArgusNewFragFlow() returned NULL.
> -bash-3.2$ grep ArgusNewFlow argus.out | wc -l
>  12,058,190
> 
> server version 3.0.0
> 
> The sensor is on our resnet firewall so there is all sorts of crap on 
> the network currently we are in the middle of a break so the load is not 
> that high and argus will stay running for up to 12 hours.  When the 
> halls are filled it runs for two to three hours.
> 
> Anyone have any idea what is going on?
> 
> Russell