Argus crash after successful compile against PF_RING

Wed Dec 7 09:53:21 EST 2011

Hey Jesse,
This sounds like a PF_RING timestamp problem.  Argus believes the
timestamps it gets from the packet engine so much, that it sets its
concept of time based on those timestamps.  This allows reading packets
from files to work the same as from live interfaces. If the packet capture
layer screws up the time, argus will be seriously affected.

If we get packets in the future, argus can apparently "stop" its record output.
We end up with a flow in the status output queue with a queue time in the
future, and when the record finds its way to the end of the queue, argus
will have to wait for (timestamp + flow_status_time) before taking it off the
queue.

OK,   I put in packet rejection based on a time range (+/- 30 seconds) when
reading packets from a live interface, and I've added dumping the error
packets to a dump file.  Turn this on using both of the argus.conf file variables:

#ARGUS_PACKET_CAPTURE_FILE="/path/to/filename.pcap"
#ARGUS_PACKET_CAPTURE_ON_ERROR="yes"

This should help confirm what the nature of the problem is, and help
point the finger at whatever module is in error.

Do turn on both variables, if you don't turn on the ON_ERROR variable, all
your packets will go in the file.

I've uploaded argus-3.0.5.7.tar.gz to the developers server.  Grab it at:
   http://qosient.com/argus/dev/argus-3.0.5.7.tar.gz

Carter

On Dec 7, 2011, at 6:39 AM, Jesse Bowling wrote:

> On 12/6/11 11:38 PM, Gamarro, Estuardo wrote:
>> 
>>     Running 3.0.5.6/3.0.5.25 here and observing issues with timestamps
>> out of order as well.  Argus does not crash as easily, but rasplit
>> generates a few flows with very odd dates.  
> 
> I saw a bit of this as well, with an odd save file dated in 1980...
> 
> I am testing with
>> PF_RING_aware and regular Linux drivers, but it doesn't seem to make a
>> difference.  
> 
> I should mention that I too am using the PF_RING aware drivers on this
> machine...
> 
>> In some cases Argus has stopped collecting flows without
>> crashing.  Argus tends to generate more "timestamp wayyy out of order"
>> logs on links with higher utilization (> 600Mbps). 
> 
> Also similar here; the two argi are montitoring links that are both
> running over 700 Mbps...I also experienced the case where argus did not
> crash but stopped collecting flows...
> 
> Cheers,
> 
> Jesse
> 
>> 
>> CentOS 6  64-bit/ 2.6.32
>> PF_RING 5.2.1
>> 
>> 
>> E.J. Gamarro
>> 
>> 
>> -----Original Message-----
>> From: argus-info-bounces+egamarro=depaul.edu at lists.andrew.cmu.edu on
>> behalf of Chris Wakelin
>> Sent: Tue 12/6/2011 6:29 PM
>> To: Carter Bullard
>> Cc: argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] Argus crash after successful compile against PF_RING
>> 
>> 64-bit Ubuntu 10.04 (but with kernel 2.6.38) and 8 cores (also running
>> Suricata IDS on 6 of them). I'm now trying 3.0.5.6/3.0.5.25 as of 5
>> minutes ago :)
>> 
>> Best Wishes,
>> Chris
>> 
>> On 07/12/2011 00:26, Carter Bullard wrote:
>>> Hey Chris,
>>> Is this a 32-bit or 64-bit machine?
>>> Carter
>>> 
>>> On Dec 6, 2011, at 6:18 PM, Chris Wakelin wrote:
>>> 
>>>> On 06/12/2011 22:20, Carter Bullard wrote:
>>>>> Hey Chris,
>>>>> Sorry to hear that you're having problems !!!!!
>>>>> Lets try to fix this thing before the end of the year, if you have
>> some time,
>>>>> as I'd like 3.0.6 to be solid.
>>>>> 
>>>>> What version are you running, and do you get any log output?
>>>> 
>>>> Argus 3.0.5.5, Argus-clients 3.0.5.20 and PF_RING 5.1.0 at the moment.
>> Log output is pretty much as Jesse said:
>>>> 
>>>>> Dec  6 15:49:25 vinms2 argus[20162]: 06 Dec 11 15:49:25.539044 started
>>>>> Dec  6 15:49:25 vinms2 argus[20162]: 06 Dec 11 15:49:25.541036 started
>>>>> Dec  6 15:49:25 vinms2 argus[20162]: 06 Dec 11 15:49:25.622050
>> ArgusGetInterfaceStatus: interface eth4 is up
>>>>> Dec  6 15:58:06 vinms2 argus[20162]: 06 Dec 11 15:58:06.520232
>> ArgusInterface timestamps wayyy out of order: now 1323187086 then 999165474
>>>>> Dec  6 15:58:11 vinms2 argus[20162]: 06 Dec 11 15:58:11.520141
>> ArgusGenerateRecord: packet size type not defined
>>>>> Dec  6 15:58:56 vinms2 argus[20290]: 06 Dec 11 15:58:56.742608 started
>>>>> Dec  6 15:58:56 vinms2 argus[20290]: 06 Dec 11 15:58:56.744638 started
>>>>> Dec  6 15:58:56 vinms2 argus[20290]: 06 Dec 11 15:58:56.931989
>> ArgusGetInterfaceStatus: interface eth4 is up
>>>>> Dec  6 16:52:06 vinms2 argus[20290]: 06 Dec 11 16:52:06.238769
>> ArgusInterface timestamps wayyy out of order: now 1323190326 then 1811344957
>>>> 
>>>> etc.
>>>> 
>>>> I'll try updating to the latest!
>>>> 
>>>> Best Wishes,
>>>> Chris
>>>> 
>>>> --
>>>> --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
>>>> Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
>>>> IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
>>>> Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094
>>>> 
>>> 
>> 
>> --
>> --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
>> Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
>> IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
>> Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094
>> 
> 
> -- 
> Jesse Bowling
> Security Architect::Office of Information Security::UGA
> jesseb at uga dot edu::706-542-2127
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20111207/77269c10/attachment.bin>