Problem with argus under load not reopening output file.

[Carter Bullard]

Hmmm,
Well, if the mailing list can decide how to respond to the bug,  then  
I'll put a fix in this week.
Should we for packet streams that are "PCAP_OPEN_LIVE" interfaces:
    1) Adjust the time for packets that are way out of scope (>  5-10  
seconds away from real time)
        to current time.

    2) drop packets that are way out of scope

In both cases, we should not adjust our notion of global time?

Carter




On Jan 11, 2009, at 12:56 PM, Martijn van Oosterhout wrote:

> Sorry, hit the wrong button. No, its actually a kernel bug:
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d8bb6f4c1670c8324e4135c61ef07486f7f17379
>
> Causes the time as read by gettimeday() to jump an hour in the future
> and back for a packet or two.
>
> At least, I hope it's this, otherwise I don't know.
>
> Have a nice day,
>
> On Sun, Jan 11, 2009 at 5:42 PM, Carter Bullard <carter at qosient.com>  
> wrote:
>> Hey Martijn,
>> If you don't mind, I'd like to keep the thread in the argus mailing  
>> list.
>>
>> Sounds like an ntp or a rdate() issue, where the machine's time is
>> getting modified to a bogus value.  This is of course an old way
>> of attacking a machine ;o)
>>
>> Argus can attempt to catch jumps in time by testing the timestamp
>> in the packets against the systems global time, or a delta of the
>> two.  But, how to respond to the jump is a problem, because
>> we don't know which system (packet capture/host os) is correct.
>>
>> Carter
>>
>> On Jan 11, 2009, at 8:17 AM, Martijn van Oosterhout wrote:
>>
>>> On Sat, Jan 10, 2009 at 5:01 PM, Carter Bullard <carter at qosient.com>
>>> wrote:
>>>>
>>>> Hey Martijn,
>>>> Sorry for the delayed response.  I've gone over the code a bit,  
>>>> and I
>>>> don't
>>>> see how this jump can occur.   But that is the nature of bugs,  
>>>> sometimes.
>>>>
>>>> There are a few things that I need to proceed.  What platform, OS,
>>>> 64-bit?
>>>> What are we connecting to.  1Gbps?  10Gbps?   And, what is the  
>>>> processor
>>>> load for argus?  And the interrupt rate, any since as to how many  
>>>> packets
>>>> per second?
>>>
>>> The load is about 30%, IIRC. The line is 200-250Mbs, so high but not
>>> exceptional. In the meantime there has arisen another theory: namely
>>> that under certain circumstances the Linux kernel version we're
>>> running (2.6.20) can for a very short time return a time 1 hour 13
>>> minutes in the future. If argus happens to pick that one as
>>> GlobalTime, it will cause the symptoms seen.
>>>
>>> As these things go, as soon as you try to verify this is the  
>>> problem,
>>> the problem stops happening... I hope monday I'll have evidence one
>>> way or the other.
>>>
>>>> When this occurs, are we getting any packets at all? (your strace  
>>>> should
>>>> have
>>>> packet reading, since we do a select() )
>>>
>>> It's using the linux PCAP ringbuffer, so there are no kernel calls  
>>> for
>>> fetching the packets, only writes.
>>>
>>>> My suspicion is that if all is as it should be, but we all of a  
>>>> sudden
>>>> get a
>>>> leap in our  global time, argus maybe so loaded that it is getting
>>>> behind.
>>>> But, that is a very preliminary guess.
>>>
>>> That sounds like the most plausible idea so far. It's possible to  
>>> work
>>> around this is argus, the question is should we. In ArgusWrite* if  
>>> you
>>> change the test with lastwritten to be not-equal (!=) rather than
>>> less-than (<) the issue shouldn't arise either.
>>>
>>> I hope to have more info soon.
>>>
>>> Have a nice day,
>>> --
>>> Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/
>>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York  10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>>
>
>
>
> -- 
> Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/
>

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax