[ARGUS] Malfunctioning of argus -S option for some pcaps

[carter at qosient.com]

Hey Hang,
Each flow cache that is being tracked is placed in a single "status” queue, in start time order.  The end of the queue is used to manage the status interval.   The output processor just looks at the end of this queue to decide if it's time to report the flow cache's values.  This saves having to manage a timer for each flow cache, which can number in the millions.

Of course, each flow cache entry has an independent timestamp, the first packet’s timestamp, which is the “start time".  This works, because argus, as a real-time flow sensor, reads packets off the wire, where all the packets MUST come in in time order (nature of networking a time stamping each packet as it comes in, using the same clock).  When argus reads packets from a file, or a stream of packets from a distributed packet broker, it gets a little weird, as the arrival of packets are not real-time, and can be delayed, shaped and / or reordered.  

To deal with the weirdness, argus uses a very specific strategy.  When not reading realtime packets using a real-time sensor, you have to decide if you want to “believe" the timestamp in the packet or do you instead use the realtime system clock when argus receives the packet (whenever that might be).  Argus uses the timestamp provided by the packet capture facility as the realtime clock, and plays a lot of games trying to figure out what to do next when there are gaps in time.  The reasons are huge, from a practical perspective, as you want deterministic results when processing packets.  The only requirement is that the packets come in mostly in time order.

Carter
       <http://qosient.com/>    	 	
Carter Bullard   <mailto:carter at qosient.com>•  Chairman / Founder / CTO
150 E 57th Street, Suite 12D
New York, New York 10022-2795
Phone +1.212.588.9133 • Mobile +1.917.497.9494

 

> On Jul 2, 2019, at 10:48 PM, Hang Guo <hangguo at usc.edu> wrote:
> 
> Thanks Carter, that saves life!!!!
> 
> Could you help me understand one more bit about -S option: when implemting this report interval (-S), I guess argus uses one timer for all flows instead of a seperate timer for each flow right?
> 
> Thanks,
> -Hang 
> 
> On Tue, Jul 2, 2019 at 6:28 PM <carter at qosient.com <mailto:carter at qosient.com>> wrote:
> Hey Hang,
> The problem is that your packets are not in time order, so argus is doing the right thing.  Best example is around packet # 3848, where the packet timestamps jumps back 6.5 hours.  Nothing argus can do with that but tally the packets … and all the flows will be messed up for a little while, until the packet timestamps move ahead in front of the largest startime.  If you can get the packets sorted in time, then things should work fine.
> 
> Carter
> 
>> On Jul 2, 2019, at 4:19 PM, Hang Guo <hangguo at usc.edu <mailto:hangguo at usc.edu>> wrote:
>> 
>> Hi,
>> 
>> I found argus -S option malfunctioning for some pcaps. For example, when running argus -S 10 with the pcap attached (MAC and IP anamoyzed, payload dropped for privacy), instead of reportting every 5-tuple flows every 10 seconds, duration of some reported 5-tuple flows (as pasted below) are hundreds of seconds. Just wonder what is the possible cause and is there a fix to this?
>> 
>> argus -S 10 -r dur_test_anon.pcapng -w - | ra -c "," -r - -s dur | sort -nr | head -5
>> 4865.550781
>> 296.601562
>> 296.393066
>> 294.411255
>> 292.840790
>> 
>> Thanks,
>> -Hang 
>> <dur_test_anon.pcapng>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20190703/02580784/attachment.html>