Another vote for packet drop detection

[Peter Van Epp]

On Mon, Jan 30, 2012 at 11:23:17AM -0500, Carter Bullard wrote:
> Hey Peter,
> Yes, argus already does what you are suggesting, tracking the bi-directional
> state of TCP to determine what is lost, what is retransmitted, what is not seen, etc?
> So, we can generate the data needed to drive this analytic, I believe. 
> 
> Argus can't do what I imagine wireshark is doing.  I suspect that wireshark is storing
> all the sequence numbers seen on a specific TCP, and when an ACK shows up, it
> then searches its list to see that it didn't see a sequence of bytes in the window.
> Even wireshark can't know the number of packets that are missing, I assume that it
> assumes that all of its missing bytes are in a minimum number of packets.
> 

	Without having looked at the wireshark code (only the suggested fix
which I suspect is indeed the tip of the iceburg :-)), I expect you are 
correct. I saw the addition of a single new variable to track acked state 
on the monitored link separate from that on the wireshark link and thought 
that shouldn't be to expensive. However you are correct, wireshark is saving
all the packets (and a limited by buffer size number of those) and thus has
much more data than argus can keep. 
	The packet trace(s) that demonstrate the problem in wireshark are 
available from the bug report at

https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=6081

in the test traces attachment I believe. I think from the commentary that one
of those is a trace of a successful (on the monitored link) http connection
with one packet removed from the pcap to simulate loss on the tapped link 
that didn't occur on the monitored link which should demonstrate the condition

> This type of algorithm can't perform at line rate (1Gbps +) while tracking 1M+ 
> concurrent connections, which is a performance target for argus, so I'll try to
> derive a metric from our existing loss algorithm.   We should be able to report 
> a reliable "unobserved bytes" metric.
> 
> Argus, however, IMHO, is not capable of realizing that a specific dynamic in a
> specific flow should be tallied and reported as a sensor loss metric to be reported
> in a MAR record.  There just isn't enough information, at the time a packet shows
> up, or when a flow record is exported, to be able to unambiguously declare that
> the "haven't seen this packet" statistic is really sensor loss (packet may actually
> show up 1 nSec after we declared that the packet didn't show up, or we don't
> know what the current libpcap drop rate is).
> 

	Thinking about this, the correct answer may well be entirely external
to argus. I suspect the right thing to do is to insert known test traffic 
in to the monitored link and collect data on the flow from the 2 test endpoint
machines and compare that with what argus saw of the selected flow. That should
have almost the same effect (i.e. detecting loss in the sensor link after the
tap point) as the loss metric we are discussing. It is more expensive, because 
it is more intrusive (eating bandwith on the main link for testing) but may be
the only reasonable way to go as well. 


> So,  I will recommend that the sensor metric should be derived by an an argus client
> looking at a specific argus data stream.  It would look at the TCP gaps, (and any other
> connection oriented protocol's gap metric, like RTP or UDT that we should also add),
> compare it with the libpcap packet drop stat in the MAR status, and then figure out
> if the gaps are hidden loss.
> 
> I would like to report the new TCP flow dynamic "gaps" in the TCP performance
> DSR, and then design a client that will read those records and figure it out.  Gaps
> would be "TCP traffic not seen", which I'm hoping can be reported in bytes.
> 
> So if I report TCP gaps, which I think is a good new metric, can you guys build a
> client that will take that indication and figure it out?
> 
> Carter
> 

	We can certainly try, although I don't have access to a well 
instrumented network any more since I retired, I've only got argus on my
ADSL line at home these days :-)

Peter