Another vote for packet drop detection

Charles Smutz csmutz at masonlive.gmu.edu
Wed Feb 29 09:59:21 EST 2012


Carter,

I've been very pleased with the packet loss (or rather gap) detection. 
In my experience, it works very well. It is possible to now infer 
problems with network monitor visibility (even tapping issues, etc 
outside the monitor) with a pretty high degree of confidence. 
Furthermore, you have some ability to quantify that loss. Even though 
you can only detect some of the packets that are not seen by network 
monitor but were transferred through network you can extrapolate if you 
want (assuming the loss is randomly distributed). Lastly, that you can 
do this without any special effort on the collection end, is just 
awesome. This opens the door for continuous monitoring of this facet of 
network monitor visibility with little cost.

A couple notes:
-I've found the "g" flag to be reliable, but I've found the sgap and 
dgap to have erroneous values (very infrequently). Not a big issue, but 
wanted to point it out. See example below.
-The logic for flagging gaps seems solid--in my evaluation the vast 
majority of the flows flagged with "g" do involve a packet that was 
transferred through network, but not seen by the monitor. I've found a 
very few where I believe the "g" flag is erroneous, but those have been 
relatively few and I'll admit determining ground truth in most 
situations can be pretty difficult.
-to date (3.0.5.34), I can't seem to get filtering based sgap/dgap to 
work. This isn't a big issue to me, and may well be my fault, but wanted 
to point out.

Thanks,

Charles Smutz

Examples taken from test traces here using argus defaults:
https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=6081

Example showing erroneous gap values (note that flag is not g):
$ ra -nn -s flgs +sbytes +dbytes +sgap +dgap -r 
test-B_trace-1_network-side-packet-loss.argus | head -n 7
  e s              1405         2912        0        0
  e                3920        14286        0        0
  e s              3419         5847        0     4848
  e d               148         3599        0        0
  e d               549         3216        0        0
  e                 330          909        0        0
  e                 330         1054        0        0


Example showing my inability to filter:
$ ra -nn -s flgs +sbytes +dbytes +sgap +dgap -r 
test-B_trace-1_network-side-packet-loss.argus -- "dst bytes gt 0" | head 
-n 7
  e s              1405         2912        0        0
  e                3920        14286        0        0
  e s              3419         5847        0     4848
  e d               148         3599        0        0
  e d               549         3216        0        0
  e                 330          909        0        0
  e                 330         1054        0        0
$ ra -nn -s flgs +sbytes +dbytes +sgap +dgap -r 
test-B_trace-1_network-side-packet-loss.argus -- "dst gap gt 0" | head -n 7
ra[8308]: 09:46:25.470929 dst gap gt 0 filter syntax error
$ ra -nn -s flgs +sbytes +dbytes +sgap +dgap -r 
test-B_trace-1_network-side-packet-loss.argus -- "dst gaps gt 0" | head -n 7
ra[8311]: 09:46:28.365700 dst gaps gt 0 filter syntax error






On 2/1/2012 10:45 PM, argus-info-request at lists.andrew.cmu.edu wrote:
> Send Argus-info mailing list submissions to
> 	argus-info at lists.andrew.cmu.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
> or, via email, send a message with subject or body 'help' to
> 	argus-info-request at lists.andrew.cmu.edu
>
> You can reach the person managing the list at
> 	argus-info-owner at lists.andrew.cmu.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Argus-info digest..."
>
>
> Today's Topics:
>
>     1. Re:  Another vote for packet drop detection (Carter Bullard)
>     2.  argus-3.0.5.10 and argus-clients-3.0.5.31 (Carter Bullard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 1 Feb 2012 15:22:37 -0500
> From: Carter Bullard<carter at qosient.com>
> Subject: Re: [ARGUS] Another vote for packet drop detection
> To: Argus<argus-info at lists.andrew.cmu.edu>
> Message-ID:<AB1A8208-C06B-4DF3-8689-6568ADB1ADCA at qosient.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Gentle people,
> As I test the algorithms for gap detection, a few things are coming up, causing
> me to simplify the strategy a bit.
>
> There are two situations for TCP, and any other connection oriented transport
> protocol.  When we are seeing both sides of the conversation, and when we
> are not.  When we are seeing both sides, we can correlate received ACK's with
> transmitted sequence numbers, and tick a status bit saying we've observed gaps
> if there are discrepancies.   This will be very sensitive, but the condition itself
> is not enough to know how much data was missed.
>
> When we are seeing only one side, we can report gaps in the sequence numbers,
> just by knowing the sequence number range, and comparing that against the
> observed bytes.  As long as we account for sequence numbers out of order
> this works very well.  However, when there are retransmissions, this method
> breaks down.  Our observed bytes within a sequence number range can't
> really be compared to give a realistic value, unless we are tracking retransmitted
> bytes.  Currently argus does not track this metric, it tracks numbers of retransmitted
> packets.  I'll change this for argus-3.1.0.
>
> As a result, we will use the uni-directional algorithm to report gaps as unobserved
> bytes, when there is no evidence of retransmissions.  I'm going to put a ' g ' in the
> status flags field in the same column (#4) as the loss indicators,  when there are gaps.
> I won't worry about the indication from argus, until argus-3.1.0 when we can change
> the "retrans" metrics and their reporting.
>
> Fields are called 'sgap' and 'dgap', and the Column labels are "SrcGap" and "DstGap".
> There is support for printing, including xml, graphing, etc?. but there is no support
> yet for filtering or sorting yet.  That is important, so I'll put that in before
> I release the code.  Should be today or tomorrow.
>
> The other protocol that we should implement this for is RTP.  Because RTCP, the
> reverse control channel for RTP, contains loss values, we can compare the two
> to realize that we are not seeing all the RTP traffic, but RTCP is not reporting loss.
> This would be our indication of gaps in the RTP stream.  I'll work on that much later.
>
> Carter
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20120201/c057c591/attachment-0001.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 4367 bytes
> Desc: not available
> Url : https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20120201/c057c591/attachment-0001.bin
>
> ------------------------------
>
> Message: 2
> Date: Wed, 1 Feb 2012 22:45:31 -0500
> From: Carter Bullard<carter at qosient.com>
> Subject: [ARGUS] argus-3.0.5.10 and argus-clients-3.0.5.31
> To: Argus<argus-info at lists.andrew.cmu.edu>
> Message-ID:<A08D4D55-2D66-403F-BCD2-C3364BD39483 at qosient.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Gentle people,
> New argus and argus-client software is now available on the server:
>
>     http://qosient.com/argus/dev/argus-latest.tar.gz
>     http://qosient.com/argus/dev/argus-clilents-latest.tar.gz
>
> These packages have a number of bug fixes and new features, as discussed
> on the mailing list.  Argus implements new support for gap tracking and
> reporting, and fixes a few bugs in loss reporting, and TCP byte accountability
> for when flows exceed 4GB.  Clients now can report on "sgap" and "dgap"
> metrics.  I still need to implement gap filtering.  For argus data earlier than
> argus-3.0.5.30, gap reporting is not supported.   ra* programs should do
> the right thing.  I tested with argus-3.0.5.30, argus-3.0.4 and argus-2.x data.
>
> There are a few bugs not on the mailing list that have also
> been fixed, particularly, printing user buffers larger than 300+ bytes.
> This would cause a core dump.
>
> Manpages, and usage output when you run a program with the -h option
> should be up to date now.  New manpages are available, and most have
> been updated.
>
> Please take a look at these packages, and provide feedback, if you
> notice anything a miss.
>
> Thanks for all the support,
>
> Carter
>
>
> On Feb 1, 2012, at 3:22 PM, Carter Bullard wrote:
>
>> Gentle people,
>> As I test the algorithms for gap detection, a few things are coming up, causing
>> me to simplify the strategy a bit.
>>
>> There are two situations for TCP, and any other connection oriented transport
>> protocol.  When we are seeing both sides of the conversation, and when we
>> are not.  When we are seeing both sides, we can correlate received ACK's with
>> transmitted sequence numbers, and tick a status bit saying we've observed gaps
>> if there are discrepancies.   This will be very sensitive, but the condition itself
>> is not enough to know how much data was missed.
>>
>> When we are seeing only one side, we can report gaps in the sequence numbers,
>> just by knowing the sequence number range, and comparing that against the
>> observed bytes.  As long as we account for sequence numbers out of order
>> this works very well.  However, when there are retransmissions, this method
>> breaks down.  Our observed bytes within a sequence number range can't
>> really be compared to give a realistic value, unless we are tracking retransmitted
>> bytes.  Currently argus does not track this metric, it tracks numbers of retransmitted
>> packets.  I'll change this for argus-3.1.0.
>>
>> As a result, we will use the uni-directional algorithm to report gaps as unobserved
>> bytes, when there is no evidence of retransmissions.  I'm going to put a ' g ' in the
>> status flags field in the same column (#4) as the loss indicators,  when there are gaps.
>> I won't worry about the indication from argus, until argus-3.1.0 when we can change
>> the "retrans" metrics and their reporting.
>>
>> Fields are called 'sgap' and 'dgap', and the Column labels are "SrcGap" and "DstGap".
>> There is support for printing, including xml, graphing, etc?. but there is no support
>> yet for filtering or sorting yet.  That is important, so I'll put that in before
>> I release the code.  Should be today or tomorrow.
>>
>> The other protocol that we should implement this for is RTP.  Because RTCP, the
>> reverse control channel for RTP, contains loss values, we can compare the two
>> to realize that we are not seeing all the RTP traffic, but RTCP is not reporting loss.
>> This would be our indication of gaps in the RTP stream.  I'll work on that much later.
>>
>> Carter
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20120201/40cf8817/attachment.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 4367 bytes
> Desc: not available
> Url : https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20120201/40cf8817/attachment.bin
>
> ------------------------------
>
> _______________________________________________
> Argus-info mailing list
> Argus-info at lists.andrew.cmu.edu
> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>
>
> End of Argus-info Digest, Vol 78, Issue 2
> *****************************************
>
> .
>




More information about the argus mailing list