Duplicate packets
elof2 at sentor.se
elof2 at sentor.se
Wed Oct 9 10:20:51 EDT 2013
On Tue, 1 Oct 2013, Carter Bullard wrote:
> Well, hmmmmmmm…. Everyone else wants to do de-duping of the packet stream.
> Why would you want to be different from everyone else ????? ;O)
I'm curious. What exactly is it that everyone wants (or not wants)?
> The strategy is to differentiate loss, retrans and dups, and report
> them as independent metrics, with loss being observable loss, retrans
> being observable duplicates, and dups (for TCP) being retrans arriving
> in less than an RTT.
I don't fully agree about the dups.
A dupe is, in my opinion, an *exact* copy of the original packet.
A retransmission is not a dupe, it is a new packet, crafted because the
original supposedly got lost.
Therefore, the logic need not be so expensive. If the very next packet is
identical to the last packet and it was received within a microsecond
from the last packet then it is a dupe.
Taking the RTT into consideration seem a bit excessive for the simple task
of dupe recognition. Also, a RTT is not always possible to calulate if the
flow only consist of a single SYN, or is just unidirectional UDP traffic,
etc, but the packets of the flow are still duplicated.
> We still will need to derive gaps, which are lost packets that were not
retransmitted.
Oooh, are you talking about distinguishing between external loss and
internal loss?
When argus see a gap in tcp sequence numbers you know there has been a
drop, but not where it occurred.
If argus then see a tcp retransmission for that gap, we know the drop was
external, otherwise it was probably internal.
That kind of logic seem expensive. If it *is* really expensive, I would
say don't do it. Only do the first gap detection (as you already do today)
and leave the task of understanding where the drops occurr to the user.
/Elof
> On Oct 1, 2013, at 1:58 PM, elof2 at sentor.se wrote:
>
>>
>> Hi Carter!
>>
>> Your response made no sense to me. I have never said anything about packet rejection in argus.
>>
>>
>> So let me start over:
>>
>> Given the below (faulty) mirrored packets, I would like the command
>> 'ra -Zb -nr argus.log' to output something like this:
>>
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
>> 12:55:41.558571 M = tcp 1.1.1.1.49369 -> 93.93.93.93.80 8 6 3408 4228 FSPA_FSPA
>> ^
>> |
>> flag that tells me there are duplicate packets in this flow
>>
>> That's it. That's all I'm asking for.
>>
>>
>>
>>
>>
>>
>> When the SPAN is correct (no duplicates), this line would read:
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
>> 12:55:41.558571 M tcp 1.1.1.1.49369 -> 93.93.93.93.80 4 3 1704 2114 FSPA_FSPA
>>
>>
>>
>>
>>
>> One way to do this:
>>
>>
>> The current ra manual states:
>> * - Both Src and Dst loss/retransmission
>> s - Src loss/retransmissions
>> d - Dst loss/retransmissions
>> g - Gaps in sequence numbers were observed
>> & - Both Src and Dst packet out of order
>> i - Src packets out of order
>> r - Dst packets out of order
>> Add these new flags:
>> = - Both Src and Dst duplicates
>> S - Src duplicates
>> D - Dst duplicates
>> (this is just an example, perhaps other letters are better, and perhaps another flag-column should be used?)
>>
>>
>>
>> Example of SPAN duplicates for one flow:
>> These can happen if you configure your switch to mirror a VLAN using the 'both' keyword instead of using 'rx' or 'tx':
>>
>> 1.1.1.1:49369 -> 93.93.93.93:80 S
>> 1.1.1.1:49369 -> 93.93.93.93:80 S <duplicate from badly configured SPAN>
>> 93.93.93.93:80 -> 1.1.1.1:49369 SA
>> 93.93.93.93:80 -> 1.1.1.1:49369 SA <duplicate>
>> 1.1.1.1:49369 -> 93.93.93.93:80 A
>> 1.1.1.1:49369 -> 93.93.93.93:80 A <duplicate>
>> 1.1.1.1:49369 -> 93.93.93.93:80 PA GET /foo HTTP/1.0
>> 1.1.1.1:49369 -> 93.93.93.93:80 PA GET /foo HTTP/1.0 <duplicate>
>> 93.93.93.93:80 -> 1.1.1.1:49369 FPA some server response
>> 93.93.93.93:80 -> 1.1.1.1:49369 FPA some server response <duplicate>
>> 1.1.1.1:49369 -> 93.93.93.93:80 FA
>> 1.1.1.1:49369 -> 93.93.93.93:80 FA <duplicate>
>> 93.93.93.93:80 -> 1.1.1.1:49369 A
>> 93.93.93.93:80 -> 1.1.1.1:49369 A <duplicate>
>>
>> For the above data, I'd like ra to set the flag '=' to indicate that this flow has duplicated mirrored traffic in both directions.
>>
>>
>>
>>
>>
>> For me, the flags '=', 'S' or 'D' is enough. Argus don't need to count the amount of duplicates or simillar. I'm only interested in having a visual way to see if lots of flows have a strange flag, just like looking for the 'g' flag for tcp gaps.
>> If you want to add 'sdup' and 'ddup' fields, just like 'sgap' and 'dgap', feel free to do so. Personally I don't need that level of verbosity though.
>>
>> PS. In my opinion, these new flags should have *lower* priority than the other ones in this column. That is, flags for retransmissions etc should have precedence over the '=', 'S' and 'D' flags. (or you could reuse another column for this, like the first or second column with less risc of multiple flag collissions).
>>
>>
>>
>> Today, 'ra' count the duplicates above as retransmissions. This is wrong, since in the SPAN above there are no tcp retransmissions. A retransmission is a *new* packet with a new IPid, meaning it differs from the original packet. Duplicates are the immediate SPAN-tx-copy arriving right after the SPAN-rx-copy. The tx-copy is 100% identical to the rx-copy and the tx-copy always arrive immediately after the rx-copy, no other mirrored packet can come in between these two packets. Also, when a retransmission is received by the sniffer, timewise this is WAAAAAAY longer than the time of the duplicate.
>> ...therefore, I think it would be a simple task to track duplicates and distinguish them from retransmissions.
>>
>> /Elof
>>
>>
>>
>>
>> On Thu, 26 Sep 2013, Carter Bullard wrote:
>>
>>> Hey /Elof,
>>> So this is, again, a somewhat sticky situation, as I'm thinking that
>>> there are a few situations in which rejecting a duplicate/identical
>>> packet is the wrong thing to do, so I'm still at a loss as to how
>>> to approach this problem. If I could get some more packet traces
>>> that had duplicates in them, I maybe able to work something out.
>>>
>>> I haven't been idle on this topic. We've enhanced the flow keys to
>>> try to count the duplicates, but discriminate the dups and track them
>>> in their own flows. I believe this is the right thing to do, as the
>>> packets are really on the wire, and we're all about telling you
>>> about what is really on the wire. I have implemented silly packet
>>> rejection, which does get close to the situations you described,
>>> but I'm non thinking its the best solution.
>>>
>>> Send some sample packet files, and lets discuss in detail.
>>> It will be a key feature in post argus-3.0.8, which is coming
>>> up in hopefully a few weeks.
>>>
>>> Carter
>>>
>>>
>>> On Sep 24, 2013, at 10:16 AM, elof2 at sentor.se wrote:
>>>
>>>>
>>>> Hi Glen and Carter.
>>>>
>>>> Glen, Carter and I discussed this problem some years back, but I don't think anything was officially released.
>>>>
>>>> The thoughts at that time was that:
>>>> * argus receive a packet from SPAN
>>>> * argus do its processing
>>>> * a new packet is received from SPAN
>>>> * argus compares if
>>>> * this very next packet is identical to the previous one and
>>>> * it arrived within an extremely short period of time
>>>> = tag this flow as having duplicates instead of retransmissions.
>>>>
>>>> The faulty SPAN will always send the two identical packets directly after eachother. No other packet will be inserted in between, so no need for argus to check longer back than the very previous packet.
>>>>
>>>> There might be situations where identical packets are seen on the wire that are NOT SPAN-duplicates. Say you have a completely silent network except for some machine that send an UDP broadcast announcement every second.
>>>> This packet could be identical to the last one. Same ports, same TTL, same MAC-addresses and even the same IPid (usually set to 0).
>>>> These packets won't be considered duplicates since they weren't received extremely fast after the first packet.
>>>>
>>>>
>>>> The result if argus had this functionality would be to
>>>> a) detect bad/faulty SPAN setup (which is very common)
>>>> b) no longer log duplicates as restransmissions as is done today
>>>>
>>>>
>>>>
>>>> In Feb 2012, Carter wrote:
>>>> "The duplicates, such as multiple copies of the exact same packet, is detectable and I put code in to do this"
>>>> *snip*
>>>>
>>>> I don't think that code was ever tested or put into production though.
>>>> Do you remember, Carter?
>>>>
>>>> /Elof
>>>>
>>>>
>>>> On Wed, 18 Sep 2013, Carter Bullard wrote:
>>>>
>>>>> Hey Glen,
>>>>> No, at least not at the moment, there isn't any logic in argus to realize that there are dups on the wire, but there are indirect methods for the clients to detect this.
>>>>>
>>>>> Things like very high retransmit rates, i.e. 50-100% for TCP, flow rates that exceed known rates for specific flows, and duplicates for transactions that should not have duplicates, like DNS, are all indications of packet collection duplication that is not really on the wire.
>>>>>
>>>>> I'm not sure that we should put it in argus, but we could do something in the clients !!!
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>>
>>>>> On Sep 18, 2013, at 5:34 AM, Glen Bojsza <gbojsza at finepoint.com> wrote:
>>>>>
>>>>>> Is it possible or is there way that would allow argus to notify the user of duplicate packets detected?
>>>>>>
>>>>>> For example we are looking to alert a user if the switch mirror port is configured incorrectly and is reporting packets twice, once on ingress and once on egress.
>>>>>>
>>>>>> regards
>>>>>> GB
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
More information about the argus
mailing list