argus-3.x request (forwarded)

[Carter Bullard]

Hey Martin,
OK, well I can put in a simple duplicate packet detection scheme based on time and
equivalency.  If I see the same packet within say 10 uSec, count it as a duplicate and
set a status bit.   Would that catch your condition?

You can have duplicates that are not exact packets (seems weird) in constrained
sensors.  The criteria for a duplicate packet is the same as the criteria for the
flow classification (5-tuple for most of these systems), and it is a subset of all
the attributes in the packet.  Some of the packet attributes actually specify scope
for the 5-tuple ids, such as VLAN tags, MPLS labels, etc.....  If you only look at
a few attributes, packets can appear to be duplicates, but not at all the same packet.

So the scenario I was using, goes something like this:

Some people will port mirror the ingress and egress ports of their router uplink interface
(as an example) to a probe that is on another interface.  The probe is attached to the
network using only that one interface.  If the probe talks to something where the
traffic goes through the uplink, then the probe see's its own traffic from its promiscuous
interface, and then it gets the same packet copied from the mirror, which is one router
interface away.  

From the perspective of a limited 5-tuple flow monitor,  these packets are duplicates.
The reality is that these packets are from two different "observation domains", but
the sensor doesn't know.

You also see this with overlay networks, where the same Layer 3 packet goes by
unmodified on the same wire, but in different Layer 2 tunnels.   Or when you track
UDP traffic from separate private networks that are tunneled onto the same wire,
like you will see in core networks.

So, what do you think, 10 uSecs?  50 uSecs?  100?   Maybe 50.  If you can get that
packet capture file, I'll be able to get your situation down.

Carter



On Jan 26, 2010, at 5:52 AM, Martin Xxxxx wrote:

> 
> On Mon, 25 Jan 2010, Carter Bullard wrote:
>> Problem is the duplicates occur for all protocols, so when there are
>> duplicates, you end up with counter problems for all the flows, etc...
>> If we're going to fix it, I think we should fix it well.
> 
> Ok. To fix it for real (not just a quick fix), I believe it will be much more work - 'cause then you have to consider cases where the duplicate is *not* the immediate next received packet.
> This implies you have to have a short history of the last received packets + timestamps and match every new received packet against it. This will "waste" CPU looking for problems that shouldn't be there in the first place.
> 
> So all this start to feel a bit overwhelming, just to fix the simple problem of having possible false statistics for the Retransmission counters -- *if* the SPAN setup was setup incorrectly in the first place. :-)
> 
>> The duplicate, as I suggested, isn't always an exact copy,  (say if you
>> port mirror the egress port of the router, rather than the ingress port you're
>> connected to).   The TTL's will have changed, the L2 addresses will be
>> completely different etc.....
> 
> I don't quite understand...
> Are you talking about a packet that has entered a router, been routed (TTL reduced by 1) and then sent out again?
> This is is simple routing. It is not a duplicate packet.
> 
> The duplicates I'm talking about occurr when you configure e.g. a Cisco 6509 with the keyword 'both' (rx AND tx). Like: 'monitor session 1 source vlan nnn both' or if you monitor two ports with 'monitor session 1 source interface gigabit <port 1 and port 2> both'
> 
> 1.
> An incoming packet to port 1 will be copied to the destination port when received (rx).
> 
> 2.
> The packet will be copied to the destination port *again* when port 1 send (tx) the packet elsewhere, like to a router.
> This second copy is a 100% duplicate of the previous one. (note: this process is so fast that other mirrored packets seldom (never?) manage to cut in between.)
> 
> 3.
> The router receive the packet, route it and send it to another VLAN where port 2 is a member.
> When port 2 receive (rx) the packet, a copy is sent to the destination port.
> 
> 4.
> When port 2 transmit (tx) the packet, another copy is sent to the destination port (a duplicate).
> 
> 
> In this example you have two duplicates, packets 2 and 4.
> To be clear: the 1st and 3rd packets seen by the sensor are NOT duplicates.
> 
> 
> PS: Duplicates could probably also occurr if you have short-circuited your network or if a NIC/machine is behaving badly (I once saw an ILO interface retransmit all broadcasts/multicasts it received)
> 
> 
> 
> I leave to you to figure out if it is worth the effort to detect duplicates (and distinguish them from TCP Retransmissions).
> 
> /Martin
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100126/6cb43d2a/attachment.bin>