Duplicate detection
Carter Bullard
carter at qosient.com
Wed Jul 1 11:54:57 EDT 2015
Well, I start counting at 0 … bad habit I suppose.
The fragment column only uses [V,v,F] so we have a lot to choose from.
We do track the src and dst dup occurences, so why not ’s’ and ‘d’ ??
For anything but RTP traffic, there isn’t an ’s’ or ‘d’ in the right
half of the flags field ??? For RTP we use them for silence suppress
indicators.
Carter
> On Jul 1, 2015, at 11:44 AM, elof2 at sentor.se wrote:
>
>
> Your reasoning is sane.
> Actually, you mean column 7 and 8 according to the ra manual.
>
> So col 7 (fragments) sounds good.
>
> Dupes are *least* important.
> Since they will appear on more or less all IP flows, it is unnecessary to flag them all.
> Instead, the dupe-flag could have lower precedence and only show up on ip-flows that don't have any other column-7-flag.
> That will still be almost all ip-flows, so the SPAN dupe problem is easily spotted.
>
> In my opinion, column 7 don't need a * (multiple conditions detected) flag. V, f and F could have precedence over the dupe flag.
>
> Choise of dupe character:
> D - dupe nah, confusing with "dst" and "Dst Win Closure"
> c - copy
> d - dupe nah, confusing with "dst"
> # - indicating more than 1
> ! - indicating a problem
> / - indicating a problem
>
> I think I like the '!' character best.
>
> /Elof
>
>
> On Wed, 1 Jul 2015, Carter Bullard wrote:
>
>> Well the problem with that column, is that its for connection
>> state reporting (TCP, UDT, RTP, ESP, +), and its overloaded.
>> Many connections have multiple issues, and we end up either
>> having a precedence issue (which one to you report), or we
>> use ‘*’ to indicate multiple occurences.
>>
>> So are dups the most important ???
>>
>> Because this dup detection method is ipid based, we have
>> column 6 and 7 just for IP (fragments and IP options) which
>> we can use. Maybe col 6, fragments, since that has an
>> ipid angle to it ??
>>
>> Carter
>>
>>
>>> On Jul 1, 2015, at 9:12 AM, elof2 at sentor.se wrote:
>>>
>>>
>>> Hi Carter!
>>>
>>> That sounds nice!
>>>
>>> I'm curious... What flag(s) will you use and in what flag-char-column?
>>> I think column 4 (with retransmissions, gaps and out of order) is fitting.
>>>
>>> /Elof
>>>
>>>
>>> On Wed, 1 Jul 2015, Carter Bullard wrote:
>>>
>>>> Hey /Elof,
>>>> Yes, more like non-sequential ipid. Yep, have to have a value other than 0 to test.
>>>> So this turned on in the argus.conf file. I this is reported in the flags, and you will
>>>> be able to filter for dups. I have a version that reports the dups, but so far I have
>>>> it modifying the TCP performance DSR, which would not be backward compatible.
>>>>
>>>> I’ll generalize the approach, and have it report the occurrence, and then for 3.0.10,
>>>> we’ll see if anyone needs/wants duplicate reporting.
>>>>
>>>> Carter
>>>>
>>>>> On Jul 1, 2015, at 4:32 AM, elof2 at sentor.se wrote:
>>>>>
>>>>>
>>>>> Hi Carter!
>>>>>
>>>>> On Tue, 30 Jun 2015, Carter Bullard wrote:
>>>>>
>>>>>> So there will be a new argus-3.0.8.2, and that will have the argus parts
>>>>>> of the puzzle. We’re still holding out on argus-clients-3.0.8.[1 | 2],
>>>>>> which will have a bunch of the support for this.
>>>>>
>>>>> Sounds great.
>>>>>
>>>>>
>>>>>> OK, so duplicate detection, I have it so that we know how many dups there
>>>>>> are. This is a tough one to figure out what to do. Should the flow records
>>>>>> present what it saw on the wire with regard to total packets and bytes, but
>>>>>> report the number of dups ?? Should we not count the duplicated packets and
>>>>>> bytes in the flow total, but report dups so you can understand total load
>>>>>> on the flow ???
>>>>>> I’m more inclined to report all data observed in the flow record, but
>>>>>> indicate that there are dups, so you can realize there is a problem.
>>>>>
>>>>> I agree with you. I'm all for the latter. The flag is enough to indicate the problem.
>>>>> All flow records should present totals of what they actually see on the wire.
>>>>> For me, counting the dupes is not important. A flag is sufficient.
>>>>> (But if it is easy to do and have little performance impact, naturally you could also add a counter field.)
>>>>>
>>>>>
>>>>> BTW, when you say "Duplicate detection is working based on sequential ipid" I assume you mean that argus see the same ipid twice immediately after the previous one? (not 'sequential' as in 1, 2, 3, 4, 5...)
>>>>>
>>>>> If so, remember to exclude ipid '0' from the dupe test, since id '0' can be present multiple times (fragmented packets).
>>>>>
>>>>> /Elof
>>>>>
>>>>>
>>>>>>> Hi Carter!
>>>>>>>
>>>>>>> That sounds most excellent! All of it. Thanks.
>>>>>>>
>>>>>>> I'll find an old email regarding the gap problems and reply to your question below in that thread instead.
>>>>>>>
>>>>>>> /Elof
>>>>>>>
>>>>>>>
>>>>>>> On Mon, 29 Jun 2015, Carter Bullard wrote:
>>>>>>>
>>>>>>>> Fixed your counting bug.
>>>>>>>> Configurable timeout on filters implemented.
>>>>>>>> Deadman error msg sez filter timed out.
>>>>>>>> Syntax error msg doesn’t print the entire filter.
>>>>>>>> Duplicate detection is working based on sequential ipid.
>>>>>>>> Need to export the duplicate stats in a new TCP DSR extension.
>>>>>>>> I’ll add that before I release 3.0.8.2.
>>>>>>>> Currently looking at gaps.
>>>>>>>> So, to refresh my memory, we’re reporting gas when there is TCP
>>>>>>>> port reuse. Old TCP is done, new TCP uses same ports and we
>>>>>>>> don’t adjust the gap reporting ???
>>>>>>>> Carter
>>>>>>>>> On Jun 17, 2015, at 10:17 AM, elof2 at sentor.se wrote:
>>>>>>>>> Hi Carter!
>>>>>>>>> Any comments? I guess not, just a simple fix in the code.
>>>>>>>>> /Elof
>>>>>>>>> On Tue, 2 Jun 2015, elof2 at sentor.se wrote:
>>>>>>>>>> Hi Carter!
>>>>>>>>>> I don't understand how _not_ counting someting yields 1 too much. :-)
>>>>>>>>>> Anyhow, I think that all MAR records that exist in the file should be counted and TotalMarRecords show the correct sum.
>>>>>>>>>> In my example, TotalMarRecords should be 16, not 17.
>>>>>>>>>> /Elof
>>>>>>>>>> On Tue, 2 Jun 2015, Carter Bullard wrote:
>>>>>>>>>>> Yes, in fact you included my email that stated that we don’t count the first MAR.
>>>>>>>>>>> Do you think we should ??? Seems redundant since the first MAR must be there ??
>>>>>>>>>>> Carter
>>>>>>>>>>>> On May 27, 2015, at 5:40 AM, elof2 at sentor.se wrote:
>>>>>>>>>>>> Hi Carter!
>>>>>>>>>>>> I investigated this a little bit further.
>>>>>>>>>>>> I have this logfile with 4 MAR records:
>>>>>>>>>>>> ra -Zb -M man -nr elof.log | grep -i man
>>>>>>>>>>>> 10:28:01.149822 man 0 0 0 0 0 0 0 0 STA
>>>>>>>>>>>> 10:28:44.149779 man 0 0 25749 1 69980 2839 28277263 0 CON
>>>>>>>>>>>> 10:29:44.149756 man 0 0 25191 1 78426 3104 36560040 0 CON
>>>>>>>>>>>> 10:30:44.150890 man 0 0 25060 1 107220 2715 37037618 0 CON
>>>>>>>>>>>> I concatenate it four times:
>>>>>>>>>>>> cat elof.log elof.log elof.log elof.log >> elof2.log
>>>>>>>>>>>> ra -Zb -M man -nr elof2.log | grep -i man
>>>>>>>>>>>> Now I have 16 MAR records.
>>>>>>>>>>>> So far everything is sane and logical.
>>>>>>>>>>>> The file has 38852 flows (I checked with wc -l).
>>>>>>>>>>>> The file has 16 MAR records.
>>>>>>>>>>>> Total records should therefore be 38868.
>>>>>>>>>>>> I now add -A :
>>>>>>>>>>>> ra -Zb -M man -A -nr elof2.log | tail -1
>>>>>>>>>>>> Totalrecords 38868 TotalMarRecords 17 TotalFarRecords 38852 TotalPkts 1211504 TotalBytes 462893084
>>>>>>>>>>>> So, the problem is that TotalMarRecords show 1 too much.
>>>>>>>>>>>> It should be 16.
>>>>>>>>>>>> /Elof
>>>>>>>>>>>> On Tue, 26 May 2015, Carter Bullard wrote:
>>>>>>>>>>>>> Hey /Elof,
>>>>>>>>>>>>> We are not counting the first MAR record. If you were to filter the call using "not man" or "far", it should be correct. All streams have to have the first MAR record, so didn't think that we should count it ??
>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>> On May 26, 2015, at 10:50 AM, elof2 at sentor.se wrote:
>>>>>>>>>>>>>> Hi Carter!
>>>>>>>>>>>>>> Just found a silly error.
>>>>>>>>>>>>>> When adding option
>>>>>>>>>>>>>> -A Print aggregate statistics for the input stream on termination.
>>>>>>>>>>>>>> I get this line:
>>>>>>>>>>>>>> Totalrecords 26282 TotalMarRecords 12 TotalFarRecords 26271 TotalPkts 1069033 TotalBytes 654980030
>>>>>>>>>>>>>> The silly error is that 26271+12=26283, not 26282.
>>>>>>>>>>>>>> Very minor, but still wanted you to know. :)
>>>>>>>>>>>>>> /Elof
>>>>>>
>>>>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150701/62034966/attachment.bin>
More information about the argus
mailing list