Duplicate detection

[elof2 at sentor.se]

Hi Carter!

On Tue, 30 Jun 2015, Carter Bullard wrote:

> So there will be a new argus-3.0.8.2, and that will have the argus parts
> of the puzzle.  We’re still holding out on argus-clients-3.0.8.[1 | 2],
> which will have a bunch of the support for this.

Sounds great.


> OK, so duplicate detection, I have it so that we know how many dups there
> are.  This is a tough one to figure out what to do.  Should the flow records
> present what it saw on the wire with regard to total packets and bytes, but
> report the number of dups ??  Should we not count the duplicated packets and
> bytes in the flow total, but report dups so you can understand total load
> on the flow ???
> I’m more inclined to report all data observed in the flow record, but
> indicate that there are dups, so you can realize there is a problem.

I agree with you. I'm all for the latter. The flag is enough to indicate 
the problem.
All flow records should present totals of what they actually see on the 
wire.
For me, counting the dupes is not important. A flag is sufficient.
(But if it is easy to do and have little performance impact, naturally you 
could also add a counter field.)


BTW, when you say "Duplicate detection is working based on sequential 
ipid" I assume you mean that argus see the same ipid twice immediately 
after the previous one? (not 'sequential' as in 1, 2, 3, 4, 5...)

If so, remember to exclude ipid '0' from the dupe test, since id '0' can 
be present multiple times (fragmented packets).

/Elof


>> Hi Carter!
>>
>> That sounds most excellent! All of it. Thanks.
>>
>> I'll find an old email regarding the gap problems and reply to your question below in that thread instead.
>>
>> /Elof
>>
>>
>> On Mon, 29 Jun 2015, Carter Bullard wrote:
>>
>>> Fixed your counting bug.
>>> Configurable timeout on filters implemented.
>>> Deadman error msg sez filter timed out.
>>> Syntax error msg doesn’t print the entire filter.
>>> Duplicate detection is working based on sequential ipid.
>>> Need to export the duplicate stats in a new TCP DSR extension.
>>> I’ll add that before I release 3.0.8.2.
>>> Currently looking at gaps.
>>> So, to refresh my memory, we’re reporting gas when there is TCP
>>> port reuse.  Old TCP is done, new TCP uses same ports and we
>>> don’t adjust the gap reporting ???
>>> Carter
>>>> On Jun 17, 2015, at 10:17 AM, elof2 at sentor.se wrote:
>>>> Hi Carter!
>>>> Any comments? I guess not, just a simple fix in the code.
>>>> /Elof
>>>> On Tue, 2 Jun 2015, elof2 at sentor.se wrote:
>>>>> Hi Carter!
>>>>> I don't understand how _not_ counting someting yields 1 too much. :-)
>>>>> Anyhow, I think that all MAR records that exist in the file should be counted and TotalMarRecords show the correct sum.
>>>>> In my example, TotalMarRecords should be 16, not 17.
>>>>> /Elof
>>>>> On Tue, 2 Jun 2015, Carter Bullard wrote:
>>>>>> Yes, in fact you included my email that stated that we don’t count the first MAR.
>>>>>> Do you think we should ???  Seems redundant since the first MAR must be there ??
>>>>>> Carter
>>>>>>> On May 27, 2015, at 5:40 AM, elof2 at sentor.se wrote:
>>>>>>> Hi Carter!
>>>>>>> I investigated this a little bit further.
>>>>>>> I have this logfile with 4 MAR records:
>>>>>>> ra -Zb -M man -nr elof.log | grep -i man
>>>>>>> 10:28:01.149822              man                  0      0 0      0 0 0            0            0           STA
>>>>>>> 10:28:44.149779              man                  0      0 25749      1 69980     2839     28277263            0           CON
>>>>>>> 10:29:44.149756              man                  0      0 25191      1 78426     3104     36560040            0           CON
>>>>>>> 10:30:44.150890              man                  0      0 25060      1 107220     2715     37037618            0           CON
>>>>>>> I concatenate it four times:
>>>>>>> cat elof.log elof.log elof.log elof.log >> elof2.log
>>>>>>> ra -Zb -M man -nr elof2.log | grep -i man
>>>>>>> Now I have 16 MAR records.
>>>>>>> So far everything is sane and logical.
>>>>>>> The file has 38852 flows (I checked with wc -l).
>>>>>>> The file has 16 MAR records.
>>>>>>> Total records should therefore be 38868.
>>>>>>> I now add -A :
>>>>>>> ra -Zb -M man -A -nr elof2.log | tail -1
>>>>>>> Totalrecords 38868     TotalMarRecords 17        TotalFarRecords 38852 TotalPkts 1211504  TotalBytes 462893084
>>>>>>> So, the problem is that TotalMarRecords show 1 too much.
>>>>>>> It should be 16.
>>>>>>> /Elof
>>>>>>> On Tue, 26 May 2015, Carter Bullard wrote:
>>>>>>>> Hey /Elof,
>>>>>>>> We are not counting the first MAR record.  If you were to filter the call using "not man" or "far", it should be correct.  All streams have to have the first MAR record, so didn't think that we should count it ??
>>>>>>>> Carter
>>>>>>>>> On May 26, 2015, at 10:50 AM, elof2 at sentor.se wrote:
>>>>>>>>> Hi Carter!
>>>>>>>>> Just found a silly error.
>>>>>>>>> When adding option
>>>>>>>>>    -A  Print aggregate statistics for the input stream on termination.
>>>>>>>>> I get this line:
>>>>>>>>> Totalrecords 26282     TotalMarRecords 12        TotalFarRecords 26271 TotalPkts 1069033  TotalBytes 654980030
>>>>>>>>> The silly error is that 26271+12=26283, not 26282.
>>>>>>>>> Very minor, but still wanted you to know. :)
>>>>>>>>> /Elof
>
>