Aggregate statistics off by 1

[Carter Bullard]

So there will be a new argus-3.0.8.2, and that will have the argus parts
of the puzzle.  We’re still holding out on argus-clients-3.0.8.[1 | 2],
which will have a bunch of the support for this.

OK, so duplicate detection, I have it so that we know how many dups there
are.  This is a tough one to figure out what to do.  Should the flow records
present what it saw on the wire with regard to total packets and bytes, but
report the number of dups ??  Should we not count the duplicated packets and
bytes in the flow total, but report dups so you can understand total load
on the flow ???

I’m more inclined to report all data observed in the flow record, but
indicate that there are dups, so you can realize there is a problem.

Carter

> On Jun 30, 2015, at 12:39 PM, elof2 at sentor.se wrote:
> 
> 
> Hi Carter!
> 
> That sounds most excellent! All of it. Thanks.
> 
> I'll find an old email regarding the gap problems and reply to your question below in that thread instead.
> 
> /Elof
> 
> 
> On Mon, 29 Jun 2015, Carter Bullard wrote:
> 
>> Fixed your counting bug.
>> Configurable timeout on filters implemented.
>> Deadman error msg sez filter timed out.
>> Syntax error msg doesn’t print the entire filter.
>> Duplicate detection is working based on sequential ipid.
>> Need to export the duplicate stats in a new TCP DSR extension.
>> I’ll add that before I release 3.0.8.2.
>> Currently looking at gaps.
>> So, to refresh my memory, we’re reporting gas when there is TCP
>> port reuse.  Old TCP is done, new TCP uses same ports and we
>> don’t adjust the gap reporting ???
>> Carter
>>> On Jun 17, 2015, at 10:17 AM, elof2 at sentor.se wrote:
>>> Hi Carter!
>>> Any comments? I guess not, just a simple fix in the code.
>>> /Elof
>>> On Tue, 2 Jun 2015, elof2 at sentor.se wrote:
>>>> Hi Carter!
>>>> I don't understand how _not_ counting someting yields 1 too much. :-)
>>>> Anyhow, I think that all MAR records that exist in the file should be counted and TotalMarRecords show the correct sum.
>>>> In my example, TotalMarRecords should be 16, not 17.
>>>> /Elof
>>>> On Tue, 2 Jun 2015, Carter Bullard wrote:
>>>>> Yes, in fact you included my email that stated that we don’t count the first MAR.
>>>>> Do you think we should ???  Seems redundant since the first MAR must be there ??
>>>>> Carter
>>>>>> On May 27, 2015, at 5:40 AM, elof2 at sentor.se wrote:
>>>>>> Hi Carter!
>>>>>> I investigated this a little bit further.
>>>>>> I have this logfile with 4 MAR records:
>>>>>> ra -Zb -M man -nr elof.log | grep -i man
>>>>>> 10:28:01.149822              man                  0      0 0      0 0 0            0            0           STA
>>>>>> 10:28:44.149779              man                  0      0 25749      1 69980     2839     28277263            0           CON
>>>>>> 10:29:44.149756              man                  0      0 25191      1 78426     3104     36560040            0           CON
>>>>>> 10:30:44.150890              man                  0      0 25060      1 107220     2715     37037618            0           CON
>>>>>> I concatenate it four times:
>>>>>> cat elof.log elof.log elof.log elof.log >> elof2.log
>>>>>> ra -Zb -M man -nr elof2.log | grep -i man
>>>>>> Now I have 16 MAR records.
>>>>>> So far everything is sane and logical.
>>>>>> The file has 38852 flows (I checked with wc -l).
>>>>>> The file has 16 MAR records.
>>>>>> Total records should therefore be 38868.
>>>>>> I now add -A :
>>>>>> ra -Zb -M man -A -nr elof2.log | tail -1
>>>>>> Totalrecords 38868     TotalMarRecords 17        TotalFarRecords 38852 TotalPkts 1211504  TotalBytes 462893084
>>>>>> So, the problem is that TotalMarRecords show 1 too much.
>>>>>> It should be 16.
>>>>>> /Elof
>>>>>> On Tue, 26 May 2015, Carter Bullard wrote:
>>>>>>> Hey /Elof,
>>>>>>> We are not counting the first MAR record.  If you were to filter the call using "not man" or "far", it should be correct.  All streams have to have the first MAR record, so didn't think that we should count it ??
>>>>>>> Carter
>>>>>>>> On May 26, 2015, at 10:50 AM, elof2 at sentor.se wrote:
>>>>>>>> Hi Carter!
>>>>>>>> Just found a silly error.
>>>>>>>> When adding option
>>>>>>>>    -A  Print aggregate statistics for the input stream on termination.
>>>>>>>> I get this line:
>>>>>>>> Totalrecords 26282     TotalMarRecords 12        TotalFarRecords 26271 TotalPkts 1069033  TotalBytes 654980030
>>>>>>>> The silly error is that 26271+12=26283, not 26282.
>>>>>>>> Very minor, but still wanted you to know. :)
>>>>>>>> /Elof

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150630/7ebf5fde/attachment.bin>