Misc problems with argus-clients v. 3.0.5.34

[Carter Bullard]

Hey Markku,
I found the bug, where rabins would generate results inconsistent with racluster. The bug
originated from rabins applying flow correction logic when aggregating with a modified
flow key (incorrect), and racluster not doing the flow correction (the correct behavior).

I have fixed this in argus-clients-3.0.5.35, which I will upload very late tonight.
Thanks for the sample data, I wouldn't not have been able to debug without it.
I believe the new code fixes all the bugs that you reported.
Thanks for all the help !!!!!

Carter



On Mar 1, 2012, at 9:24 AM, Markku Parviainen wrote:

> Hi,
> 
> 2012/2/29 Carter Bullard <carter at qosient.com>:
>> I think the problems you're getting from rabins.1 may not be bugs, but
>> its definitely not the right results.   When processing time based bins,
>> rabins.1 will reject data if its not within its working time range.  If you
>> don't tell rabins.1 what time range to use, with the -t option, rabins.1 will
>> have to figure out what the time range is.  It does this when reading
>> from a file by reading the file twice.  Once to figure out the time ranges,
>> then another run to process the data into its preallocated bins.
>> 
>> When you pipe data into rabins.1, it has to guess what the time range
>> is going to be, and so there is potential to throw data away if its all over
>> the calendar, so to speak.    This doesn't solve your problem, but it just
>> explains why I think you're getting unpredicted results.
> 
> Actually on my original script where I noticed this problem, I already
> was using -t option. It doesn't affect to the results. But thanks for
> pointing out that rabins will read the data twice without it.
> 
> Related to this problem, rabins also seems to generate new records.
> Look at the port 0 (and 3074) below.
> 
> This is what the data looks like:
> # racluster -m proto dport -r test.arg.gz -w - - udp | rasort -m pkts
> -s proto dport pkts -n  | head -5
>   udp 161      353877
>   udp 137      133430
>   udp 3074     108009
>   udp 53        89020
>   udp 8612      69988
> 
> Rabins produces port 0 and changes the packet counter of port 3074:
> # rasort -m stime -r test.arg.gz -w - - udp | time rabins -M hard time
> 5s -m proto dport -w - | racluster -m proto dport -w - | rasort -m
> pkts -s proto dport pkts -n  | head -5
>   udp 161      353877
>   udp 137      133430
>   udp 3074     105026
>   udp 0        103673
>   udp 53        89020
> 
> But the individual search for 3074 produces the correct result:
> # rasort -m stime -r test.arg.gz -w - - udp and dst port 3074 | rabins
> -M hard time 5s -m proto dport -w - | racluster -m proto dport -w - |
> rasort -m pkts -s proto dport pkts -n  | head -5
>   udp 3074     108009
> 
> # racluster -m srcid -r test.arg.gz -s pkts - udp and dst port 3074
>  108009
> 
> There are no really many port 0 packets there:
> # rasort -m stime -r test.arg.gz -w - - udp and dst port 0 | rabins -M
> hard time 5s -m proto dport -w - | racluster -m proto dport -w - |
> rasort -m pkts -s proto dport pkts -n  | head -5
>   udp 0           107
> # racluster -m srcid -r test.arg.gz -s pkts - udp and dst port 0
>     107
> 
> # ratimerange -r test.arg.gz -u
> 1327489126.869327 - 1327492741.717686
> 
> I tried all commands also with -t 1327489126-1327492741 (with every
> rasort, rabins and racluster). It didn't change the results, nor did
> it make those commands any faster.
> 
> I also noticed this:
> 
> # ranonymize -r test.arg.gz -w anon.arg
> # racluster -m srcid -r anon.arg -s pkts -  udp and dst port 3074
>       1
> # racluster -m srcid -r anon.arg -s pkts - udp and dst port 0
>     107
> # racluster -m srcid -r anon.arg -s pkts
> 4046512
> # racluster -m srcid -r test.arg.gz -s pkts
> 4046512
> 
> There are packets in the anonymized file, but the filter does not find
> all of them?
> 
> I'll send the original packet file separately for your analysis.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120307/af6b0812/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120307/af6b0812/attachment.bin>