rabins does not process all records

[carter at qosient.com]

Hey Maketsi,
Several possibilities.
If the records are not time sorted, rabins() could be throwing records away.  You can test this by:
   rasort -r file -w - | rabins .......

The issue is that rabins() has to figure out what the startime and range is, in one pass of the data.

However, if you specify the time range with a "-t" option, you by-pass this logic.  Try a time filter that spans the records to see if it gets better.

In all cases, if you run with some debug information, using the "-D" option, rabins() may tell us the problem.

Please send any results, so we can address this problem.

Carter

Carter
Carter
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: maketsi <maketsi at gmail.com>
Sender: argus-info-bounces+carter=qosient.com at lists.andrew.cmu.edu
Date: Mon, 4 Oct 2010 10:23:16 
To: <argus-info at lists.andrew.cmu.edu>
Subject: [ARGUS] rabins does not process all records

Hi,

Rabins 3.0.3.17 does not seem to work the way it should. I don't know
if the problem is related to client (rabins) or server (argus 3.0.2)
problems. I have tried all 3.x versions of rabins and none of them
work.

# ratimerange -r t5.2010.09.28.10.50.00
2010-09-28T10:50:00 - 2010-09-28T10:55:00

The following is correct (i.e. matches with actual data seen with ra):

# rabins -M hard time 1m -r t5.2010.09.28.10.50.00  -m srcid -s stime
trans - host x.x.x.x
2010-09-28T10:50:00      2
2010-09-28T10:51:00      2
2010-09-28T10:52:00      1
2010-09-28T10:53:00      3

These are not correct:

# rabins -M hard time 1m -r t5.2010.09.28.10.50.00 -m srcid -s stime trans
2010-09-28T10:54:00  30237
2010-09-28T10:55:00   1850

# rabins -M hard time 1m -r t5.2010.09.28.10.50.00 -m srcid -s stime trans - tcp
2010-09-28T10:52:00  30131
2010-09-28T10:53:00  29577
2010-09-28T10:54:00  29542
2010-09-28T10:55:00   1746

See how the result changes with flow filter, but to the wrong
direction (i.e. more filtered = more data)?
The most bizarre result is this:

# rabins -M nomodify hard time 1m -r t5.2010.09.28.10.50.00 -m srcid
-s stime trans
2010-09-28T10:54:00  22070


This is what the data really looks like, without rabins:

# rasplit -r t5.2010.09.28.10.50.00 -M hard time 1m -w t1.
# for i in t1.*; do racluster -r $i -m srcid -s stime trans; done
2010-09-28T10:50:00  36992
2010-09-28T10:51:00  38443
2010-09-28T10:52:00  30744
2010-09-28T10:53:00  30202
2010-09-28T10:54:00  30237
2010-09-28T10:55:00   1850

Can you confirm the issue or do you need some debug data to play with?