rasqlinsert data sometimes show negatives values withthe flow duration field

[carter at qosient.com]

Do you have all the timestamps turned on in argus?
Carter 

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Javier Almillategui <jalmilla at gmu.edu>
Sender: argus-info-bounces+carter=qosient.com at lists.andrew.cmu.edu
Date: Wed, 13 Oct 2010 18:54:48 
To: John Gerth<gerth at graphics.stanford.edu>
Cc: Argus<argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] rasqlinsert data sometimes show negatives values with
 the flow duration field

Hi John,

I do have the argus generated file, but it's a 100+ GB file. In any case there are only 54 Iistances of negative numbers in 600+ million flows. I'll see if I can track Down the specific instances in the argus file.

Best,  

Javier Almillategui
Center for Secure Information Systems
George Mason University
Mobile: (703)309-2060
Email: jalmilla at gmu.edu

On Oct 13, 2010, at 18:38, John Gerth <gerth at graphics.stanford.edu> wrote:

> Since every dur is negative, it sure looks to me like ltime and stime have just been reversed.
> 
> That's certainly plausible for RaTable1.csv:
> "dur","m.ltime-m.stime","ltime","stime"
> -34.945637,-34.945638,1274503174.403156,1274503209.348794
> -54.968819,-54.968817,1274503174.403156,1274503229.371973
> -74.971649,-74.971647,1274503174.403156,1274503249.374803
> -94.977325,-94.977324,1274503174.403156,1274503269.380480
> -135.011322,-135.011322,1274503174.403156,1274503309.414478
> -50.424545,-50.424546,1274574493.462583,1274574543.887129
> -306.942017,-306.942014,1274574493.462583,1274574800.404597
> -608.694763,-608.694792,1274585461.456646,1274586070.151438
> -8.358950,-8.358950,1274586143.223903,1274586151.582853
> 
> Do you have outputs from "ra" itself rather than the database?
> 
> /J
> 
> 
> On 10/13/2010 2:00 PM, Javier Almillategui wrote:
>> Hi Carter,
>> 
>> here is the description of one of my tables:
>> 
>> +-------+-----------------------+------+-----+---------+-------+
>> | Field | Type                  | Null | Key | Default | Extra |
>> +-------+-----------------------+------+-----+---------+-------+
>> | stime | double(18,6) unsigned | NO   | PRI | NULL    |       |
>> | srcid | varchar(64)           | YES  |     | NULL    |       |
>> | flgs  | varchar(32)           | YES  |     | NULL    |       |
>> | seq   | int(10) unsigned      | NO   | PRI | 0       |  ! ;   e | double(18,6) unsigned | NO   |     | NULL    |       |
>> | dur   | double(18,6)          | NO   |     | NULL    |       |
>> | proto | varchar(16)           | NO   |     | NULL    |       |
>> | saddr | varchar(64)           | NO   |     | NULL    |       |
>> | sport | varchar(10)           | NO   |     | NULL    |       |
>> | dir   | varchar(3)            | YES  |     | NULL    |       |
>> | daddr | varchar(64)           | NO   |     | NULL    |       |
>> | dport | varchar(10) &nb! sp;   NO   |     | NULL    |       |
>> | pkts  | bigint(20)            | YES  |     | NULL    |       |
>> | bytes | bigint(20)            | YES  |     | NULL    |       |
>> | state | varchar(32)           | YES  |     | NULL    |       |
>> | loss  | int(11)               | YES  |     | NULL    |       |
>> +-------+-----------------------+------+-----+---------+-------+
>> 16 rows in set (0.05 sec)
>> 
>> I'm running version mysql server version: 5.1.41-3ubuntu12.6 (Ubuntu)
>> 
>> and here is the query to verify the output of the correctness of the data:
>> op: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Monaco; ">select m.dur, m.ltime-m.stime,
>> m.ltime, m.stime from anondb.<RaTableX> m where dur < 0
>> 
>> I have attached the results as csv files
>> 
>> 
>> 
>> 
>> 
>> 
>> It looks to me that there is bad values in my argus data.
>> 
>> best,
>> 
>> Javier
>> 
>> On Oct 13, 2010, at 4:42 PM, Carter Bullard wrote:
>> 
>>> Hey Javier,
>>> Check the description of the schema that is created.  Here is an example
>>> of one of my tables;
>>> 
>>> % mysql
>>> mysql> desc inode;
>>> +--------+-----------------------+------+-----+---------+-------+
>>> | Field  | Type                  | Null | Key | Default | Extra |
>>> +--------+-----------------------+------+-----+---------+-------+
>>> | ltime  | double(18,6) unsigned | NO   |     | NULL    |       | 
>>> | dur    | double(18,6)          | NO   |     | NULL    |       | 
>>> | srcid  | varchar(64)           | NO   | PRI |         |       | 
>>> | mean   | double                | YES  |     | NULL    |       | 
>>> | inode  | varchar(64)           | NO   | PRI |         |       | 
>>> | sttl   | tinyint(3) unsigned   | NO   | PRI | 0       |       | 
>>> | pkts   | bigint(20)            | YES  |     | NULL    |       | 
>>> | record | blob                  | YES  |     | NULL    |       | 
>>> +--------+-----------------------+------+-----+---------+-------+
>>> 8 rows in set (0.04 sec)
>>> 
>>> The dur is a signed double, which is pretty big.  Can't imagine that your dur is rolling over.
>>> The dur is a calculated value (ltime - stime).  What is the stime and ltime when 
>>> you see the dur go negative?
>>> 
>>> Carter
>>> 
>>> On Oct 13, 2010, at 4:19 PM, Javier Almillategui wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> is there a possibility that argus will select the a limited value for the duration field? I'm seeing that the duration data sometimes is a negative
>>>> value for long lasting flows.
>>>> 
>>>> I have inserted the data with the following syntax:
>>>> 
>>>> rasqlinsert -w mysql://argus:123456@localhost/anondb/masontap_test_%Y_%m_%d -r mason_tap_20100521_prepared.argus -s +2seq -s +3ltime -s +4dur -s +loss -s +1srcid -s -record -m none -M time 1d
>>> 
>>> 
>>> 
>> 
>