Duration sum bug
Carter Bullard
carter at qosient.com
Mon Mar 21 15:19:50 EDT 2011
I suspect that the records are not in order, and things are actually OK.
I believe that the 20:57:18.761336 and 18:57:33.788103 are on different days,
so your list looks like its ~20 hours long, when sorted as you did, but if the
20:57 and 18:57 are on different days, it comes out to be ~22 hours.
I'm thinking everything is looking pretty good, actually. The mean values
look good based on this data.
Its clear you don't want to share the data, understandable.
One thing you could do is run ranonymize() against the ra.out file, then
you could possibly share the actual times, etc... with us, so we can see what is up.
ranonymize() will modify the IP addresses, and the times so that there isn't any hint
of what is up with the records. The default anonymization strategy is to
add a random constant offset to all the timestamps, so things like duration,
inter transaction arrival, etc... are preserved.
So try this:
ranonymize -r ra.out -w ra.anon
and then check out the data to make sure that it doesn't give up anything.
If your cool with that, then printout the full dates or use the '-u' option, so
we can see the real times.
Hope this is helpful,
Carter
On Mar 21, 2011, at 2:47 PM, Digital Ninja wrote:
> Apologies!
>
> First run against ra.out (minus the -u flag)
>
> 03:57:23.529664 03:57:23.544301 0.014637 0.014637 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 09:57:27.624699 09:57:27.639307 0.014608 0.014608 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 12:57:29.660667 12:57:29.676479 0.015812 0.015812 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 13:57:30.339190 13:57:30.354246 0.015056 0.015056 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 14:57:31.030849 14:57:31.046388 0.015539 0.015539 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 16:57:32.385680 16:57:32.400769 0.015089 0.015089 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 18:57:33.772816 18:57:33.788103 0.015287 0.015287 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 20:57:18.761336 20:57:18.776750 0.015414 0.015414 1
> 5.6.7.5 <-> 1.2.3.4 1 1
> 20:57:18.793793 20:57:18.808984 0.015191 0.015191 1
> 5.6.7.6 <-> 1.2.3.4 1 1
> 23:57:20.806478 23:57:20.822145 0.015667 0.015667 1
> 5.6.7.5 <-> 1.2.3.4 1 1
>
> Second run (-m srcid) against ra.out (minus the -u flag)
>
> 20:57:18.761336 18:57:33.788103 79215.0234 0.015230 10
> 0.0.0.0 <-> 0.0.0.0 10 10
>
>
>
> On Mon, Mar 21, 2011 at 2:34 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hmmm, well I can't help you if you don't show us what the results are.
>> Carter
>>
>>
>> On Mar 21, 2011, at 1:50 PM, Digital Ninja wrote:
>>
>>> Reran as requested. These do not generate consistent results.
>>> Unfortunately, they generate the same results I previously posted.
>>>
>>> On Mon, Mar 21, 2011 at 1:42 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> So, the "-m srcid" removes all the flow identifiers, so there won't be any IP addresses, or ports or protos.
>>>> Well, these results are not consistent. Did you apply the same filter, that you did in the first run?
>>>>
>>>> Try this:
>>>> ra -r files -w /tmp/ra.out - host 1.2.3.4
>>>>
>>>> racluster -u -r /tmp/ra.out -s stime ltime dur mean trans saddr dir daddr spkt dpkts
>>>> racluster -u -m srcid -r /tmp/ra.out -s stime ltime dur mean trans saddr dir daddr spkt dpkts
>>>>
>>>> Do these generate consistent results?
>>>> Carter
>>>>
>>>>
>>>> On Mar 21, 2011, at 1:33 PM, Digital Ninja wrote:
>>>>
>>>>> With -m srcid I'm back to the 72k seconds and no IPs listed:
>>>>> 20:57:18.761336 18:57:33.788103 79215.0234 10 0.015230 0.0.0.0 <-> 0.0.0.0 10 10
>>>>>
>>>>> On Mon, Mar 21, 2011 at 1:29 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> And because you are interested in comparing the times, you may want to
>>>>>> use the "-u" option, so that the time is printed in seconds. Makes it easier
>>>>>> for comparisons.
>>>>>> Carter
>>>>>>
>>>>>>
>>>>>> On Mar 21, 2011, at 1:26 PM, Digital Ninja wrote:
>>>>>>
>>>>>>> racluster -r <files> -s stime ltime dur trans mean saddr dir daddr
>>>>>>> spkts dpkts - host 1.2.3.4 produces:
>>>>>>>
>>>>>>> 03:57:23.529664 03:57:23.544301 0.014637 1 0.014637
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 09:57:27.624699 09:57:27.639307 0.014608 1 0.014608
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 12:57:29.660667 12:57:29.676479 0.015812 1 0.015812
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 13:57:30.339190 13:57:30.354246 0.015056 1 0.015056
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 14:57:31.030849 14:57:31.046388 0.015539 1 0.015539
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 16:57:32.385680 16:57:32.400769 0.015089 1 0.015089
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 18:57:33.772816 18:57:33.788103 0.015287 1 0.015287
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 20:57:18.761336 20:57:18.776750 0.015414 1 0.015414
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>> 20:57:18.793793 20:57:18.808984 0.015191 1 0.015191
>>>>>>> 5.6.7.6 <-> 1.2.3.4 1 1
>>>>>>> 23:57:20.806478 23:57:20.822145 0.015667 1 0.015667
>>>>>>> 5.6.7.5 <-> 1.2.3.4 1 1
>>>>>>>
>>>>>>> On Mon, Mar 21, 2011 at 1:19 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>> Hmmm, it doesn't look like you have any bugs getting in your way,
>>>>>>>> so you should be able to do what you want. So what does this generate?
>>>>>>>>
>>>>>>>> racluster -r files -s stime ltime dur trans mean saddr dir daddr spkts dpkts host 1.2.3.4
>>>>>>>>
>>>>>>>> Do these numbers look reasonable?
>>>>>>>> Carter
>>>>>>>>
>>>>>>>> On Mar 21, 2011, at 12:56 PM, Digital Ninja wrote:
>>>>>>>>
>>>>>>>>> Ok, so going back to what Rafael said about the 5-tuple aggregation...
>>>>>>>>> When I run ra against the files with the following flags:
>>>>>>>>>
>>>>>>>>> ra -nn -c "," -r <file> <file> <file> ... -L0 -s stime proto saddr dir
>>>>>>>>> daddr dur sport dport - host 1.2.3.4
>>>>>>>>>
>>>>>>>>> I get the following:
>>>>>>>>>
>>>>>>>>> 03:57:23.529664,17,5.6.7.5,<->,1.2.3.4,0.014637,30416,53
>>>>>>>>> 09:57:27.624699,17,5.6.7.5,<->,1.2.3.4,0.014608,29294,53
>>>>>>>>> 12:57:29.660667,17,5.6.7.5,<->,1.2.3.4,0.015812,49771,53
>>>>>>>>> 13:57:30.339190,17,5.6.7.5,<->,1.2.3.4,0.015056,6923,53
>>>>>>>>> 14:57:31.030846,17,5.6.7.5,<->,1.2.3.4,0.015539,31211,53
>>>>>>>>> 16:57:32.385680,17,5.6.7.5,<->,1.2.3.4,0.015089,14851,53
>>>>>>>>> 18:57:33.772816,17,5.6.7.5,<->,1.2.3.4,0.015287,1052,53
>>>>>>>>> 20:57:18:761336,17,5.6.7.5,<->,1.2.3.4,0.015414,6004,53
>>>>>>>>> 20:57:18:793793,17,5.6.7.5,<->,1.2.3.4,0.015191,31141,53
>>>>>>>>> 23:57:20.806478,17,5.6.7.5,<->,1.2.3.4,0.015667,30562,53
>>>>>>>>>
>>>>>>>>> Eyeballing the total time from beginning to end looks to be ~20 hours,
>>>>>>>>> with each connection actually lasting < .02 seconds. The 72k seconds
>>>>>>>>> from the racluster works out to about 22 hours, which would make sense
>>>>>>>>> if there were overlaps in connection time, but there aren't. What am
>>>>>>>>> I missing here? Is there a way to get the aggregated results I
>>>>>>>>> expected (in the original email) from racluster without summing them
>>>>>>>>> external to argus?
>>>>>>>>>
>>>>>>>>> Can it then be assumed then, based on my racluster flags, racluster is
>>>>>>>>> aggregating all sessions for the 1.2.3.4 IP based on the 5-tuple of
>>>>>>>>> 1.2.3.4:53 -> 0.0.0.0:0?
>>>>>>>>>
>>>>>>>>> Thanks for all your help!
>>>>>>>>>
>>>>>>>>> On Mon, Mar 21, 2011 at 11:05 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>>>>>>>>> racluster(), by default, aggregates all records with the same 5-tuple in a
>>>>>>>>>> single one. The resulting record has the start time of the first record and
>>>>>>>>>> the end time of the last one.
>>>>>>>>>> In your example the duration should be the end time of the last record (the
>>>>>>>>>> one with 96 bytes) minus the start time of the first one (with 213 bytes).
>>>>>>>>>> However without the files you are using is hard to say for sure.
>>>>>>>>>> Best regards,
>>>>>>>>>> Rafael Barbosa
>>>>>>>>>> http://www.vf.utwente.nl/~barbosarr/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 21, 2011 at 3:34 PM, Digital Ninja <dn1nj4 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I ran across something with racluster v3.0.2 & v3.0.4 that I can't
>>>>>>>>>>> quite explain and need some help. I have 9 different argus files. I
>>>>>>>>>>> am running racluster with the following options:
>>>>>>>>>>>
>>>>>>>>>>> racluster -M rmon -nn -c "," -m saddr proto sport -r <file> -L0 -s
>>>>>>>>>>> saddr proto sport sbytes dur dbytes - not arp
>>>>>>>>>>>
>>>>>>>>>>> When I run this command on the 9 files separately, for a single IP I
>>>>>>>>>>> get something like this:
>>>>>>>>>>>
>>>>>>>>>>> 1.2.3.4,17,53,289,0.47648,213
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015667,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.014637,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.014608,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015812,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015056,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015539,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015089,117
>>>>>>>>>>> 1.2.3.4,17,53,133,0.015287,96
>>>>>>>>>>>
>>>>>>>>>>> Summing the bytes and duration columns up, I would expect the totals to
>>>>>>>>>>> be:
>>>>>>>>>>> 1.2.3.4,17,53,1376,0.169343,1128
>>>>>>>>>>>
>>>>>>>>>>> However, when I run racluster on all 9 files simultaneously (-r <file>
>>>>>>>>>>> <file> <file>...etc) I get the following results for the above data:
>>>>>>>>>>> 1.2.3.4,17,53,1376,79215.023438,1128
>>>>>>>>>>>
>>>>>>>>>>> What's going on with the duration field??
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110321/a0e5dbd7/attachment.bin>
More information about the argus
mailing list