irtime (was: pause (or ssilence and dsilence))

[elof2 at sentor.se]

Hi Carter!
Answers inline


On Tue, 16 Jun 2015, Carter Bullard wrote:

> Hey /Elof,
> Yes, new metrics are a good thing. The trick is what to call it, and how to generate it based on our existing methods and tools.
> We have a set of ‘delta’ metrics that are already designed into the clients, such as DeltaStartTime, DeltaDuration, DeltaSrcPkts, etc... that were designed for programs like racluster.1 and ratop.1.  ratop.1 manages a list of argus records, sorts them based on whatever criteria you want, which sets up the notion of “the previous” record for calculating the type of metrics you’re interested in.  We have a routine called ArgusSubtractRecord() and ArgusIntersectRecord(), which provide the diffs / differences between this record and any other record, but we’ll have to add a little to formulate this inter-record time information.
> So, …, there are 4 timestamps in an argus record.  SrcStartTime, SrcEndTime, DstStartTime and DstEndTime, from these 4 timestamps we formulate the SrcTime (minimum of all timestamps) and EndTime, (maximum of all timestamps).   So, …, do you want src inter-record time, dst inter-record time and just inter-record time ???
>
>   irtime  - inter-record idle time, the idle time between the end and start of two records
>   sirtime - src inter-record idle time
>   dirtime - dst inter-record idle time

Perfect! I was trying to come up with good names but "irtime" was much 
better than my examples.

Oh, good, so we have all four timestamps in the record. Then my very last 
section could actually be implemented. That one gives the most details 
and are therefore the best. :-)

Your three irtime fields look good. The only thing that might confuse 
people, and therefore need better explaining in the manual would be the 
general 'irtime'. Compare with sbytes, dbytes and bytes. Here 'bytes' is 
the sum of the two. 'irtime' is _not_ the sum but rather a special field 
that depend on whatever the previous record is.

Something like this:

    irtime  - inter-record idle time, the idle time between the end and
              start of two printed records (note that this depends on the
              previous record, whatever that might be (including MAR
              records))
    sirtime - src inter-record idle time
    dirtime - dst inter-record idle time


The reason why I use the word "printed" is to describe that the counter is 
based on what you see on screen, after any filters have been applied. You 
*could add* a fourth field...

    uirtime  - unfiltered inter-record idle time, the idle time between the
               end of the current record and the start of the very next
               record in the logfile (excluding MAR records)

...but I believe this is overkill. Only include it if you see the benefit 
and if it is easy to code. :-)


> When we print the value for the first record, should it be 0.0, or is it blank ???

I think it should say 0.0.

> You will have to generate the list of records, in the proper order for these values to be meaningful, and you will have to print out the list in order to generate the values.

Yepp, that's why I'm only discussing 'ra', which will print things in 
the correct order.

> We’ll need to be able to print, filter, graph and process these values, as we do with every printable field.  But what do you want to do about sorting on the values, as that is a bit of a puzzle.  If we sort on a value that is based on sorting, what does the value mean ???
> Do you want to label the flow with the value, and then be able to process that value, like sort, print, filter etc ??????

Heh, yes thats a catch 22 puzzle. :-)

The simple solution is to let the 'irtime' field be unique to 'ra', since 
it need the raw data to be sequentially ordered in the first place.

The advanced solution would probably become over-complex. Something like 
this:
We remove 'irtime' completely, since it depends on the printed output. 
Instead we only have 'uirtime', 'sirtime' and 'dirtime' who are all based 
on the sequential records.
Then all ra* tools could do their magic on these fields as well.

Nah, I say skip the possibility to sort/cluster/aggregate/label 
inter-record idle times.

> If we can get some agreement, then we should add it, sooner than later.

:-)

/Elof







>> On Jun 16, 2015, at 11:36 AM, elof2 at sentor.se wrote:
>>
>>
>> Hi Carter!
>>
>> Could the following feature be added to ra:
>>
>> Say you have a tcp flow (e.g. smtp) that send some packets (EHLO, banner, MAIL, RCPT, DATA, etc). Then it pause for 45 seconds. Then it continues with a new MAIL, RCPT, etc. It keeps doing this with various pauses between the mails until hours later it is FINished.
>>
>> In this traffic between A and B, I would like to see how long those pauses are. Something like this:
>>          sport      dport state   pause
>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0
>> 12:00:45 A:32123 -> B:25  PA_PA    44.35
>> 12:05:00 A:32123 -> B:25   A_A     254.1
>>
>> ...where pause (or whatever you want to call it) is simply:
>> stime minus ltime from the row before.
>>
>>
>> That's it really.
>> A "dumb" field that simply display the pause between records (rows) displayed on screen so I don't have to use a calculator and manually subtract and calculate the difference.
>>
>>
>>
>>
>> If you want to do it a bit more complex, you could have two fields, in the same manner as wireshark has capture filters and display filters:
>> ra could distinguish between 'displayed' pauses and pauses within the 'flow'.
>>
>> rpause - pause between printed records on screen
>>  rpause = stime minus ltime from the row before.
>>
>> fpause - pause since the last record for this particular flow
>>  fpause = stime minus ltime from the previous record within this flow.
>>
>>
>> Example:
>> When line 4 is to be printed, we have a NTP-packet in line 3 which would set the pause to 134.1 seconds.
>>
>>          sport      dport state   pause
>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0
>> 12:00:45 A:32123 -> B:25  PA_PA    44.35
>> 12:03:00 Q:123   -> Z:123  CON     134.1
>> 12:05:00 A:32123 -> B:25   A_A     119.95
>>
>> Instead, we cold have:
>>          sport      dport state   rpause  fpause
>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0     0.00
>> 12:00:45 A:32123 -> B:25  PA_PA    44.35   44.35
>> 12:03:00 Q:123   -> Z:123  CON     134.1   0.00
>> 12:05:00 A:32123 -> B:25   A_A     119.95  254.1
>>
>>
>> (perhaps 'rpause' should be called 'dpause' as in "displayed pause", but 'd' usually indicate "dst", so I go for 'r' as in Record or Row)
>>
>>
>>
>> What do you say?
>>
>>
>>
>> (
>> If the flow records store stime and ltime for *each direction*, we could measure the pauses in the client data stream and pauses in the server responses.
>> ...but I don't think the records store a src-stime, src-ltime, dst-stime and dst-ltime.
>> )
>>
>> /Elof
>>
>
>