irtime (was: pause (or ssilence and dsilence))

[Carter Bullard]

Hey /Elof,
I really dislike very long emails ....
OK, so we cannot have this as a "ra only" feature, as this type of metric is really something you want to play with.  How soon after the arp response did the DNS request start ... Or time between smallest and biggest flow sent ...  Weird questions but valuable forensics methods.
So it will go in the overall library so all the ra* programs will be able to work with the value !!
Especially ratop, as that can filter and sort on all fields and so you want it there.

Irtime is (flow2StartTime - flow1EndTime) so it is the easiest of the metrics, as all flows have a start and end time, but may not have src or dst time stamps, if they don't have src or dst traffic.

We use these differences already in flow aggregation, as the inter record gap is an idle time measurement, and it is tallied in the intpktidle metric fields, but we don't provide access to the actual values, just the mean, stdev, max and min.

I'll think about it a bit and see what we can do in the short time.
Carter



> On Jun 17, 2015, at 6:15 AM, elof2 at sentor.se wrote:
> 
> 
> Hi Carter!
> Answers inline
> 
> 
>> On Tue, 16 Jun 2015, Carter Bullard wrote:
>> 
>> Hey /Elof,
>> Yes, new metrics are a good thing. The trick is what to call it, and how to generate it based on our existing methods and tools.
>> We have a set of ‘delta’ metrics that are already designed into the clients, such as DeltaStartTime, DeltaDuration, DeltaSrcPkts, etc... that were designed for programs like racluster.1 and ratop.1.  ratop.1 manages a list of argus records, sorts them based on whatever criteria you want, which sets up the notion of “the previous” record for calculating the type of metrics you’re interested in.  We have a routine called ArgusSubtractRecord() and ArgusIntersectRecord(), which provide the diffs / differences between this record and any other record, but we’ll have to add a little to formulate this inter-record time information.
>> So, …, there are 4 timestamps in an argus record.  SrcStartTime, SrcEndTime, DstStartTime and DstEndTime, from these 4 timestamps we formulate the SrcTime (minimum of all timestamps) and EndTime, (maximum of all timestamps).   So, …, do you want src inter-record time, dst inter-record time and just inter-record time ???
>> 
>>  irtime  - inter-record idle time, the idle time between the end and start of two records
>>  sirtime - src inter-record idle time
>>  dirtime - dst inter-record idle time
> 
> Perfect! I was trying to come up with good names but "irtime" was much better than my examples.
> 
> Oh, good, so we have all four timestamps in the record. Then my very last section could actually be implemented. That one gives the most details and are therefore the best. :-)
> 
> Your three irtime fields look good. The only thing that might confuse people, and therefore need better explaining in the manual would be the general 'irtime'. Compare with sbytes, dbytes and bytes. Here 'bytes' is the sum of the two. 'irtime' is _not_ the sum but rather a special field that depend on whatever the previous record is.
> 
> Something like this:
> 
>   irtime  - inter-record idle time, the idle time between the end and
>             start of two printed records (note that this depends on the
>             previous record, whatever that might be (including MAR
>             records))
>   sirtime - src inter-record idle time
>   dirtime - dst inter-record idle time
> 
> 
> The reason why I use the word "printed" is to describe that the counter is based on what you see on screen, after any filters have been applied. You *could add* a fourth field...
> 
>   uirtime  - unfiltered inter-record idle time, the idle time between the
>              end of the current record and the start of the very next
>              record in the logfile (excluding MAR records)
> 
> ...but I believe this is overkill. Only include it if you see the benefit and if it is easy to code. :-)
> 
> 
>> When we print the value for the first record, should it be 0.0, or is it blank ???
> 
> I think it should say 0.0.
> 
>> You will have to generate the list of records, in the proper order for these values to be meaningful, and you will have to print out the list in order to generate the values.
> 
> Yepp, that's why I'm only discussing 'ra', which will print things in the correct order.
> 
>> We’ll need to be able to print, filter, graph and process these values, as we do with every printable field.  But what do you want to do about sorting on the values, as that is a bit of a puzzle.  If we sort on a value that is based on sorting, what does the value mean ???
>> Do you want to label the flow with the value, and then be able to process that value, like sort, print, filter etc ??????
> 
> Heh, yes thats a catch 22 puzzle. :-)
> 
> The simple solution is to let the 'irtime' field be unique to 'ra', since it need the raw data to be sequentially ordered in the first place.
> 
> The advanced solution would probably become over-complex. Something like this:
> We remove 'irtime' completely, since it depends on the printed output. Instead we only have 'uirtime', 'sirtime' and 'dirtime' who are all based on the sequential records.
> Then all ra* tools could do their magic on these fields as well.
> 
> Nah, I say skip the possibility to sort/cluster/aggregate/label inter-record idle times.
> 
>> If we can get some agreement, then we should add it, sooner than later.
> 
> :-)
> 
> /Elof
> 
> 
> 
> 
> 
> 
> 
>>> On Jun 16, 2015, at 11:36 AM, elof2 at sentor.se wrote:
>>> 
>>> 
>>> Hi Carter!
>>> 
>>> Could the following feature be added to ra:
>>> 
>>> Say you have a tcp flow (e.g. smtp) that send some packets (EHLO, banner, MAIL, RCPT, DATA, etc). Then it pause for 45 seconds. Then it continues with a new MAIL, RCPT, etc. It keeps doing this with various pauses between the mails until hours later it is FINished.
>>> 
>>> In this traffic between A and B, I would like to see how long those pauses are. Something like this:
>>>         sport      dport state   pause
>>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0
>>> 12:00:45 A:32123 -> B:25  PA_PA    44.35
>>> 12:05:00 A:32123 -> B:25   A_A     254.1
>>> 
>>> ...where pause (or whatever you want to call it) is simply:
>>> stime minus ltime from the row before.
>>> 
>>> 
>>> That's it really.
>>> A "dumb" field that simply display the pause between records (rows) displayed on screen so I don't have to use a calculator and manually subtract and calculate the difference.
>>> 
>>> 
>>> 
>>> 
>>> If you want to do it a bit more complex, you could have two fields, in the same manner as wireshark has capture filters and display filters:
>>> ra could distinguish between 'displayed' pauses and pauses within the 'flow'.
>>> 
>>> rpause - pause between printed records on screen
>>> rpause = stime minus ltime from the row before.
>>> 
>>> fpause - pause since the last record for this particular flow
>>> fpause = stime minus ltime from the previous record within this flow.
>>> 
>>> 
>>> Example:
>>> When line 4 is to be printed, we have a NTP-packet in line 3 which would set the pause to 134.1 seconds.
>>> 
>>>         sport      dport state   pause
>>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0
>>> 12:00:45 A:32123 -> B:25  PA_PA    44.35
>>> 12:03:00 Q:123   -> Z:123  CON     134.1
>>> 12:05:00 A:32123 -> B:25   A_A     119.95
>>> 
>>> Instead, we cold have:
>>>         sport      dport state   rpause  fpause
>>> 12:00:00 A:32123 -> B:25 SPA_SPA   0.0     0.00
>>> 12:00:45 A:32123 -> B:25  PA_PA    44.35   44.35
>>> 12:03:00 Q:123   -> Z:123  CON     134.1   0.00
>>> 12:05:00 A:32123 -> B:25   A_A     119.95  254.1
>>> 
>>> 
>>> (perhaps 'rpause' should be called 'dpause' as in "displayed pause", but 'd' usually indicate "dst", so I go for 'r' as in Record or Row)
>>> 
>>> 
>>> 
>>> What do you say?
>>> 
>>> 
>>> 
>>> (
>>> If the flow records store stime and ltime for *each direction*, we could measure the pauses in the client data stream and pauses in the server responses.
>>> ...but I don't think the records store a src-stime, src-ltime, dst-stime and dst-ltime.
>>> )
>>> 
>>> /Elof
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150617/c3450263/attachment.html>