rahisto and 'dur'

[James A. Robinson]

On Tue, Apr 9, 2013 at 8:11 AM, Carter Bullard <carter at qosient.com> wrote:
> Argus records are bi-directional flow " status " records, so no
> primitive record, unmodified right out of argus, should have a
> duration greater than the ARGUS_FLOW_STATUS_INTERVAL.  If you want
> the duration of transactions that live longer than the status
> interval, you need to merge the status records to create transaction
> records, using racluster().


Thank you.  So far every message I've seen from you is what I'd call
"information rich". :-) Thank you for the details, I *think* I
understand the gist of what you are telling me.

So if most send/recieve cycles of a protocol like http or memcache
have a dur under 5 seconds, then ARGUS_FLOW_STATUS_INTERVAL=5 will be
fine for capturing sltime, dltime, sstime, dstime, allowing rahisto to
compute dur?

If we have instances of protocols where the exchange might exceed 5
seconds then, instead of increasing ARGUS_FLOW_STATUS_INTERVAL, we
ought to be able to use racluster to fit together the spaced out data?
Should I then be able to feed that clustered data to rahisto?

What I've configured so far is a handful of hosts running argus, not
writing to any local disk but making data available on the private
network:

ARGUS_COLLECTOR=no
ARGUS_DAEMON=yes
ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
ARGUS_FLOW_STATUS_INTERVAL=5
ARGUS_FLOW_TYPE="Bidirectional"
ARGUS_GENERATE_APPBYTE_METRIC=yes
ARGUS_GENERATE_BIDIRECTIONAL_TIMESTAMPS=yes
ARGUS_GENERATE_MAC_DATA=yes
ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
ARGUS_GO_PROMISCUOUS=no
ARGUS_INTERFACE=eth0
ARGUS_MAR_STATUS_INTERVAL=60

A radium server is configured to pull from those hosts, and to write
the data to its local disk.  We've got a cron job rotating the radium
output data file every five minutes.

I was trying something like this on the radium server:

$ racluster -nn -R /var/log/radium/radium.out.?.gz \
  -m  saddr daddr port \
  -s stime dur saddr sport dir daddr dport pkts state \
  - port 80 and \( fin and not rst \)

and could see it producing numbers that indicate it is capturing the
longer running http transactions (I see dur values that cover the
typical range from sub second to multi second times, including some
over 5).

I then tried feeding data to rahisto:

$ racluster -R /var/log/radium/radium.out.?.gz \
  -m saddr daddr port \
  -w - \
 - port 80 and \( fin and not rst \) \
  | rahisto -r - -H dur 5

But got nothing back.  I assumed that I am misunderstanding exactly
what racluster is doing when it groups and then outputs an
argus-stream.  I found that if just run rahisto on the individual
files:

$ rahisto -r /var/log/radium/radium.out.?.gz -H dur 10

I do get output, but only for the 0-5 second range that I assume is
the default.

Jim

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
James A. Robinson                       jimr at highwire.stanford.edu
HighWire | Stanford University          http://highwire.stanford.edu/
+1 650 7237294 (Work)                   +1 650 7259335 (Fax)