rahisto and 'dur'

[Carter Bullard]

When you don't provide rahisto() with the range of the data for the
histogram, rahisto() has to take 2 passes on the data, the first to 
determine the range, and the second to do the bin aggregations.

If you pipe the data into rahisto(), it won't be able to read the data twice.
It will try, and find that the file descriptor has already closed, and it will
think there wasn't any data.

So, you shouldn't pipe data into rahisto() without providing a range for
the histogram.....

Use default settings for racluster().  Do you get anything with this?
 
   racluster -r /var/log/radium/radium.out.?.gz -w - - syn and synack | rahisto -H dur 25:0-100

If you want to know if there were " outlayers ", add a "-M outlayers" and
you may get an additional row or two of data for the counts outside the range.

I changed your filter, because RST is used by Microsoft stacks to close
normal connections.  I thought you were filtering to avoid rejected connections?

Carter


On Apr 10, 2013, at 2:03 PM, "James A. Robinson" <jimr at highwire.stanford.edu> wrote:

> On Tue, Apr 9, 2013 at 8:11 AM, Carter Bullard <carter at qosient.com> wrote:
>> Argus records are bi-directional flow " status " records, so no
>> primitive record, unmodified right out of argus, should have a
>> duration greater than the ARGUS_FLOW_STATUS_INTERVAL.  If you want
>> the duration of transactions that live longer than the status
>> interval, you need to merge the status records to create transaction
>> records, using racluster().
> 
> 
> Thank you.  So far every message I've seen from you is what I'd call
> "information rich". :-) Thank you for the details, I *think* I
> understand the gist of what you are telling me.
> 
> So if most send/recieve cycles of a protocol like http or memcache
> have a dur under 5 seconds, then ARGUS_FLOW_STATUS_INTERVAL=5 will be
> fine for capturing sltime, dltime, sstime, dstime, allowing rahisto to
> compute dur?
> 
> If we have instances of protocols where the exchange might exceed 5
> seconds then, instead of increasing ARGUS_FLOW_STATUS_INTERVAL, we
> ought to be able to use racluster to fit together the spaced out data?
> Should I then be able to feed that clustered data to rahisto?
> 
> What I've configured so far is a handful of hosts running argus, not
> writing to any local disk but making data available on the private
> network:
> 
> ARGUS_COLLECTOR=no
> ARGUS_DAEMON=yes
> ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
> ARGUS_FLOW_STATUS_INTERVAL=5
> ARGUS_FLOW_TYPE="Bidirectional"
> ARGUS_GENERATE_APPBYTE_METRIC=yes
> ARGUS_GENERATE_BIDIRECTIONAL_TIMESTAMPS=yes
> ARGUS_GENERATE_MAC_DATA=yes
> ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
> ARGUS_GO_PROMISCUOUS=no
> ARGUS_INTERFACE=eth0
> ARGUS_MAR_STATUS_INTERVAL=60
> 
> A radium server is configured to pull from those hosts, and to write
> the data to its local disk.  We've got a cron job rotating the radium
> output data file every five minutes.
> 
> I was trying something like this on the radium server:
> 
> $ racluster -nn -R /var/log/radium/radium.out.?.gz \
>  -m  saddr daddr port \
>  -s stime dur saddr sport dir daddr dport pkts state \
>  - port 80 and \( fin and not rst \)
> 
> and could see it producing numbers that indicate it is capturing the
> longer running http transactions (I see dur values that cover the
> typical range from sub second to multi second times, including some
> over 5).
> 
> I then tried feeding data to rahisto:
> 
> $ racluster -R /var/log/radium/radium.out.?.gz \
>  -m saddr daddr port \
>  -w - \
> - port 80 and \( fin and not rst \) \
>  | rahisto -r - -H dur 5
> 
> But got nothing back.  I assumed that I am misunderstanding exactly
> what racluster is doing when it groups and then outputs an
> argus-stream.  I found that if just run rahisto on the individual
> files:
> 
> $ rahisto -r /var/log/radium/radium.out.?.gz -H dur 10
> 
> I do get output, but only for the 0-5 second range that I assume is
> the default.
> 
> Jim
> 
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> James A. Robinson                       jimr at highwire.stanford.edu
> HighWire | Stanford University          http://highwire.stanford.edu/
> +1 650 7237294 (Work)                   +1 650 7259335 (Fax)
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130410/2f668023/attachment.bin>