rabins and racluster.conf

[Mark E. Mallett]

On Mon, Apr 23, 2012 at 03:40:52PM -0400, Carter Bullard wrote:
> Hey Mark,
> 
> Yes, the "-f " is for a racluster.conf file.  It should aggregate
> within the bin, so its as if you ran racluster on each bin as the data
> arrives.
> 
> Sorry about the man page, I'll take a look later today.  Any updates
> will go on the web server.  You're using 3.0.6 code?

yes, 3.0.6, on various systems.

And fantastic, that's what I hoped. Let me describe my simple tests
and perhaps somebody can tell me what dumb thing I am doing.

It looked like the racluster.conf aggregation rules could be pretty
useful so I thought I'd try it out with some simple reports where the
details aren't too important but the method is. I figured if the method
works I can go onwards from there. But I got stuck early on.

The simple test is to get some per-hour totals broken down into
potentially high-interest categories in different ways.  I chose http
and https and ssh and "everything else" as categories for the test.
Here's my file c3.conf

=======
# I've tried this on or off
RACLUSTER_PRESERVE_FIELDS=no
#
filter="tcp and dst port eq 80" model="proto dport"
filter="tcp and dst port eq 443" model="proto dport"
filter="tcp and dst port eq 22" model="proto dport"
# filter="tcp or udp" model="proto"
filter="" model="proto"
=========

I processed this with rabins doing time binning (see command below),
expecting this to give me one record of aggregated data for http, one
for https, one for ssh per time bin, and then other records in each bin
aggregated only by protocol.

When I run rabins I either write the argus records to a file and look at
that with some other ra* utility or just look at the output directly;
the result seems to be the same.  This is the sort of command I use to
time bin and look at the output:

  rabins -n -f c3.conf -M zero -M time 1h -r input-file \
      -s stime proto dport pkts bytes dur:12 | less

With that command, I do not seem to get any aggregation at all. I get a
lot of individual flow records matching the tcp statements (http, https,
ssh), and no records for any other protocol or ports.  This last part
surprises me most because I thought the c3.conf file will catch
everything, even if the aggregation doesn't happen.

If I add a "model" set on the command line, i.e. an '-m' switch like this:

  rabins -n -f c3.conf -M zero -M time 1h -m proto dport -r input-file \
      -s stime proto dport pkts bytes dur:12 | less

then I get the per-bin aggregation I expected of the http, https, and ssh
traffic, but I still get no other output (except what's generated by the
"-M zero" for otherwise empty bins).  Here I would still expect the last
line in c3.conf to catch some things and produce things aggregated
by "proto".

If I uncomment the next-to-last filter line, then I do see some other
records show up; but for the most part they also appear unaggregated
except by flow.

I could go on for a while but this is already long enough and probably
quite enough to introduce my confusion.  One of the issues seems to be
that the "model" section of each line isn't being applied, and the only
model that gets applied is the one in '-m' on the command line.  But I
don't think that's the only issue.  There's probably some dots that I
haven't connected..

Yours,
-mm-