argus-3.0 availability

[Carter Bullard]

Gentle people,
     Well, I did get the initial set of clients finished last night,  
so, as
our good european friends say, we've got a complete kit now.   I
still have to review it one more time to ensure that its scrubbed
down good (there is some intellectual property that had to be
extracted), and then I'll make an announcement on this mailing
list.   Should be tomorrow.

    The set of programs are:
       argus-3.0.0.rc.1
          argus
       argus-client-3.0.0.rc.1
          ra, radium, ratop, racluster, ragrep, ragraph,
          rasort, rasplit, rastrip, racount, rabins.

    The big difference is a modification to the record format, to
handle IPv6, 64-bit counters, and 64-bit processing.   We
are parsing more encapsulation headers, such as MPLS
and GRE.   Argus supports multiple flow models, so you can
configure it to run as a 5-tuple bidirectional probe, or a
uni-diirectional probe, or as say just an MPLS flow monitor.
I think we also do VLANs, and MAC address flow monitoring,
which just means that the flow model stops at these layers
in the stack.  This is in case you're interested in monitoring
say 10 Gbps core links, and have no interest in micro-flow
monitoring.

    New features for all the ra* programs is more filtering, and
better field printing support.   ratop() should be considered
the client of choice to look at data, (maybe we should call it
raview? or something, top maybe over used?)

    The new programs are radium(),  racluster(), which obsoletes
ragator(), and rasplit.   radium() is our data collection system,
which will maintain connections with upto, what, 256 argus
probes and multiplex the data out to upto, what, 256 data readers.
Seems to work well.   rasplit(), like the unix split command,
will take an argus stream and split it based on file size, record
number, time, and it can write data into file names that have
record values as a part of the path, and/or strftime strings.  As
an example:

    rasplit -M time 1h -r file -w data/$srcid/%Y/%m/%d/argus.%H

This will split data into 1 hour chunks and write the data into
a semi-standard argus archive format, based on probe id ($srcid),
with the sub directories based on year, month and day.
This is the solution for say, time sorting large data files.

racluster() is the most important modification, as
it now can support specific aggregation models and/or allow
you to aggregate on any field in the argus record.   As an
example, if you were interested in reporting matrix data:

      racluster -M matrix -r file

Matrix data is structured so that the src is < the dst in numeric
value, so you can get deterministic reports regardless of input.

If you were interested in how much traffic a host transmitted and
received categorized by, say address and DiffServ codepoint for
any IP traffic:

     racluster -M rmon -m saddr proto stos - ip

If you wanted to ignore the source port in a standard argus
record:

    racluster -m saddr daddr proto dport

There are man pages, so hopefully there isn't an amazing
amount of work to be done.

Ok, I'll try to have this stuff ready tomorrow.  Thanks for the
patience.

Carter



>
> Quoting Carter Bullard <carter at qosient.com>:
>
>> Hey Russell and Robin,
>>    Yes, in fact, I gave CMU the first copy of argus-3.0 for a  
>> sanity check
>> on Friday.   I think it passed.  I still have some work to do on  
>> the clients,
>> so hopefully this week, I'll have the first round ready.
>>
>>    Russell is right, ragator() is the way to do it in argus-2.0.
>> Because some subnet's will have longer masks than others,
>> be sure and have a separate line to specify the mask length
>> for the subnets of interest.   If you need help, just holler.
>>
>>    In argus-3.0, you will do this using racluster().  It is designed
>> to do most of what you want on the command line rather than
>> having to have a complex configuration file strategy like
>> ragator uses.  For racluster() you would do:
>>
>>    racluster -M rmon -m proto saddr/16 - ip
>>
>> this will read all the ip data, modifying the records to track  
>> single IP
>> addresses  (the '-M rmon') instead of flows, and to aggregate the
>> flows based on proto and CIDR address (when you use the 'rmon'
>> option, the unique address is in the saddr field).   There are many
>> options for racluster().
>>
>> If you wanted different subnets to have different CIDR mask lengths,
>> you would do passes, one to aggregate on the IP address, and then
>> another pass to aggregate into the subnets of interest.   racluster()
>> supports a configuration file that allows you to have lots of  
>> directives.
>> To do this pipe two raclusters together:
>>
>>    racluster -M rmon -m saddr proto -w - - ip | racluster -f  
>> conf.file
>>
>> where conf.file contains lines like:
>>    filter="src net 10.23.0.0/16" model="proto saddr/16"
>>    filter="src net 10.12.0.0/16" model="proto saddr/23"
>>    filter=""                                     model="proto  
>> saddr/24"
>>
>>
>> Hopefully I'll have argus-clients-3.0.0.rc.1 out soon enough for
>> you to use (maybe this week, maybe next week).   As soon as I
>> do that, argus-3.0 will be the version that we'll answer questions
>> to on the list (unless there is a compelling reason).  It will be  
>> able
>> to process argus-2.x data, so it should work for you.
>>
>> Carter
>>
>>