Reconstituting flows

Wed Mar 27 17:48:43 EST 2002

Hey Roy,
   Since 2.0, Argus has been using a TLV (type, length, value)
record strategy that accomplishes much of what you describe in
your mail.  The record header has the total length and the
type (management vs. data) and for the data records, all the
data is contained in a number of DSR's, Data Specific Records.
We've got one for the flow descriptors, the metrics, tcp
state information, user data capture buffers, and when records
are aggregated, a separate metric record that contains stats on
the various aggregated values.

   You can use ragator to build the summaries that you're
interested in creating.  Using a set of ragator configuration
files, you can create the aggregate descriptions that you
describe, aggregated by protocol, by destination address, etc..

   The key to argus performance is keeping the per packet
processing to a minimum and exporting as much general purpose
data as possible to enable the expensive processing off of
the probe.  That's why we support capturing the first N
bytes of data on each transaction.  Also I'm not sure that
some of this deep packet inspection is going to be legal in the
next few years, so I'd like to avoid that hole if possible.

Hope this helps!

Carter

Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 18K
New York, New York  10022

carter at qosient.com
Phone +1 212 588-9133
Fax   +1 212 588-9134
http://qosient.com

> -----Original Message-----
> From: owner-argus-info at lists.andrew.cmu.edu 
> [mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of Roy Hooper
> Sent: Wednesday, March 27, 2002 1:24 PM
> To: argus-info at lists.andrew.cmu.edu
> Subject: Fw: Reconstituting flows
> 
> 
> (If this comes in duplicate, I apologise, I sent the orginal 
> from the wrong
> email address)
> 
> Chris (Carter),
> 
> You've probably already thought of this, but it occurs to me 
> that there might
> be possibility to "do the right thing" exists at the moment.  
> That right thing
> is to allow any version tool to read any version's data. I 
> haven't read the
> details on the argus datafile format, so these ideas may already be
> addressed...
> 
> Since the argus data format is intended to be processed in 
> streams, that is, it
> is not indexed, and the client tools are designed to use the 
> datafiles in a
> sequantial manner, records need not be the same length...  
> Given this, a record
> format that is tagged with the generating application and 
> version (2 bytes?)
> and also containing optional extended data blocks after the 
> fixed length
> portion containing something like this:
>     extended = extended_record ... NULL
>     extended_record = length generating_app version recordtype data
> 
> An extension of this idea to make processing slightly faster 
> that I have used
> in the past is a meta record type that specifies that the 
> following N bytes of
> data are all of the same type, and that type is X, so you can 
> read them all in
> one read, or seek past them.  Synchronizing this against 
> datafile rotation
> could be a pain in the ass...
> 
> This basic idea would allow you to keep extending the data 
> format and still
> have the data be useful by older ra* clients and 
> non-commercial versions ...
> 
> And another idea that's been kicking around in my head is 
> either some form of
> datafile indexer or better "racount" style 
> processor/summarizer.  I'd like to
> be able to keep in the order of 7 days of argus logs, but 
> also keep protocol
> stats indefinitely (well, okay, up to a specific limit of 
> time) in an RRD-like
> format but that doesn't require all of the space be allocated at
> initialization.  The stats i'd like to summarize out of the 
> argus data on an
> periodic (5 minutes?) hourly, daily, weekly, monthly, yearly 
> basis are:
>     usage by node based on MAC (not as important, but would be nice)
>     usage by IP
>     usage by IP per protocol
>     usage by IP per protocol + service
> 
> In order to generate such stats, I would not be interested in 
> conversations,
> but instead on data bytes in/out and would want to view the 
> data (and store it)
> based on a local network.  That is to say that for a 
> conversation between
> 192.168.0.1 and 10.0.0.1 where 10.0.0.0/24 is the local 
> network and 192.168.0.1
> is some remote site on the internet, I would be interested in 
> keeping the stats
> from the point of view of 10.0.0.1 and would never record 
> 192.168.0.1 anywhere
> in this summary data.
> 
> BTW, argus is way cool.  Have you thought about including 
> hooks in argus to be
> able to load modules to do further data processing on packets 
> to be able to
> generate further accounting data, extracting by URLs, FTP 
> filenames, POP/IMAP
> logins, POP/IMAP messages received, messages uploaded for 
> IMAP, SMTP messages
> in/out - sender/recipients, etc.  I think snort already has 
> most of the code
> you'd need to do some of this in its modules.
> 
> Anyway, enough questions for now, feel free to answer with 
> see FAQ or see code
> directory X or file Y, see CVS, etc.
> 
> ---
> Roy Hooper
> Project Manager & Senior UNIX Consultant
> Decisive Technologies, Inc.
> rhooper at decisivetechnologies.com
> 
> 
> ----- Original Message -----
> From: "newton" <newton at unb.ca>
> To: <argus-info at lists.andrew.cmu.edu>; <carter at qosient.com>
> Sent: Wednesday, March 27, 2002 12:07 PM
> Subject: RE: Reconstituting flows
> 
> 
> > Ahh, great, thats good to know.  That makes the decision easier. :)
> >
> > Chris
> >
> > >===== Original Message From <carter at qosient.com> =====
> > >The commercial data is a superset of argus data,
> > >but at present, the freebie ra* programs read
> > >commercial data fine, however they can't read the
> > >enhanced data.  That may change, since the commercial
> > >stuff isn't going to be in concrete for a few months
> > >still.
> > >
> > >Carter
> > >
> > >Carter Bullard
> > >QoSient, LLC
> > >300 E. 56th Street, Suite 18K
> > >New York, New York  10022
> > >
> > >carter at qosient.com
> > >Phone +1 212 588-9133
> > >Fax   +1 212 588-9134
> > >http://qosient.com
> 
> 
> 
> 
> 