Fw: Reconstituting flows

[Roy Hooper]

(If this comes in duplicate, I apologise, I sent the orginal from the wrong
email address)

Chris (Carter),

You've probably already thought of this, but it occurs to me that there might
be possibility to "do the right thing" exists at the moment.  That right thing
is to allow any version tool to read any version's data. I haven't read the
details on the argus datafile format, so these ideas may already be
addressed...

Since the argus data format is intended to be processed in streams, that is, it
is not indexed, and the client tools are designed to use the datafiles in a
sequantial manner, records need not be the same length...  Given this, a record
format that is tagged with the generating application and version (2 bytes?)
and also containing optional extended data blocks after the fixed length
portion containing something like this:
    extended = extended_record ... NULL
    extended_record = length generating_app version recordtype data

An extension of this idea to make processing slightly faster that I have used
in the past is a meta record type that specifies that the following N bytes of
data are all of the same type, and that type is X, so you can read them all in
one read, or seek past them.  Synchronizing this against datafile rotation
could be a pain in the ass...

This basic idea would allow you to keep extending the data format and still
have the data be useful by older ra* clients and non-commercial versions ...

And another idea that's been kicking around in my head is either some form of
datafile indexer or better "racount" style processor/summarizer.  I'd like to
be able to keep in the order of 7 days of argus logs, but also keep protocol
stats indefinitely (well, okay, up to a specific limit of time) in an RRD-like
format but that doesn't require all of the space be allocated at
initialization.  The stats i'd like to summarize out of the argus data on an
periodic (5 minutes?) hourly, daily, weekly, monthly, yearly basis are:
    usage by node based on MAC (not as important, but would be nice)
    usage by IP
    usage by IP per protocol
    usage by IP per protocol + service

In order to generate such stats, I would not be interested in conversations,
but instead on data bytes in/out and would want to view the data (and store it)
based on a local network.  That is to say that for a conversation between
192.168.0.1 and 10.0.0.1 where 10.0.0.0/24 is the local network and 192.168.0.1
is some remote site on the internet, I would be interested in keeping the stats
from the point of view of 10.0.0.1 and would never record 192.168.0.1 anywhere
in this summary data.

BTW, argus is way cool.  Have you thought about including hooks in argus to be
able to load modules to do further data processing on packets to be able to
generate further accounting data, extracting by URLs, FTP filenames, POP/IMAP
logins, POP/IMAP messages received, messages uploaded for IMAP, SMTP messages
in/out - sender/recipients, etc.  I think snort already has most of the code
you'd need to do some of this in its modules.

Anyway, enough questions for now, feel free to answer with see FAQ or see code
directory X or file Y, see CVS, etc.

---
Roy Hooper
Project Manager & Senior UNIX Consultant
Decisive Technologies, Inc.
rhooper at decisivetechnologies.com


----- Original Message -----
From: "newton" <newton at unb.ca>
To: <argus-info at lists.andrew.cmu.edu>; <carter at qosient.com>
Sent: Wednesday, March 27, 2002 12:07 PM
Subject: RE: Reconstituting flows


> Ahh, great, thats good to know.  That makes the decision easier. :)
>
> Chris
>
> >===== Original Message From <carter at qosient.com> =====
> >The commercial data is a superset of argus data,
> >but at present, the freebie ra* programs read
> >commercial data fine, however they can't read the
> >enhanced data.  That may change, since the commercial
> >stuff isn't going to be in concrete for a few months
> >still.
> >
> >Carter
> >
> >Carter Bullard
> >QoSient, LLC
> >300 E. 56th Street, Suite 18K
> >New York, New York  10022
> >
> >carter at qosient.com
> >Phone +1 212 588-9133
> >Fax   +1 212 588-9134
> >http://qosient.com