Argus-2.0 Output record format

Fri Aug 25 10:23:40 EDT 2000

Gentle people,
Its time to nail down some of the new features
of Argus-2.0, and so working on TCP and its support
is a good starting point.  There is a lot in this
e-mail, so go get some coffee if you plan to read
this now.

There are at least 3 things to do for TCP as a result
of the new data strategy that we've got for Argus-2.0.

1. We need to define the TCP Data Supplement Record (DSR).
2. We need to define the set of TCP Event Records and 
3. We need to define the TCP default ra() output string.

I have already started some of this but it would really
be nice to get some dialog going.  Hopefully this will
get us into other issues that have been brought up on
the list.

At the end of this mail is a little description of some
of the Argus-2.0 record format.  If you have time,
please take a look at it.  It is not complete, and
any changes are possible, so don't hesitate
to voice an opinion/objection/comment/whatever.  This is
just a collection of thoughts on the matter.  I'll
have a word document that describes it in detail later.

So just to dive right in there:

For TCP we will generate a standard Flow Activity
Record (FAR), but there won't be any TCP specific
information in it.  For this we will generate a
TCP specific Data Supplement Record (DSR).  Now the
TCP DSR could be a separate Argus Record or it could
be a section that is added to the standard Flow Record. 
This is up in the air.

If we separate them, then both will contain a common
Argus Transaction ID that will allow Argus data readers
to correlate the two together.

Arguments for a separate record are that it doesn't
have to be written at the time of the Flow Record.
This may or may not be an issue.
Separate DSR's can be ripped out of the Argus Record
stream very quickly, where removing sections will
cause us to have to copy the sections that we will be
keeping and recalculate the length field and some other
things that may prove to be computationally expensive.

Arguments against a separate record.  It will cost
us an additional 16 bytes for the Argus Record header.
It will/could mess up our sequence numbers if we start
ripping DSRs out of the stream, and sorting Argus files
might become computationally expensive, but maybe not.
XML conversion will be faster.

Now this record is for TCP Layer information only.
User data, such as HTTP URLs and the like would be
in another DSR, so we'll hold off on that for a short
while.

We are currently collecting these src and dst 
metrics for all TCP connection reports in Argus-2.0:

   1. TCP options (window scale, echo, Sack, timestamp, ECN)
   2. Base sequence numbers.
   3. Round Trip Connection Establishment Times (usec).
   4. TCP State Progression (syn, syn_ack, data, closing, closed, rst, etc)
   5. Accumulated TCP flags values.
   6. TCP payload bytes and Acknowledged bytes.
   7. Packet/Byte Retransmission counts.
   9. Flow Control Indication.
  10. Last Reported Window size
  11. Number of Windows completed.
  12. Explicit Congestion Notification Indications.

Although this may seem like a lot, are there any other
things that we need to report for TCP?  I'd like to add
things like "Slow Start Seen" indicators and the like,
but we aren't doing that today, but we can definitely put
it on the list.

If we are happy with the strategy, I'll develop a TCP DSR
format that includes this data and send it out early next
week.  We'll get to the TCP Event Record and the default
ra() output string soon.

carter

Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 17A
New York, New York  10022

carter at qosient.com
Phone +1 212 813-9426
Fax   +1 212 813-9426

Argus-2.0 Flow Activity Record (FAR) Format

In Argus-2.0 we are introducing two new record classes,
the Supplemental Data Record (SDR) and the Argus Event
Record (AER).  These are in addition to the standard
Argus Flow Activity Record (FAR) and the Management
Records that we generate in Argus-1.x.

So, understanding what is in the FAR is a good thing.
So, here we go ......

Argus will generate a well formed stream of Argus Records.
There are two general classes of Argus Records, Management
Action Records (MAR) and Flow Activity Records (FAR).

Argus Records are variable length records.  They all
must have an Argus Record Header, in order to be an
Argus Record. The Argus Header is composed of:

Argus Record Header Format

    0                   1                   2                   3   
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Type      |     Cause     |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Ver  |  Opt  |                    Status                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Argus Identifier                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

So far here are the Types and Causes that I've got.

/* Argus Operation Status Type */

#define ARGUS_MAR			0x80    /* Argus Management Action Record */
#define ARGUS_INDEX   		0x90    /* New Argus Index Record */
#define ARGUS_EVENT		0xA0    /* New Argus Event/Message Record */

#define ARGUS_FAR			0x01    /* Argus Flow Activity Record */
#define ARGUS_DSR			0x02    /* New Argus Data Supplemental Record */

/* Record Cause */

#define ARGUS_START		0x01   /* INIT */
#define ARGUS_CONNECTED		0x02   /* CON */
#define ARGUS_STATUS		0x04   /* STATUS */
#define ARGUS_STOP		0x08   /* CLOSE */
#define ARGUS_SHUTDOWN		0x10   /* Administrative shutdown */
#define ARGUS_TIMEOUT		0x20   /* TIMEOUT */
#define ARGUS_ERROR		0x40   /* MAJOR PROBLEM */

/* Record Version (Ver) */

#define ARGUS_VERSION		0x2    /* Version 2 */

/* Record Options (Opt)*/

#define ARGUS_DETAIL		0x1
#define ARGUS_MERGED		0x2

/*
   Record Status

   The record status is supplemental information
   to the record type and cause field.

*/

/* Type:  DSR    Cause:  ANY  */

#define ARGUS_TCP_DSR		0x000010
#define ARGUS_ICMP_DSR		0x000020
#define ARGUS_RTP_DSR		0x000040
#define ARGUS_IGMP_DSR		0x000080
#define ARGUS_ARP_DSR		0x000100
#define ARGUS_FRAG_DSR		0x000200

#define ARGUS_JITTER_DSR	0x001000
#define ARGUS_USRDATA_DSR	0x002000

/* Type:  FAR    Cause:  ANY  */

#define ARGUS_IP4_FLOW		0x010000
#define ARGUS_IP6_FLOW		0x020000
#define ARGUS_ARP_FLOW		0x040000
#define ARGUS_ETHERTYPE		0x00FFFF

/* Argus Error Messages go into the status field when
   the Record Cause is ARGUS_ERROR.
*/

#define ARGUS_ACCESSDENIED	0x000010
#define ARGUS_MAXXLISTENEXCD	0x000020

The Management record will stay pretty much the same.
The ARGUS_START MAR is designed to identify the monitor
and report on its status, uptime and the like.
ARGUS_STATUS MAR records are to report on monitor
activity, packets, bytes, drops, flows, etc...
The ARGUS_STOP MAR records are designed to report what
happened.

All FARs are structured the same.  A FAR is a Flow
Activity Record, and so it has to have a Flow Descriptor,
and it has to have some activity metrics bounded over
a time range.  So to support this, each FAR will have
an Argus Transaction Identifier, a Time Range Descriptor,
a Flow Descriptor and the standard flow metrics.

Argus Transaction ID
This is an Argus internally relevant flow identifier,
and is used to correlate FARs, DSRs and Events
together.

Time Range Descriptor
I'm changing the time range descriptor.  In Argus-1.x
we had 2 timeval structures (16 bytes), but in Argus-2.0
we're moving to start time and duration in usec.
This will save us 4 bytes.

Flow Descriptor

There are two basic types of Flow Descriptors.
The Flow Key Descriptor and the Flow Attributes.
The Flow Key Descriptor is the collection of values
that were used as the key to classify the packets
into flows.

The Flow Attributes are the rest of the identifiers
that a flow possesses, but were not used in the key.
As an example, the MAC addresses use by the flow
are not a part of IP flow key, since MAC address is
not a part of the IP Flow model.  But the MAC addresses
are definitely attributes of the flow.

Reporting Flow Attributes is optional, and when you
aggregate Argus Records together, the Flow Attributes
generally are discarded, so we need to be able to
have optional Flow Descriptors and have something that
describes which Flow Descriptors were used to make the
key.

The metrics are src and dst packets and bytes, and inter
packet arrival time information.

So far, my thinking is that we've got Data Supplement
Records (DSRs) potentially at each Layer.  So a layer
based DSR breakdown would have:

   1. Layer 2 DSR - This is used to report MAC 
          addresses used by the flow.  We use to put this
          in each record by default, but I think that we'll
          save some space if we make it optional.

   2. Layer 4 DSR - Here is where we get TCP supplemental
          flow metric data reporting.

   3. Layer 5 DSR - Here we'll have RTP and RTCP protocol
          data reporting (this usually sits just above UDP).

   3. Layer 7 DSR - Here is where we get stuff like HTTP
          URL labels and the generic user data reporting.