argus data parsing

[Carter Bullard via Argus-info]

Hey Taran,
Hmmmm, there are a lot of features, capabilities, record types and protocol commands in argus.   I personally wouldn’t try to rewrite the argus-clients common library in Python.   The client common libraries are designed to make writing your own programs easier that doing it yourself.  The library can provide you with a complete core application.   You just have to write a few routines that the library calls out to; initialize, process the records, deal with timeouts and finally cleanup when the input is done. 

The library is designed provide most of the code, including the main() routine.  You just need to provide callback routines, to do the specific work you want to do with the records.  This strategy is codified in the example ratemplate.1 (code found in ./examples/ratemplate/ratemplate.c).  ra.1 is just ratemplate.1, but it prints out the contents of the records, so there are only a few lines of code to make ra.1.

But to make a Python program that reads argus data, if you can call out to C library routines from Python (ctypes), I would suggest that you check out how some of ratop.1 works and copy that in Python.  The routine to study is ArgusProcessData() in ./examples/ratop/raclient.c.  This sets up the type of access you want to do, and uses ArgusReadConnection() or ArgusReadFileStream() to read either socket or file based argus data.  It deals with encrypted argus data, compressed files, wireline compressed data, extracting floats (which is a huge pain in the rear end) allows you to seek to specific records, and extract the various argus data record types that could be in the data source.

So you would still provide C or Python callbacks for ArgusClientInit(), RaProcessRecord(), RaArgusInputComplete(), RaParseComplete(), and ArgusClientTimeout(), the routines the library wants to call.   

Now that your reading argus data records, use ra.1 as a guide to how to process the argus record types.  You want to deal with ARGUS_MAR, ARGUS_EVENT, ARGUS_NETFLOW and ARGUS_DATA record types.  There are others, but you won’t see them from open source argus.

Once you get the records from the library, then the argus_defs.h and argus_out.h structs are all you need to extract the info.
To “understand” the 180+ fields, you can use the printing routines to show you how the data is structured in the various DSRs (data supplement records).  They will show you how some DSRs use the TLV strategies that argus uses to process the data elements in the records.

Hopefully this will get you a little further along.

Carter

> On Dec 28, 2015, at 1:35 PM, Taran Stryker via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
> 
> I'm currently attempting to write my own Argus Client in Python to get a better sense of the data\ format that is contained in the Argus records as it does it's collection. I am to the point where I am connecting to the tcp socket established by Argus and then sending the "START: " command to the socket and gathering the data into a file, to inspect the records manually and try and reconcile what is contained in the record against what's shown in the argus_v3_def.h file, but it seems like the records that I'm collecting don't match up with the information in the .h file. My first question is the obvious, am I looking in the right place in the code for the info that I need to actually parse what I'm getting, and secondly, if I am, could you offer any insight into reading the Argus records. (For an example I'm looking at what should be the "type" field, but what I'm collecting doesn't match up with anything regarding the "Type" in the argus_v3_def.h file). Thanks for any help you can offer.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151228/b159ec2b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151228/b159ec2b/attachment.bin>