Developing an Argus client

[Harry Bock]

Thanks again for your help Carter; I've fixed the reference to QoSient on
our project page and
have progressed a bit more since my last email.

In my current copy of the argus-clients source tree, I've broken Periscope
out into its own directory, akin to ramysql and co., with a slightly
modified Makefile.in.  I'm building a small framework atop Argus to make it
more convenient for non-C programs to access data within Argus records, and
I've added a callback layer on top of the current scheme for
RaParseComplete/RaArgusInputComplete/etc., since the report-generating part
of Periscope won't be directly compiled against the Argus client libraries.

The most critical thing that I need to do at this point is to build a shared
lib of the Argus client library; I know how to do this for autotools when
the input files are for automake, but not for pure autoconf.  Any
suggestions on how to do this without breaking linking the other
applications to the static Argus libraries?  That would make it extremely
easy to do something similar to:

gcc periscope.c -o periscope -I/usr/include/argus-client -largus_client
-largus_common -largus_event

As shared library support might be useful to other developers working on
Argus clients, I'd suggest enabling this support in the default distribution
of argus-clients, so users can build applications like Periscope against
their distribution's build of Argus without needing a copy of these
modifications to the vanilla source tree.  I'd assume it'd be proper to make
the initial SOVERSION 3.0.2 for libargus_client, etc.?

Harry

On Mon, May 18, 2009 at 9:55 AM, Carter Bullard <carter at qosient.com> wrote:

> Hey Harry,This, actually, is what the development list is for, so if we
> could move the thread back
>  there, that would be great!!!!
>
> There are a lot of things in the new argus and argus-clients that will help
> you in
> what you are wanting to do, but I would suggest that you start looking at
> programs
> like rabins(), racluster(), rasqlinsert() and rasql() to begin developing
> your
> database schemas and analytic methods for Periscope, which I think are the
> hard parts of what you are getting into.
>
> If you are going to build intrusion/anomaly detection like technology, your
> persistent
> data store that holds your normal behavior (history), is the key to the
> strength of
> your detection capabilities. (need to know what is normal to know what is
> abnormal).
>
> If you were to help keep a thread going on the developer's mailing list on
> how to
> approach this kind of argus application, I will keep it active with answers
> and
> that can help to keep the new features on track.
>
> Just for a brief response, the libraries are structured so that you don't
> have to use
> the argus_main library.  I have many programs that have their own main(),
> but
> I don't have any documentation on specific routines, so it will be a bit of
> an effort
> to do the "whole nine yards", so to speak.  If you are interested in this
> path, no
> problem, I can help, but definately want this dialog on the developers
> mailing list.
>
> "Man" records are the management records.  They identify sources of argus
> data,
> and they are keep alives, so that you know that the transport stream is
> active and
> well.
>
> "Event" records are messages/data records that are not flow records, per
> se, and
> are generated at single points in time, thus the term 'event'.  The next
> release of
> argus() can do SNMP polling, and collect  information such as temperature,
> etc...
> and write the "events" into the argus data stream.
>
> Oh, and the company name is QoSient.
>
> Carter
>
>
> On May 14, 2009, at 2:17 PM, Harry Bock wrote:
>
> Hey Carter,
>
> To prevent spamming the list with development questions, I'll just email
> you directly for now.  If you feel our
> discussion would be helpful to others and should be on the list, I'd be
> happy to discuss this there. :)
>
> Thanks so much for the information!! It's been extremely helpful so far.
> I've modified ra to do some basic
> processing and data collection, and I've been working out how to handle the
> ArgusRecordStruct format and
> pull the information I need from it.
>
> I've got a few questions, as what I'd like to do with Argus is a bit
> different from how the available clients are
> written.  I need to compile the client library as a shared object, as the
> majority of the work done by the client
> will be done in Common Lisp atop a thin layer I'll be writing in C.  I'm
> not terribly familiar with autotools
> (more of a CMake guy myself, autotools is too hairy sometimes), so how
> difficult would it be to modify the
> current build system to produce a SO in addition to the current .a
> libraries?
>
> Since I will be calling into Argus from another language, I'm also going to
> need to modify the common client
> code slightly to eliminate dependance on argus_main.c, but that shouldn't
> be too difficult.  I'm not sure if this
> functionality would be useful to others; if so, I'll happily send you the
> patches for review.  Speaking of which,
> is Argus still maintained in CVS?  If you wouldn't mind, it would be
> helpful to have anonymous checkout access to
> the client repo to be able to easily integrate any important changes to the
> library.
>
> One more question for now... what are "Man" and "Event" records?  How are
> they relevant to client operations?
>
> If you're interested in the project we're working on, feel free to check
> out our project site at http://nautilus.oshean.org/wiki/Periscope .
> It's essentially intended to be a more useful and changeable replacement
> for IPAudit, and using the Argus server as
> a flow probe would vastly improve Periscope's operation.
>
> Thanks again for your help, I really appreciate it.
>
> Regards,
> Harry
>
> On Wed, May 13, 2009 at 6:54 PM, Carter Bullard <carter at qosient.com>wrote:
>
>> Hey Harry,
>> One key concept is that all the ra* programs in the argus-clients-3.0.2
>> distribution
>> are example programs, so don't be shy to grab one and use it as a start.
>>
>> The best example to use the complete library is ./clients/ra.c.  This
>> program
>> takes in a single record, and prints its contents, based on a
>> configuration.
>> If you wanted to write something that processed single records at a time,
>> ra.c
>> is a great starting point.
>>
>> Copy ./clients/ra.c to ./clients/rawhatever.c, and add lines in the
>> ./clients/Makefile.in so your new program will be compiled.  Below is a
>> patch
>> to ./clients/Makefile.in that adds rawhatever() as a new program to
>> compile.
>> Once you apply the patch,  then do this in the root directory of the
>> client distribution:
>>
>>  % cp ./clients/ra.c ./clients/rawhatever.c
>>  % ./configure
>>  % make
>>
>> At this time, you have a rawhatever() program in ./bin, that can do
>> everything,
>> open files, compressed or not, attach to remote data sources, negotiate
>> security
>> services, filter and process records, etc....  print the contents of each
>> record that is
>> read in, and can write out records to a new file.
>>
>> The idea is that the library provides main(), and calls out to the stub
>> routines that
>> are defined, in your rawhatever.c file, that deal with the basic states of
>> the
>> program, initialization, processing and termination.
>>
>> The argus-clients library makes calls to a few routines that you need to
>> provide.
>>
>> The first is ArgusClientInit(), which allows you to initialize whatever
>> you need to
>> initialize before you begin to process argus records.
>>
>> After your ArgusClientInit() routine returns, the argus library will open
>> all the
>> argus data sources, and start reading records.  After it has read the
>> input stream,
>> and generated the first record, the library applies any configured
>> filters, and if the
>> record passes,  it calls RaProcessRecord(), the routine where you will do
>> all the
>> per record processing.
>>
>> Every ArgusParser->RaClientTimeout period, the routine
>> ArgusClientTimeout() is
>> called.  This is an opportunity for you to get control to do maintenance
>> functions,
>> timeout things, check for whatever.  By default, it is called every
>> second, but you
>> can change this value, usually in ArgusClientInit().  The program
>> rastream() changes
>> this value, and specifies a lot of stuff to do during these
>> ArgusClientTimeout()
>> calls, so you can use it as an example, although a complicated one.
>>
>> When a record is ready to be processed, the library calls the routine
>> RaProcessRecord().
>> You supply what to do in this case.  The program ra.c is a great example.
>>
>> When each input source is finished, the routine RaArgusInputComplete() is
>> called.
>> Most programs don't do anything, but rasort.c is a great example of what
>> you can
>> do at the end of each file.  rasort() has an option to sort files in
>> place, which means its
>> going to sort each input file, output the sorted records into a temporary
>> file, and then
>> rename the temporary file to the original filename.  We need to do this as
>> each file is
>> processed.
>>
>> After all the inputs have closed, the argus-client library will call the
>> routine
>> RaParseComplete().   Here you want to close everything, write out stuff,
>> deallocate stuff, whatever you need to do after all the records have been
>> read in and processed.  ra.c has a great "flush data to any output file
>> that is
>> interested, and then close them all" strategy that is a minimum.
>>
>> Hopefully this will get you started, once you get your rawhatever.c to do
>> something,
>> holler if you like, and I'll describe strategies for doing simple and not
>> so simple clients.
>>
>>
>>
>> Carter
>>
>> ----------begin patch-----------
>> set:clients carter$ diff -c Makefile.in Makefile.in.new
>> *** Makefile.in Fri Apr 24 16:53:58 2009
>> --- Makefile.in.new     Wed May 13 18:22:24 2009
>> ***************
>> *** 82,88 ****
>>
>>  SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c racluster.c
>> rahisto.c ralabel.c \
>>        rapolicy.c ranonymize.c rapath.c rastream.c ratree.c rafilteraddr.c
>> ratimerange.c \
>> !       rauserdata.c raservices.c ratemplate.c
>>
>>  PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/rasort
>> @INSTALL_BIN@/rasplit \
>>        @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/rastrip
>> \
>> --- 82,88 ----
>>
>>  SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c racluster.c
>> rahisto.c ralabel.c \
>>        rapolicy.c ranonymize.c rapath.c rastream.c ratree.c rafilteraddr.c
>> ratimerange.c \
>> !       rauserdata.c raservices.c ratemplate.c rawhatever.c
>>
>>  PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/rasort
>> @INSTALL_BIN@/rasplit \
>>        @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/rastrip
>> \
>> ***************
>> *** 90,96 ****
>>        @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/rastream
>> \
>>        @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr @INSTALL_BIN@/ratimerange
>> \
>>        @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/radark \
>> !       @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices @INSTALL_BIN@
>> /ratemplate
>>
>>  all: $(PROGS)
>>
>> --- 90,97 ----
>>        @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/rastream
>> \
>>        @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr @INSTALL_BIN@/ratimerange
>> \
>>        @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/radark \
>> !       @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices @INSTALL_BIN@/ratemplate
>> \
>> !       @INSTALL_BIN@/rawhatever
>>
>>  all: $(PROGS)
>>
>> ***************
>> *** 163,168 ****
>> --- 164,172 ----
>>  @INSTALL_BIN@/ratemplate: ratemplate.o $(LIB)
>>        $(CC) $(CCOPT) -o $@ ratemplate.o $(LIB) $(COMPATLIB)
>>
>> + @INSTALL_BIN@/rawhatever: rawhatever.o $(LIB)
>> +       $(CC) $(CCOPT) -o $@ rawhatever.o $(LIB) $(COMPATLIB)
>> +
>>  # We would like to say "OBJ = $(SRC:.c=.o)" but Ultrix's make cannot
>>  # hack the extra indirection
>>
>> ----------end patch-----------
>>
>>
>> On May 13, 2009, at 12:31 PM, Harry Bock wrote:
>>
>>  Hi all,
>>>
>>> I was wondering if I could get any guidance on where to start for
>>> developing a new Argus client.  The programming model used in the current
>>> Argus clients is a little hard to follow, and there doesn't seem to be a
>>> whole lot of documentation in the source code or on the website for
>>> developing one.
>>>
>>> I have been working on a very similar application for my company, and
>>> I've realized that Argus does most of what I'm looking for already, and it's
>>> already usable :) If I could leverage Argus as the core of our application,
>>> that would save us a lot of development time and headaches, so any hints in
>>> the right direction would be very much appreciated.
>>>
>>> Regards,
>>> Harry Bock
>>>
>>>
>>>
>>
>
>
> --
> Harry Bock
> Software Developer, Package Maintainer
> OSHEAN, Inc.
> Email: harry at oshean.org
> PGP Key ID: 546CC353
>
>
>
>
>
>


-- 
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry at oshean.org
PGP Key ID: 546CC353
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090520/a0f4a511/attachment.html>