Developing an Argus client
Carter Bullard
carter at qosient.com
Wed May 20 14:53:15 EDT 2009
Hey Harry,
I don't want to keep you from doing anything, but I think I should
probably
do this thing that you are seeking.
Do you have any experience with gnu's libtool?
Carter
On May 20, 2009, at 2:19 PM, Harry Bock wrote:
> Thanks again for your help Carter; I've fixed the reference to
> QoSient on our project page and
> have progressed a bit more since my last email.
>
> In my current copy of the argus-clients source tree, I've broken
> Periscope out into its own directory, akin to ramysql and co., with
> a slightly modified Makefile.in. I'm building a small framework
> atop Argus to make it more convenient for non-C programs to access
> data within Argus records, and I've added a callback layer on top of
> the current scheme for RaParseComplete/RaArgusInputComplete/etc.,
> since the report-generating part of Periscope won't be directly
> compiled against the Argus client libraries.
>
> The most critical thing that I need to do at this point is to build
> a shared lib of the Argus client library; I know how to do this for
> autotools when the input files are for automake, but not for pure
> autoconf. Any suggestions on how to do this without breaking
> linking the other applications to the static Argus libraries? That
> would make it extremely easy to do something similar to:
>
> gcc periscope.c -o periscope -I/usr/include/argus-client -
> largus_client -largus_common -largus_event
>
> As shared library support might be useful to other developers
> working on Argus clients, I'd suggest enabling this support in the
> default distribution of argus-clients, so users can build
> applications like Periscope against their distribution's build of
> Argus without needing a copy of these modifications to the vanilla
> source tree. I'd assume it'd be proper to make the initial
> SOVERSION 3.0.2 for libargus_client, etc.?
>
> Harry
>
> On Mon, May 18, 2009 at 9:55 AM, Carter Bullard <carter at qosient.com>
> wrote:
> Hey Harry,
> This, actually, is what the development list is for, so if we could
> move the thread back
> there, that would be great!!!!
>
> There are a lot of things in the new argus and argus-clients that
> will help you in
> what you are wanting to do, but I would suggest that you start
> looking at programs
> like rabins(), racluster(), rasqlinsert() and rasql() to begin
> developing your
> database schemas and analytic methods for Periscope, which I think
> are the
> hard parts of what you are getting into.
>
> If you are going to build intrusion/anomaly detection like
> technology, your persistent
> data store that holds your normal behavior (history), is the key to
> the strength of
> your detection capabilities. (need to know what is normal to know
> what is abnormal).
>
> If you were to help keep a thread going on the developer's mailing
> list on how to
> approach this kind of argus application, I will keep it active with
> answers and
> that can help to keep the new features on track.
>
> Just for a brief response, the libraries are structured so that you
> don't have to use
> the argus_main library. I have many programs that have their own
> main(), but
> I don't have any documentation on specific routines, so it will be a
> bit of an effort
> to do the "whole nine yards", so to speak. If you are interested in
> this path, no
> problem, I can help, but definately want this dialog on the
> developers mailing list.
>
> "Man" records are the management records. They identify sources of
> argus data,
> and they are keep alives, so that you know that the transport stream
> is active and
> well.
>
> "Event" records are messages/data records that are not flow records,
> per se, and
> are generated at single points in time, thus the term 'event'. The
> next release of
> argus() can do SNMP polling, and collect information such as
> temperature, etc...
> and write the "events" into the argus data stream.
>
> Oh, and the company name is QoSient.
>
> Carter
>
>
> On May 14, 2009, at 2:17 PM, Harry Bock wrote:
>
>> Hey Carter,
>>
>> To prevent spamming the list with development questions, I'll just
>> email you directly for now. If you feel our
>> discussion would be helpful to others and should be on the list,
>> I'd be happy to discuss this there. :)
>>
>> Thanks so much for the information!! It's been extremely helpful so
>> far. I've modified ra to do some basic
>> processing and data collection, and I've been working out how to
>> handle the ArgusRecordStruct format and
>> pull the information I need from it.
>>
>> I've got a few questions, as what I'd like to do with Argus is a
>> bit different from how the available clients are
>> written. I need to compile the client library as a shared object,
>> as the majority of the work done by the client
>> will be done in Common Lisp atop a thin layer I'll be writing in
>> C. I'm not terribly familiar with autotools
>> (more of a CMake guy myself, autotools is too hairy sometimes), so
>> how difficult would it be to modify the
>> current build system to produce a SO in addition to the current .a
>> libraries?
>>
>> Since I will be calling into Argus from another language, I'm also
>> going to need to modify the common client
>> code slightly to eliminate dependance on argus_main.c, but that
>> shouldn't be too difficult. I'm not sure if this
>> functionality would be useful to others; if so, I'll happily send
>> you the patches for review. Speaking of which,
>> is Argus still maintained in CVS? If you wouldn't mind, it would
>> be helpful to have anonymous checkout access to
>> the client repo to be able to easily integrate any important
>> changes to the library.
>>
>> One more question for now... what are "Man" and "Event" records?
>> How are they relevant to client operations?
>>
>> If you're interested in the project we're working on, feel free to
>> check out our project site at http://nautilus.oshean.org/wiki/Periscope
>> .
>> It's essentially intended to be a more useful and changeable
>> replacement for IPAudit, and using the Argus server as
>> a flow probe would vastly improve Periscope's operation.
>>
>> Thanks again for your help, I really appreciate it.
>>
>> Regards,
>> Harry
>>
>> On Wed, May 13, 2009 at 6:54 PM, Carter Bullard
>> <carter at qosient.com> wrote:
>> Hey Harry,
>> One key concept is that all the ra* programs in the argus-
>> clients-3.0.2 distribution
>> are example programs, so don't be shy to grab one and use it as a
>> start.
>>
>> The best example to use the complete library is ./clients/ra.c.
>> This program
>> takes in a single record, and prints its contents, based on a
>> configuration.
>> If you wanted to write something that processed single records at a
>> time, ra.c
>> is a great starting point.
>>
>> Copy ./clients/ra.c to ./clients/rawhatever.c, and add lines in the
>> ./clients/Makefile.in so your new program will be compiled. Below
>> is a patch
>> to ./clients/Makefile.in that adds rawhatever() as a new program to
>> compile.
>> Once you apply the patch, then do this in the root directory of
>> the client distribution:
>>
>> % cp ./clients/ra.c ./clients/rawhatever.c
>> % ./configure
>> % make
>>
>> At this time, you have a rawhatever() program in ./bin, that can do
>> everything,
>> open files, compressed or not, attach to remote data sources,
>> negotiate security
>> services, filter and process records, etc.... print the contents
>> of each record that is
>> read in, and can write out records to a new file.
>>
>> The idea is that the library provides main(), and calls out to the
>> stub routines that
>> are defined, in your rawhatever.c file, that deal with the basic
>> states of the
>> program, initialization, processing and termination.
>>
>> The argus-clients library makes calls to a few routines that you
>> need to provide.
>>
>> The first is ArgusClientInit(), which allows you to initialize
>> whatever you need to
>> initialize before you begin to process argus records.
>>
>> After your ArgusClientInit() routine returns, the argus library
>> will open all the
>> argus data sources, and start reading records. After it has read
>> the input stream,
>> and generated the first record, the library applies any configured
>> filters, and if the
>> record passes, it calls RaProcessRecord(), the routine where you
>> will do all the
>> per record processing.
>>
>> Every ArgusParser->RaClientTimeout period, the routine
>> ArgusClientTimeout() is
>> called. This is an opportunity for you to get control to do
>> maintenance functions,
>> timeout things, check for whatever. By default, it is called every
>> second, but you
>> can change this value, usually in ArgusClientInit(). The program
>> rastream() changes
>> this value, and specifies a lot of stuff to do during these
>> ArgusClientTimeout()
>> calls, so you can use it as an example, although a complicated one.
>>
>> When a record is ready to be processed, the library calls the
>> routine RaProcessRecord().
>> You supply what to do in this case. The program ra.c is a great
>> example.
>>
>> When each input source is finished, the routine
>> RaArgusInputComplete() is called.
>> Most programs don't do anything, but rasort.c is a great example of
>> what you can
>> do at the end of each file. rasort() has an option to sort files
>> in place, which means its
>> going to sort each input file, output the sorted records into a
>> temporary file, and then
>> rename the temporary file to the original filename. We need to do
>> this as each file is
>> processed.
>>
>> After all the inputs have closed, the argus-client library will
>> call the routine
>> RaParseComplete(). Here you want to close everything, write out
>> stuff,
>> deallocate stuff, whatever you need to do after all the records
>> have been
>> read in and processed. ra.c has a great "flush data to any output
>> file that is
>> interested, and then close them all" strategy that is a minimum.
>>
>> Hopefully this will get you started, once you get your rawhatever.c
>> to do something,
>> holler if you like, and I'll describe strategies for doing simple
>> and not so simple clients.
>>
>>
>>
>> Carter
>>
>> ----------begin patch-----------
>> set:clients carter$ diff -c Makefile.in Makefile.in.new
>> *** Makefile.in Fri Apr 24 16:53:58 2009
>> --- Makefile.in.new Wed May 13 18:22:24 2009
>> ***************
>> *** 82,88 ****
>>
>> SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c
>> racluster.c rahisto.c ralabel.c \
>> rapolicy.c ranonymize.c rapath.c rastream.c ratree.c
>> rafilteraddr.c ratimerange.c \
>> ! rauserdata.c raservices.c ratemplate.c
>>
>> PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/
>> rasort @INSTALL_BIN@/rasplit \
>> @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/
>> rastrip \
>> --- 82,88 ----
>>
>> SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c
>> racluster.c rahisto.c ralabel.c \
>> rapolicy.c ranonymize.c rapath.c rastream.c ratree.c
>> rafilteraddr.c ratimerange.c \
>> ! rauserdata.c raservices.c ratemplate.c rawhatever.c
>>
>> PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/
>> rasort @INSTALL_BIN@/rasplit \
>> @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/
>> rastrip \
>> ***************
>> *** 90,96 ****
>> @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/
>> rastream \
>> @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr
>> @INSTALL_BIN@/ratimerange \
>> @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/
>> radark \
>> ! @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices
>> @INSTALL_BIN@/ratemplate
>>
>> all: $(PROGS)
>>
>> --- 90,97 ----
>> @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/
>> rastream \
>> @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr
>> @INSTALL_BIN@/ratimerange \
>> @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/
>> radark \
>> ! @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices
>> @INSTALL_BIN@/ratemplate \
>> ! @INSTALL_BIN@/rawhatever
>>
>> all: $(PROGS)
>>
>> ***************
>> *** 163,168 ****
>> --- 164,172 ----
>> @INSTALL_BIN@/ratemplate: ratemplate.o $(LIB)
>> $(CC) $(CCOPT) -o $@ ratemplate.o $(LIB) $(COMPATLIB)
>>
>> + @INSTALL_BIN@/rawhatever: rawhatever.o $(LIB)
>> + $(CC) $(CCOPT) -o $@ rawhatever.o $(LIB) $(COMPATLIB)
>> +
>> # We would like to say "OBJ = $(SRC:.c=.o)" but Ultrix's make cannot
>> # hack the extra indirection
>>
>> ----------end patch-----------
>>
>>
>> On May 13, 2009, at 12:31 PM, Harry Bock wrote:
>>
>> Hi all,
>>
>> I was wondering if I could get any guidance on where to start for
>> developing a new Argus client. The programming model used in the
>> current Argus clients is a little hard to follow, and there doesn't
>> seem to be a whole lot of documentation in the source code or on
>> the website for developing one.
>>
>> I have been working on a very similar application for my company,
>> and I've realized that Argus does most of what I'm looking for
>> already, and it's already usable :) If I could leverage Argus as
>> the core of our application, that would save us a lot of
>> development time and headaches, so any hints in the right direction
>> would be very much appreciated.
>>
>> Regards,
>> Harry Bock
>>
>>
>>
>>
>>
>>
>> --
>> Harry Bock
>> Software Developer, Package Maintainer
>> OSHEAN, Inc.
>> Email: harry at oshean.org
>> PGP Key ID: 546CC353
>
>
>
>
>
>
>
>
> --
> Harry Bock
> Software Developer, Package Maintainer
> OSHEAN, Inc.
> Email: harry at oshean.org
> PGP Key ID: 546CC353
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090520/cde0a389/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090520/cde0a389/attachment.bin>
More information about the argus
mailing list