Developing an Argus client
Harry Bock
harry at oshean.org
Fri May 22 10:32:37 EDT 2009
Hey Carter,
Unfortunately, I don't have very much experience with autotools, just some
basic stuff with autoconf/automake, and none at all with GNU libtool. I
prefer CMake for projects like this, but that doesn't help :)
I hope this change won't cause any headaches for you; I really appreciate
the help!
Harry
On Wed, May 20, 2009 at 2:53 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Harry,I don't want to keep you from doing anything, but I think I
> should probably
> do this thing that you are seeking.
>
> Do you have any experience with gnu's libtool?
>
> Carter
>
> On May 20, 2009, at 2:19 PM, Harry Bock wrote:
>
> Thanks again for your help Carter; I've fixed the reference to QoSient on
> our project page and
> have progressed a bit more since my last email.
>
> In my current copy of the argus-clients source tree, I've broken Periscope
> out into its own directory, akin to ramysql and co., with a slightly
> modified Makefile.in. I'm building a small framework atop Argus to make it
> more convenient for non-C programs to access data within Argus records, and
> I've added a callback layer on top of the current scheme for
> RaParseComplete/RaArgusInputComplete/etc., since the report-generating part
> of Periscope won't be directly compiled against the Argus client libraries.
>
> The most critical thing that I need to do at this point is to build a
> shared lib of the Argus client library; I know how to do this for autotools
> when the input files are for automake, but not for pure autoconf. Any
> suggestions on how to do this without breaking linking the other
> applications to the static Argus libraries? That would make it extremely
> easy to do something similar to:
>
> gcc periscope.c -o periscope -I/usr/include/argus-client -largus_client
> -largus_common -largus_event
>
> As shared library support might be useful to other developers working on
> Argus clients, I'd suggest enabling this support in the default distribution
> of argus-clients, so users can build applications like Periscope against
> their distribution's build of Argus without needing a copy of these
> modifications to the vanilla source tree. I'd assume it'd be proper to make
> the initial SOVERSION 3.0.2 for libargus_client, etc.?
>
> Harry
>
> On Mon, May 18, 2009 at 9:55 AM, Carter Bullard <carter at qosient.com>wrote:
>
>> Hey Harry,This, actually, is what the development list is for, so if we
>> could move the thread back
>> there, that would be great!!!!
>>
>> There are a lot of things in the new argus and argus-clients that will
>> help you in
>> what you are wanting to do, but I would suggest that you start looking at
>> programs
>> like rabins(), racluster(), rasqlinsert() and rasql() to begin developing
>> your
>> database schemas and analytic methods for Periscope, which I think are the
>> hard parts of what you are getting into.
>>
>> If you are going to build intrusion/anomaly detection like technology,
>> your persistent
>> data store that holds your normal behavior (history), is the key to the
>> strength of
>> your detection capabilities. (need to know what is normal to know what is
>> abnormal).
>>
>> If you were to help keep a thread going on the developer's mailing list on
>> how to
>> approach this kind of argus application, I will keep it active with
>> answers and
>> that can help to keep the new features on track.
>>
>> Just for a brief response, the libraries are structured so that you don't
>> have to use
>> the argus_main library. I have many programs that have their own main(),
>> but
>> I don't have any documentation on specific routines, so it will be a bit
>> of an effort
>> to do the "whole nine yards", so to speak. If you are interested in this
>> path, no
>> problem, I can help, but definately want this dialog on the developers
>> mailing list.
>>
>> "Man" records are the management records. They identify sources of argus
>> data,
>> and they are keep alives, so that you know that the transport stream is
>> active and
>> well.
>>
>> "Event" records are messages/data records that are not flow records, per
>> se, and
>> are generated at single points in time, thus the term 'event'. The next
>> release of
>> argus() can do SNMP polling, and collect information such as temperature,
>> etc...
>> and write the "events" into the argus data stream.
>>
>> Oh, and the company name is QoSient.
>>
>> Carter
>>
>>
>> On May 14, 2009, at 2:17 PM, Harry Bock wrote:
>>
>> Hey Carter,
>>
>> To prevent spamming the list with development questions, I'll just email
>> you directly for now. If you feel our
>> discussion would be helpful to others and should be on the list, I'd be
>> happy to discuss this there. :)
>>
>> Thanks so much for the information!! It's been extremely helpful so far.
>> I've modified ra to do some basic
>> processing and data collection, and I've been working out how to handle
>> the ArgusRecordStruct format and
>> pull the information I need from it.
>>
>> I've got a few questions, as what I'd like to do with Argus is a bit
>> different from how the available clients are
>> written. I need to compile the client library as a shared object, as the
>> majority of the work done by the client
>> will be done in Common Lisp atop a thin layer I'll be writing in C. I'm
>> not terribly familiar with autotools
>> (more of a CMake guy myself, autotools is too hairy sometimes), so how
>> difficult would it be to modify the
>> current build system to produce a SO in addition to the current .a
>> libraries?
>>
>> Since I will be calling into Argus from another language, I'm also going
>> to need to modify the common client
>> code slightly to eliminate dependance on argus_main.c, but that shouldn't
>> be too difficult. I'm not sure if this
>> functionality would be useful to others; if so, I'll happily send you the
>> patches for review. Speaking of which,
>> is Argus still maintained in CVS? If you wouldn't mind, it would be
>> helpful to have anonymous checkout access to
>> the client repo to be able to easily integrate any important changes to
>> the library.
>>
>> One more question for now... what are "Man" and "Event" records? How are
>> they relevant to client operations?
>>
>> If you're interested in the project we're working on, feel free to check
>> out our project site at http://nautilus.oshean.org/wiki/Periscope .
>> It's essentially intended to be a more useful and changeable replacement
>> for IPAudit, and using the Argus server as
>> a flow probe would vastly improve Periscope's operation.
>>
>> Thanks again for your help, I really appreciate it.
>>
>> Regards,
>> Harry
>>
>> On Wed, May 13, 2009 at 6:54 PM, Carter Bullard <carter at qosient.com>wrote:
>>
>>> Hey Harry,
>>> One key concept is that all the ra* programs in the argus-clients-3.0.2
>>> distribution
>>> are example programs, so don't be shy to grab one and use it as a start.
>>>
>>> The best example to use the complete library is ./clients/ra.c. This
>>> program
>>> takes in a single record, and prints its contents, based on a
>>> configuration.
>>> If you wanted to write something that processed single records at a time,
>>> ra.c
>>> is a great starting point.
>>>
>>> Copy ./clients/ra.c to ./clients/rawhatever.c, and add lines in the
>>> ./clients/Makefile.in so your new program will be compiled. Below is a
>>> patch
>>> to ./clients/Makefile.in that adds rawhatever() as a new program to
>>> compile.
>>> Once you apply the patch, then do this in the root directory of the
>>> client distribution:
>>>
>>> % cp ./clients/ra.c ./clients/rawhatever.c
>>> % ./configure
>>> % make
>>>
>>> At this time, you have a rawhatever() program in ./bin, that can do
>>> everything,
>>> open files, compressed or not, attach to remote data sources, negotiate
>>> security
>>> services, filter and process records, etc.... print the contents of each
>>> record that is
>>> read in, and can write out records to a new file.
>>>
>>> The idea is that the library provides main(), and calls out to the stub
>>> routines that
>>> are defined, in your rawhatever.c file, that deal with the basic states
>>> of the
>>> program, initialization, processing and termination.
>>>
>>> The argus-clients library makes calls to a few routines that you need to
>>> provide.
>>>
>>> The first is ArgusClientInit(), which allows you to initialize whatever
>>> you need to
>>> initialize before you begin to process argus records.
>>>
>>> After your ArgusClientInit() routine returns, the argus library will open
>>> all the
>>> argus data sources, and start reading records. After it has read the
>>> input stream,
>>> and generated the first record, the library applies any configured
>>> filters, and if the
>>> record passes, it calls RaProcessRecord(), the routine where you will do
>>> all the
>>> per record processing.
>>>
>>> Every ArgusParser->RaClientTimeout period, the routine
>>> ArgusClientTimeout() is
>>> called. This is an opportunity for you to get control to do maintenance
>>> functions,
>>> timeout things, check for whatever. By default, it is called every
>>> second, but you
>>> can change this value, usually in ArgusClientInit(). The program
>>> rastream() changes
>>> this value, and specifies a lot of stuff to do during these
>>> ArgusClientTimeout()
>>> calls, so you can use it as an example, although a complicated one.
>>>
>>> When a record is ready to be processed, the library calls the routine
>>> RaProcessRecord().
>>> You supply what to do in this case. The program ra.c is a great example.
>>>
>>> When each input source is finished, the routine RaArgusInputComplete() is
>>> called.
>>> Most programs don't do anything, but rasort.c is a great example of what
>>> you can
>>> do at the end of each file. rasort() has an option to sort files in
>>> place, which means its
>>> going to sort each input file, output the sorted records into a temporary
>>> file, and then
>>> rename the temporary file to the original filename. We need to do this
>>> as each file is
>>> processed.
>>>
>>> After all the inputs have closed, the argus-client library will call the
>>> routine
>>> RaParseComplete(). Here you want to close everything, write out stuff,
>>> deallocate stuff, whatever you need to do after all the records have been
>>> read in and processed. ra.c has a great "flush data to any output file
>>> that is
>>> interested, and then close them all" strategy that is a minimum.
>>>
>>> Hopefully this will get you started, once you get your rawhatever.c to do
>>> something,
>>> holler if you like, and I'll describe strategies for doing simple and not
>>> so simple clients.
>>>
>>>
>>>
>>> Carter
>>>
>>> ----------begin patch-----------
>>> set:clients carter$ diff -c Makefile.in Makefile.in.new
>>> *** Makefile.in Fri Apr 24 16:53:58 2009
>>> --- Makefile.in.new Wed May 13 18:22:24 2009
>>> ***************
>>> *** 82,88 ****
>>>
>>> SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c racluster.c
>>> rahisto.c ralabel.c \
>>> rapolicy.c ranonymize.c rapath.c rastream.c ratree.c
>>> rafilteraddr.c ratimerange.c \
>>> ! rauserdata.c raservices.c ratemplate.c
>>>
>>> PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/rasort
>>> @INSTALL_BIN@/rasplit \
>>> @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/rastrip
>>> \
>>> --- 82,88 ----
>>>
>>> SRC = ra.c racount.c rasort.c rasplit.c rastrip.c rabins.c racluster.c
>>> rahisto.c ralabel.c \
>>> rapolicy.c ranonymize.c rapath.c rastream.c ratree.c
>>> rafilteraddr.c ratimerange.c \
>>> ! rauserdata.c raservices.c ratemplate.c rawhatever.c
>>>
>>> PROGS = @INSTALL_BIN@/ra @INSTALL_BIN@/racount @INSTALL_BIN@/rasort
>>> @INSTALL_BIN@/rasplit \
>>> @INSTALL_BIN@/rabins @INSTALL_BIN@/racluster @INSTALL_BIN@/rastrip
>>> \
>>> ***************
>>> *** 90,96 ****
>>> @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/rastream
>>> \
>>> @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr @INSTALL_BIN@/ratimerange
>>> \
>>> @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/radark
>>> \
>>> ! @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices @INSTALL_BIN@
>>> /ratemplate
>>>
>>> all: $(PROGS)
>>>
>>> --- 90,97 ----
>>> @INSTALL_BIN@/ranonymize @INSTALL_BIN@/rapath @INSTALL_BIN@/rastream
>>> \
>>> @INSTALL_BIN@/ratree @INSTALL_BIN@/rafilteraddr @INSTALL_BIN@/ratimerange
>>> \
>>> @INSTALL_BIN@/raports @INSTALL_BIN@/rahosts @INSTALL_BIN@/radark
>>> \
>>> ! @INSTALL_BIN@/rauserdata @INSTALL_BIN@/raservices @INSTALL_BIN@/ratemplate
>>> \
>>> ! @INSTALL_BIN@/rawhatever
>>>
>>> all: $(PROGS)
>>>
>>> ***************
>>> *** 163,168 ****
>>> --- 164,172 ----
>>> @INSTALL_BIN@/ratemplate: ratemplate.o $(LIB)
>>> $(CC) $(CCOPT) -o $@ ratemplate.o $(LIB) $(COMPATLIB)
>>>
>>> + @INSTALL_BIN@/rawhatever: rawhatever.o $(LIB)
>>> + $(CC) $(CCOPT) -o $@ rawhatever.o $(LIB) $(COMPATLIB)
>>> +
>>> # We would like to say "OBJ = $(SRC:.c=.o)" but Ultrix's make cannot
>>> # hack the extra indirection
>>>
>>> ----------end patch-----------
>>>
>>>
>>> On May 13, 2009, at 12:31 PM, Harry Bock wrote:
>>>
>>> Hi all,
>>>>
>>>> I was wondering if I could get any guidance on where to start for
>>>> developing a new Argus client. The programming model used in the current
>>>> Argus clients is a little hard to follow, and there doesn't seem to be a
>>>> whole lot of documentation in the source code or on the website for
>>>> developing one.
>>>>
>>>> I have been working on a very similar application for my company, and
>>>> I've realized that Argus does most of what I'm looking for already, and it's
>>>> already usable :) If I could leverage Argus as the core of our application,
>>>> that would save us a lot of development time and headaches, so any hints in
>>>> the right direction would be very much appreciated.
>>>>
>>>> Regards,
>>>> Harry Bock
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Harry Bock
>> Software Developer, Package Maintainer
>> OSHEAN, Inc.
>> Email: harry at oshean.org
>> PGP Key ID: 546CC353
>>
>>
>>
>>
>>
>>
>
>
> --
> Harry Bock
> Software Developer, Package Maintainer
> OSHEAN, Inc.
> Email: harry at oshean.org
> PGP Key ID: 546CC353
>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>
--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry at oshean.org
PGP Key ID: 546CC353
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090522/6e7dbe31/attachment.html>
More information about the argus
mailing list