FlowScan w/argus?

[Dave Plonka]

On Wed, Jan 09, 2002 at 12:55:47PM -0500, Carter Bullard wrote:
> Hey Dave,
>    So many concepts in one piece of mail.  I believe that
> FlowScan could easily use argus data, although you may
> have to change a few things to use the sematics that
> argus data provides that netflow doesn't.
> 
>    As Mark mentioned in his e-mail, we're doing a lot of
> cricket based graphing of real-time stats using argus
> data at CMU, and the cricket part was pretty painful.
> So interfacing to FlowScan may be a better ticket.

Yes thanks Mark for the (out-of-band) response.  The graphs look
good...  If you don't do them already, adding pkts/sec and flows/sec
graphs is really useful to identify anomalies such as DoS attacks.

>    The argus way of things should make it pretty trivial
> for you to generate your 5 minute batches and process them
> with perl.  ra() is very flexible allowing you to specify
> whatever output format you want, so perl processing is
> really simple if you use ra() to read the data.

I know that will be to costly performance-wise to convert to ascii and
back, so I'll want to read the raw binary file directly.

I could have been more clear in characterizing how the Cflow perl
module would work with argus files.  "Cflow.pm" is an XSUB extension to
perl written in C.  The "ratemplate.c" you mention below may be
instructive, but for FlowScan, I can't give up control to the main()
that is embedded in "argus_parse.c", hence the need for me to learn
some of the argus internal API, such as ArgusReadStream.

Ideally argus would have a "mainloop" function instead of an embedded
main().  (Forgive me if I overlooked something - as I said, I've just
had a cursory look at the code.)  The way this is done with FlowScan is
that you essentially pass function pointers to a "looping" function.
In Cflow, that "looping" subroutine is called "find" and you pass it a
reference to a "wanted" subroutine that gets called once per flow, and
an optional "perfile" subroutine that gets called once per file, as in
the SYNOPSIS here:

   http://net.doit.wisc.edu/~plonka/Cflow/Cflow.html

If I do end up patching Cflow to read argus files, I'll try to
introduce a reasonable C API so that that "looping" function can
possibly be patched into the argus source in case any future users need
to write their own main().

>    In terms of writing programs to use argus data,
> there are lots of things that can be done.  Many people
> on the list have their own Perl scripts and programs for
> parsing argus data, and Russell's scan detector scripts
> are just one example.
> 
>    If you want to do perl, you may find that the output
> of raxml() will be the easiest to parse all the
> optional fields that argus supports.  I believe that
> perl has an xml parser that's pretty good, and it may be
> that that approach would provide the fastest way of
> doing the perl thing.

Yes, I've had good luck with the perl XML::Simple module.
Fast to code for it, slow to run though.
 
>    If you want to look at writing your own argus data
> parsing programs, use the ratemplate.c in the ./clients
> directory.  All you have to do is supply the appropriate
> routines, and something like racount() is a simple example,
> and all the ugly stuff is handled by the libraries.  That
> way you can get near-real time argus server access, multi-
> probe collecting and even netflow conversion for free.
> 
>    Performance with the free stuff seems to be good 
> on the type of hardware you mention.  Hundreds of megabits,
> 100K packets per second, around 20-30K records per second
> peak.  We do rather well with DDOS attacks, at least we
> don't stop running.

Great!

>    If you want to do some integration work, just send me
> mail, and we can try to figure out what is required.

OK, I'll keep you posted.  I haven't decided how to prioritize this...
It sounds like more fun than some of the maintenance/documentation work
on deck for the next FlowScan release, but one thing that's nice about
doing freely-available stuff is that, sometimes, you get to do what
_you_ want to do. ;^)

Thanks for the info!
Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka  ARS:N9HZF  Madison, WI