FlowScan w/argus?

[Carter Bullard]

Hey Dave,
   So many concepts in one piece of mail.  I believe that
FlowScan could easily use argus data, although you may
have to change a few things to use the sematics that
argus data provides that netflow doesn't.

   As Mark mentioned in his e-mail, we're doing a lot of
cricket based graphing of real-time stats using argus
data at CMU, and the cricket part was pretty painful.
So interfacing to FlowScan may be a better ticket.

   The argus way of things should make it pretty trivial
for you to generate your 5 minute batches and process them
with perl.  ra() is very flexible allowing you to specify
whatever output format you want, so perl processing is
really simple if you use ra() to read the data.

   In terms of writing programs to use argus data,
there are lots of things that can be done.  Many people
on the list have their own Perl scripts and programs for
parsing argus data, and Russell's scan detector scripts
are just one example.

   If you want to do perl, you may find that the output
of raxml() will be the easiest to parse all the
optional fields that argus supports.  I believe that
perl has an xml parser that's pretty good, and it may be
that that approach would provide the fastest way of
doing the perl thing.

   If you want to look at writing your own argus data
parsing programs, use the ratemplate.c in the ./clients
directory.  All you have to do is supply the appropriate
routines, and something like racount() is a simple example,
and all the ugly stuff is handled by the libraries.  That
way you can get near-real time argus server access, multi-
probe collecting and even netflow conversion for free.

   Performance with the free stuff seems to be good 
on the type of hardware you mention.  Hundreds of megabits,
100K packets per second, around 20-30K records per second
peak.  We do rather well with DDOS attacks, at least we
don't stop running.

   If you want to do some integration work, just send me
mail, and we can try to figure out what is required.

Carter

Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 18K
New York, New York  10022

carter at qosient.com
Phone +1 212 588-9133
Fax   +1 212 588-9134
http://qosient.com



> -----Original Message-----
> From: owner-argus-info at lists.andrew.cmu.edu 
> [mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of 
> Dave Plonka
> Sent: Wednesday, January 09, 2002 11:38 AM
> To: argus-info at lists.andrew.cmu.edu
> Subject: FlowScan w/argus?
> 
> 
> Carter, et al.
> 
> I've been playing with argus for a few hours, and I'm 
> somewhat enthusiastic about the possibility of getting 
> FlowScan working with it.  It might not be much work, which 
> is always a bonus.
> 
> Current FlowScan users all use cflowd, lfapd, or flow-tools 
> as their flow collector. BTW, FlowScan is here for those that 
> don't know:
> 
>    http://net.doit.wisc.edu/~plonka/FlowScan/
> 
> and sample output is here:
> 
>       http://wwwstats.net.wisc.edu/
> 
> Basically, FlowScan would have two primary requirements of 
> argus as the
> "collector":
> 
>  1) The collector must produce time-stamped raw flow files at five
>     minute intervals.  I've got that bit working.
> 
>  2) My Cflow perl module needs to be able to read the "raw flow files"
>     produced by the collecotr, i.e. the same files that argus' ra(1)
>     reads.  Cflow is here:
>        http://net.doit.wisc.edu/~plonka/Cflow/
> 
> So, I have a few probing questions at this point:
> 
>  * Has anyone already written a perl API to access records in argus'
>    output files, like ra(1)?  (The perl stuff in argus-2.0.4
>    distribution doesn't seem to do that, unless I'm mistaken.)
> 
>  * If not, is anyone really familiar with the Argus API and interested
>    in using FlowScan w/argus?  I thought I'd ask in case someone wants
>    to help writing or review/test a patch to Cflow.
> 
>    I've poked around in argus source a bit...
>    It looks like most of what I want is in "common/argus_parse.c".
>    This looks like the important bit for reading from files:
> 
>                   if (((ArgusReadConnection (addr, 
> addr->filename)) >= 0)) {
>                      ArgusRemoteFDs[0] = addr;
>                      ArgusReadStream();
>                      close(addr->fd);
>                   }
> 
>    The Cflow perl API calls a perl function as the read-loop visits
>    each flow in the file.  Maybe I can just call ArgusReadStream
>    directly (from the perl code) and have each of the
>    "process_whatever" callbacks callout to my perl subroutine.
> 
>  * What kind of throughput can argus typically handle?  Like on a
>    500MHz or 1GHz PIII?  Tens, hundreds of megabits/flows/pkts per
>    second?
> 
>  * What happens when argus tries to produce flows for DoS floods that
>    use forged IP address or random port numbers?  We sometimes see
>    these at rates of 5,000-20,000 packets per second.  (I'm not
>    expecting it necessarily to handle this gracefully (other flow
>    systems don't), I could just use some info from real experience to
>    set expectations.
> 
> So, I'm evaluating whether or not I should do the work to 
> make FlowScan compatible with argus.  It certainly would be a 
> boon to FlowScan since no router flow-export features would 
> be required.  Also, would increase the user-base since 
> there's a lot of SAs out there who don't do the router stuff, 
> but like colorful graphs.
> 
> Thanks for any info,
> Dave
> 
> -- 
> plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka  
> ARS:N9HZF  Madison, WI
> 
>