Perl contrib scripts
Peter Van Epp
vanepp at sfu.ca
Thu Oct 12 11:45:58 EDT 2000
This looks great to me. Coincidentally I've just been poking at mine
(because they ran out of memory on a 512 meg machine) and am looking to use
an intermediate file to hold (and sort externally) the data which should mean
that we can get data from long spreads of time (by sorting by IP address in
the external file and using fseek to be able to extract only data for a
particular IP address).
Peter Van Epp / Operations and Technical Support
Simon Fraser University, Burnaby, B.C. Canada
>
> HI All,
> With the imminent release of argus-2.0 I thought it would be a
> good time to start polishing up some of my perl scripts for the contrib
> directory. One reason I have not done this before is that most of my
> scripts have lots of site dependent stuff scattered through them and
> are thus not that 'portable'. I was forced to at least collect all the
> site dependent bits of watcher (my scan detector) into one place when I
> gave Neil a copy and this started me thinking about the "right" way to
> do it.
>
> Well this is what I have come up with:
>
> Argus.pm defines a bunch of global variable that are likely to be used
> by many scripts (things like your local domain, IP range etc. You can
> also define sub packages for things specific to various scripts. This
> becomes the configuration file for all argus scripts in the contrib
> directory.
>
> Another site dependency is how we layout our archives of argus data so
> I have another module Argus::Archive.pm which currently has a single
> function Get_file_list which takes either a start and finish date or a
> time period (eg. today, yesterday, thisweek, lastweek etc) and returns a
> list of files (in date order) to be feed to a client. Each site will
> need to mung this routine to do the right thing with their directory
> layout.
>
> I think it might be useful to include some templates for unpack to aid
> extracting data from various client output. (I have not done this yet).
> Another possibility is a module that hides RA altogether
>
> $input = $Argus::RA->new (Read=>$file, Opt=>$switches ) or die ...
>
> while( $input->read ) {
> }
>
> read would read the pipe from ra and then unpack the record (exactly
> how would depend on the switches supplied to ra). How best to return
> results? via a list or global variables?
>
> Lastly I have a very simple script that runs ra over a range of files
> with user supplied filters as an example of the system in action. I'll
> convert my watcher script to use the new scheme soon and post that when
> I have it done.
>
> I would appreciate some feedback on
> a/ is this a good approach?
> b/ if so then are there issues with the details of how I have done it.
> c/ alernative approaches
>
> I'm not wedded to any of this so please feel free to criticize ;-)
>
> Russell
>
> --------------------------------------------------------------------
>
> Sample Argus.pm (for UoA ;-)
>
> package Argus;
>
> use vars qw ($Client_bin $RA $Local_IP_re $LocalTimeZone_re $Testing
> $Local_domain_re $Def_Host $Def_Port $Archive_root $Error
> @Errors);
>
> $Testing = 0 unless defined $Testing;
>
> $Client_bin = '/home/argus/bin'; # set by configure ??
> $RA = "$Client_bin/ra";
>
> $Local_IP_re = '^130\.216\.';
> $LocalTimeZone_re = '\.nz$';
> $Local_domain_re = 'auckland\.ac\.nz$';
> $Def_Host = 'localhost';
> $Def_Port = '561';
> $Data_Root = '/home/argus/data';
>
> sub Report { # stores errors in a stack, with the last in $Error
> $Error = shift;
> push( @Errors, $Error);
> }
>
> package Argus::Watcher; # config vars for particular scripts
>
> use vars qw ($Def_ra_Filter %BUSY_MACHINES $Period $ReReport
> $Local_Interest_Threshold $Remote_Interest_Threshold
> $Packet_Save);
>
> $Def_ra_Filter = '';
>
> # local machines that talk to lots of others which need higher
> reporting
> # thresholds eg. network monitors web proxies, mail servers etc.
> # The number is the modified reporting threshold.
>
> %BUSY_MACHINES = (
> "130.216.1.1" => 250, # net
> "130.216.1.4" => 450, # mailhost
> "130.216.191.4" => 360,
> "130.216.1.6" => 400, # scream
> "130.216.1.240" => 400,# surveyor
> "130.216.191.46" => 550, # netacct
> "130.216.191.26" => 350,
> "130.216.35.100" => 600,#ec gateway
> "130.216.191.53" => 200,#ec gateway
> "130.216.35.201" => 250,#ec gateway
> "130.216.34.10" => 250,
> "130.216.193.45" => 700, # nix.tmk.auckland.ac
> );
>
> #Default reporting threshold etc.
>
> $Period = $Argus::Testing ? 100 : 300; ## seconds -- reporting period
>
> $ReReport = 3600; ## seconds - wait this long before sending more
> # reports of continuing activity
>
> # report when machine talks to more than this many port/addresses
> # during $Period seconds
>
> $LOCAL_INTEREST_THRESHOLD = $testing ? 10 : 100;
> $REMOTE_INTEREST_THRESHOLD = $testing ? 20 : 40;
>
> $Packet_Save = 20; ## Only save this many sample packets
>
> -------------------------------------------------------------------------------
> Argus/Archive.pm again specific to UoA
>
> package Argus::Archive;
>
> use strict;
> use Argus;
> use Date::Manip;
>
> Date_Init("DateFormat = NONUS"); #change to US for mm/dd
>
> # Takes two dates or a special period names and returns a list of
> # argus archive files that cover the period
>
> sub Get_File_List {
> my ($first, $last ) = @_;
> my @files;
>
> my ( $sdate, $fdate);
>
> my $today = ParseDate('today');
>
> if( ! defined $first ) {
> Argus::Report("No start date given");
> return undef;
> }
>
> if ( $first eq 'thisweek' ){
> $sdate = Date_GetPrev($today, 'Mon', 1);
> $fdate = Date_GetNext($today, 'Mon', 0);
> } elsif( $first eq 'thismonth' ){
> $sdate = ParseDate('1st');
> $fdate = ParseDate('today') ;
> $fdate = Date_SetTime($fdate, '23:59:59');
> } elsif( $first eq 'lastweek' ){
> $fdate = Date_GetPrev($today, 'Mon', 1);
> $sdate = Date_GetPrev($fdate, 'Mon', 0);
> } elsif( $first eq 'lastmonth' ){
> $fdate = ParseDate('1st');
> $sdate = DateCalc($fdate, '- 1 month');
> } else {
> $sdate =ParseDate($first);
> Argus::Report( "Invalid date '$first'") if ! $sdate;
>
> if( defined $last ) {
> $fdate =ParseDate($last);
> Argus::Report( "Invalid finish date '$last'") if ! $fdate;
> } else {
> $fdate = $sdate;
> }
> }
>
> for (my $date=Date_SetTime($sdate, '00:00:00');
> $date le $fdate; $date=DateCalc($date, '+ 1 day')) {
>
> my $name =UnixDate($date, "%Y.%m.%d" );
>
> if( -d "$Argus::Data_Root/$name" ) { # it was a directory
> if( opendir(DIR, "$Argus::Data_Root/$name") ) {
> foreach my $f ( sort grep(/^argus.*\.gz$/,
> readdir(DIR))){
> push(@files, "$Argus::Data_Root/$name/$f");
> }
> closedir(DIR);
> } else {
> Argus::Report ("Failed to open ".
> $Argus::Data_Root/$name:$!");
> return undef;
> }
>
> }
> }
> return @files;
> }
> 1;
> ------------------------------------------------------------------------
>
> My look_for script modified to use the module files (I guess we all
> have one of these ;-)
>
> #!/usr/bin/perl -w
>
> #
> # usage look_for [<options>] files or directories....
> #
>
> use lib "/home/argus/bin"; # set by configure ????
>
> use strict;
> use Getopt::Std;
> use Argus;
> use Argus::Archive;
>
> use vars qw($opt_f $opt_F $opt_s $opt_D $opt_o $opt_q $opt_P $opt_w
> $opt_t);
>
> getopts("D:s:f:o:F:qw:t:P:") || die "Invalid options found";
>
> # D <directory>-- Directory where data is held
> # s <date> -- Start date
> # f <date> -- Finish date
> # F <file> -- read filter from <file>
> # o <options> -- options for ra (default -nc)
> # P <file> -- function to post filter
> # w <filename> -- Output to filename
> # q -- quite mode
> # t <timezone> -- print times relative to alternate timezone eg UTC
>
> my $opts = defined $opt_o ? "-$opt_o" : '-znc';
>
> $Argus::Data_Root = $opt_D if defined $opt_D;
>
> my $filter = defined $opt_F ? "-F $opt_F" : join(' ', @ARGV);
>
> if (defined $opt_w) {
> open(OUT, ">$opt_w") || die "failed to open '$opt_w':$!";
> } else {
> open(OUT, ">-") || die "failed to open 'stdout':$!";
> }
>
> if( defined $opt_t ) {
> $ENV{TZ} = $opt_t ;
> }
>
> $opt_s = 'yesterday' unless defined $opt_s;
>
> my @files = Argus::Archive::Get_File_List($opt_s, $opt_f );
>
> if( ! defined @files ) {
> die "$Argus::Error";
> }
>
> foreach my $fn ( @files ) {
>
> print "$fn\n" if ! defined $opt_q;
>
> open(SW, "$Argus::RA -r $fn $opts $filter |" ) ||
> die "can't open ra:$!";
>
> while (<SW>) {
> if( defined $opt_P ) {
> next if post_filter($_);
> }
> print OUT;
> }
> }
>
>
>
>
More information about the argus
mailing list