Perl contrib scripts

[Russell Fulton]

HI All,
	With the imminent release of argus-2.0 I thought it would be a 
good time to start polishing up some of my perl scripts for the contrib 
directory.  One reason I have not done this before is that most of my 
scripts have lots of site dependent stuff scattered through them and 
are thus not that 'portable'.  I was forced to at least collect all the 
site dependent bits of watcher (my scan detector) into one place when I 
gave Neil a copy and this started me thinking about the "right" way to 
do it.

Well this is what I have come up with: 

Argus.pm defines a bunch of global variable that are likely to be used 
by many scripts (things like your local domain, IP range etc.  You can 
also define sub packages for things specific to various scripts.  This 
becomes the configuration file for all argus scripts in the contrib 
directory.

Another site dependency is how we layout our archives of argus data so 
I have another module Argus::Archive.pm which currently has a single 
function Get_file_list which takes either a start and finish date or a 
time period (eg. today, yesterday, thisweek, lastweek etc) and returns a 
list of files (in date order) to be feed to a client. Each site will 
need to mung this routine to do the right thing with their directory 
layout.

I think it might be useful to include some templates for unpack to aid 
extracting data from various client output. (I have not done this yet).
Another possibility is a module that hides RA altogether

$input = $Argus::RA->new (Read=>$file, Opt=>$switches ) or die ...

while( $input->read ) {
}

read would read the pipe from ra and then unpack the record (exactly 
how would depend on the switches supplied to ra).  How best to return 
results? via a list or global variables?

Lastly I have a very simple script that runs ra over a range of files 
with user supplied filters as an example of the system in action.  I'll 
convert my watcher script to use the new scheme soon and post that when 
I have it done.

I would appreciate some feedback on 
a/ is this a good approach?
b/ if so then are there issues with the details of how I have done it.
c/ alernative approaches

I'm not wedded to any of this so please feel free to criticize ;-)

Russell

--------------------------------------------------------------------

Sample Argus.pm  (for UoA ;-)

package Argus;

use vars qw ($Client_bin $RA $Local_IP_re $LocalTimeZone_re $Testing
	     $Local_domain_re $Def_Host $Def_Port $Archive_root $Error
	     @Errors);

$Testing = 0 unless defined $Testing;

$Client_bin = '/home/argus/bin';  # set by configure ??
$RA = "$Client_bin/ra";

$Local_IP_re = '^130\.216\.';
$LocalTimeZone_re = '\.nz$'; 
$Local_domain_re = 'auckland\.ac\.nz$';
$Def_Host = 'localhost';
$Def_Port =  '561';
$Data_Root = '/home/argus/data';

sub Report {  # stores errors in a stack, with the last in $Error
    $Error = shift;
    push( @Errors, $Error);
}

package Argus::Watcher;   # config vars for particular scripts

use vars qw ($Def_ra_Filter %BUSY_MACHINES $Period $ReReport
	     $Local_Interest_Threshold $Remote_Interest_Threshold
	     $Packet_Save);

$Def_ra_Filter = '';

# local machines that talk to lots of others which need higher 
reporting 
# thresholds eg. network monitors  web proxies, mail servers etc.  
# The number is the modified reporting threshold.

%BUSY_MACHINES = (
		  "130.216.1.1" => 250,  # net
		  "130.216.1.4" => 450,  # mailhost
		  "130.216.191.4" => 360,
		  "130.216.1.6" => 400,  # scream
		  "130.216.1.240" => 400,# surveyor
		  "130.216.191.46" => 550, # netacct
		  "130.216.191.26" => 350,
		  "130.216.35.100" => 600,#ec gateway
		  "130.216.191.53" => 200,#ec gateway
		  "130.216.35.201" => 250,#ec gateway
		  "130.216.34.10" => 250,
		  "130.216.193.45" => 700,  # nix.tmk.auckland.ac
		  );

#Default reporting threshold etc.

$Period = $Argus::Testing ? 100 : 300;  ## seconds -- reporting period

$ReReport = 3600;  ## seconds - wait this long before sending more 
                   #            reports of continuing activity

# report when machine talks to more than this many port/addresses 
# during $Period seconds

$LOCAL_INTEREST_THRESHOLD = $testing ? 10 : 100;
$REMOTE_INTEREST_THRESHOLD = $testing ? 20 : 40;

$Packet_Save = 20;	## Only save this many sample packets

-------------------------------------------------------------------------------
Argus/Archive.pm  again specific to UoA

package Argus::Archive;

use strict;
use Argus;
use Date::Manip;

Date_Init("DateFormat = NONUS");  #change to US for mm/dd

# Takes two dates or a special period names and returns a list of
# argus archive files that cover the period

sub Get_File_List {
    my ($first, $last ) = @_;
    my @files;

    my ( $sdate, $fdate);

    my $today = ParseDate('today');

    if( ! defined $first ) {
	Argus::Report("No start date given");
	return undef;
    }

    if ( $first eq 'thisweek' ){
	$sdate = Date_GetPrev($today, 'Mon', 1);
	$fdate =  Date_GetNext($today, 'Mon', 0);
    } elsif(  $first eq 'thismonth' ){
	$sdate = ParseDate('1st');
	$fdate = ParseDate('today') ;
	$fdate = Date_SetTime($fdate, '23:59:59');
    } elsif(  $first eq 'lastweek' ){
	$fdate = Date_GetPrev($today, 'Mon', 1);
	$sdate =  Date_GetPrev($fdate, 'Mon', 0);
    } elsif(  $first eq 'lastmonth' ){
	$fdate = ParseDate('1st');
	$sdate =  DateCalc($fdate, '- 1 month');
    } else {
	$sdate =ParseDate($first);
	Argus::Report( "Invalid date '$first'") if ! $sdate;

	if( defined $last ) {
	    $fdate =ParseDate($last);
	    Argus::Report( "Invalid finish date '$last'") if ! $fdate;
	} else {
	    $fdate = $sdate;
	}
    }

    for (my $date=Date_SetTime($sdate, '00:00:00');
	 $date le $fdate; $date=DateCalc($date, '+ 1 day')) {

	my $name =UnixDate($date,  "%Y.%m.%d" );
	
	if( -d "$Argus::Data_Root/$name" ) {  # it was a directory
	    if( opendir(DIR, "$Argus::Data_Root/$name") ) {
		foreach my $f ( sort grep(/^argus.*\.gz$/, 
                              readdir(DIR))){
		    push(@files, "$Argus::Data_Root/$name/$f");
		}
		closedir(DIR);
	    } else {
		Argus::Report ("Failed to open ". 
                                $Argus::Data_Root/$name:$!");
		return undef;
	    }

	}
    }
    return @files;
}
1;
------------------------------------------------------------------------

My look_for script modified to use the module files (I guess we all 
have one of these ;-)

#!/usr/bin/perl -w
 
#
#  usage look_for [<options>] files or directories....
#

use lib "/home/argus/bin";  # set by configure ????

use strict;
use Getopt::Std;
use Argus;
use Argus::Archive;

use vars qw($opt_f $opt_F $opt_s $opt_D $opt_o  $opt_q $opt_P $opt_w 
$opt_t);

getopts("D:s:f:o:F:qw:t:P:") || die "Invalid options found";

#  D <directory>-- Directory where data is held
#  s <date>     -- Start date
#  f <date>     -- Finish date
#  F <file>     -- read filter from <file>
#  o <options>  -- options for ra (default -nc)
#  P <file>     -- function to post filter
#  w <filename> -- Output to filename
#  q            -- quite mode
#  t <timezone> -- print times relative to alternate timezone eg UTC

my $opts = defined $opt_o ? "-$opt_o" : '-znc';

$Argus::Data_Root = $opt_D if defined $opt_D;  

my $filter = defined $opt_F ? "-F $opt_F" : join(' ', @ARGV);

if (defined $opt_w) {
    open(OUT, ">$opt_w") || die "failed to open '$opt_w':$!";
} else {
    open(OUT, ">-") || die "failed to open 'stdout':$!";
}

if( defined $opt_t ) {
    $ENV{TZ} = $opt_t ;
}

$opt_s = 'yesterday' unless defined $opt_s;

my @files = Argus::Archive::Get_File_List($opt_s, $opt_f );

if( ! defined @files ) {
    die "$Argus::Error";
}

foreach my $fn ( @files ) {

    print "$fn\n" if ! defined $opt_q;

    open(SW, "$Argus::RA -r $fn $opts $filter |" ) ||
	    die "can't open ra:$!";

    while (<SW>) {
	if( defined $opt_P ) {
	    next if post_filter($_);
	}
	print OUT;
    }
}