ra as a 'daemon'; filtering by timerange

[Carter Bullard]

Hey Christian,
   Thanks!  Yes, you seem to have discovered a bug.  There should
not be any records with a 0 timestamp.  I'll try to sniff out the
lead on TCP resets to see if we're bzero()'ing out a value we're
not suppose to.

   There have been some problems running ra() as a persistant
daemon due to memory usage.  Ra() tries to be clever about ip
address lookups/decoding and the like, which is not so clever
if its running for a long time.  So if your version seems to
eat too much memory for what its doing, send some mail.

   On my machines, I rotate argus data files every 10 minutes,
and that gives me a bit better performance when I'm looking for
something and I know what the time range is, and it also makes
the file sizes a bit more manageable.  It maybe that this type
of strategy could help you in excluding records.

   Because ra() appends output records to output files,
it is easy to build a single output file from multiple
input files, or multiple passes of ra().  If you do use
multiple small files, you can use ra() to generate a daily file
from the smaller files your collecting.  At the end of the day:

   ra -r *.data -w daily.log

will create your single daily file.  With some regular expression
magic and an intelligent chunk naming convention, you could
build the two files your interested in at the end of the day.

Or running two passes of ra, writing to the same output file
can generate your file for you:

   ra -r argus.out -t 00-01 -w argus.1
   ra -r argus.out -t 07-23:59:59 -w argus.1

should also do the trick.

   OK, your time range selections.  We should support the time
24:00:00, or at least allow you to get over midnight, so I'll
look at that. I have not put a lot of energy in the time filters,
so if we can come up with a behavior that we like, I can add
it to the ra() -t option.

Hope this helps,

Carter

Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 18K
New York, New York  10022

carter at qosient.com
Phone +1 212 588-9133
Fax   +1 212 588-9134
http://qosient.com



> -----Original Message-----
> From: owner-argus-info at lists.andrew.cmu.edu 
> [mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of 
> Christian Martin
> Sent: Tuesday, November 27, 2001 3:52 AM
> To: argus-info at lists.andrew.cmu.edu
> Subject: ra as a 'daemon'; filtering by timerange
> 
> 
> First, congratulations and thanks to all involved for an excellent and
> well-featured product.  I have been looking for something 
> like argus for
> some time, and it (nearly :) does everything I need and more.
> 
> Apologies if this is off-topic; please feel free to flame me 
> if it's not
> appropriate for the forum.
> 
> I have a question regarding the use of ra on a linux system.  
> I am running
> ra [v2.0.1] as a quasi-daemon on a logging server, dumping 
> records from a
> remote argus server to a file.  A logrotate script runs at midnight,
> effectively chopping my argus records into compressed 24-hour 
> chunks.  I'm
> keeping traffic logs on a fairly long-term basis, and there's 
> a fair amount
> of traffic on the network (argus is pumping out about 800Mb a day,
> uncompressed), and have found per-date argus files to be the 
> most manageable
> solution.
> 
> So far, so good.  However, I'm investigating the use of argus 
> to charge
> users according to network use.  Our upstream provider does not charge
> during the hours of 1:00am-6:00am, so we can't pass on a 
> charge during these
> hours.  Is there a way of excluding a certain time range from 
> an ra enquiry?
> At the moment, I'm using separate ra commands thus:
> 
> ra -r argus.out -t 00-01 -w argus.1
> ra -r argus.out -t 07-23:59:59 -w argus.2
> 
> ... and passing both result sets to an raclient tool for 
> analysis.  This
> seems rather clunky and inelegant: is there a better way of 
> doing the job?
> It would also be handy to capture the extra second between 
> 11:59:59pm and
> midnight...  Ideally, I would like to be able to divide 
> network sessions
> which cross my time boundary, allowing me to charge only for 
> the chargeable
> portion.  Perhaps it would be better to run three instances 
> of the 'daemon',
> writing to three separate files - one filtered to '-t 
> 00:00-01:00', one to
> '-t 01:00-07:00' and one to '-t 07:00-23:59:59' - and then 
> glue the files
> together at the end of the day.  Any suggestions?
> 
> Also, I'm seeing quite a few instances of a zero dtime - 01 Jan 70
> 01:00:00 - usually where TCP transactions have been reset.  
> Is this a bug?
> 
> Incidentally, if anyone new to argus is reading the archive 
> and wants a copy
> of my various config files, I'd be happy to oblige.  My argus server
> (actually a firewall) runs a 2.4 Linux kernel, and the 
> logging server (that
> is, the argus client) runs Debian linux.  I'm not sure if 
> running ra as a
> quasi-daemon is the best solution for a logging server - 
> other suggestions
> welcome - but it has been doing the job for weeks without a glitch...
> 
> Many thanks in advance,
> 
> Christian
> 
> --
> Christian Martin
> IT Department
> Jesus College, Cambridge
> e-mail: c.martin at jesus.cam.ac.uk
> telephone: +44-1223-339944
> 
> 
> ---------- End Forwarded Message ----------
> 
> 
> 
> ---
> Jeffrey T. Eaton                    esp              
> jeaton+ at andrew.cmu.edu
> Research Systems Programmer                      
> Carnegie-Mellon University
> 
>