Argus Client Problems Reading with Record Offsets

[Dave Edelman]

I think that part of the  confusion was related to the man page and usage()
output of ra. I can specify offset as an output field to ra so I just
thought I could feed it back to a subsequent invocation of ra.
That begs the question -  is it possible to add filename as an output field
so that full context is available even when the read is wildcard and
recursive? 

Maybe help file wording similar to this would avoid the confusion.

-r <[type:]file[::ostart[:ostop]] ...>
read <type> data from <file>. '-' denotes stdin. types supported are argus
(default), 'cisco' and 'ft' (flow-tools)
optionally provide starting and ending byte offsets for retrieving data from
the file. Offsets must conform to record boundaries and are only valid for
uncompressed files.


 -r [- | <[type:]file[::soffset[:eoffset]] ...>]
Read  <type>  data  from  <files>  in the order presented on the command
line. '-' denotes stdin.  Ra supports reading argus type data (default),
cisco and ft, flow-tools type data.  If you want to read a set of files and
then, when done, read stdin, use multiple occurrences of the -r option.  Ra
can read gzip(1), bzip2(1) and compress(1) compressed data files.  If you
are working with uncompressed files, you may use the optional byte offset
values to select the range of records that are retrieved. Byte offsets must
conform to record boundaries. 



--Dave


> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Monday, November 26, 2012 5:57 PM
> To: Dave Edelman
> Cc: Argus
> Subject: Re: [ARGUS] Argus Client Problems Reading with Record Offsets
> 
> * PGP - S/MIME Signed by an unverified key: 11/26/2012 at 5:57:12 PM
> 
> Hey Dave,
> rasql() and rasqlinsert() both have specific support for uncompressing
files
> that maybe compressed and indexed in the archive, so using rasql() to read
> the compressed and indexed files should work fine.
> 
>    rasql() -t startTime-endTime
> 
> If the file that contains the data in the time range is compressed, it
will be
> uncompressed at the end of the command.
> 
> If you look at the Filename table that rasqltimeindex() generates, you can
see
> that the ".gz" extension is there, and rasql() will see the extension, and
> decompress the file, before reading the data.  It leaves the file
decompressed
> after its done.  This was designed into rasql(), thinking that if you hit
the file
> once, you would probably hit it again.
> 
> All you need is a cron script that runs through the archive and
recompresses
> any uncompressed files lying around.  I think we have find() scripts that
do
> that pretty well, if you need one.
> 
> This strategy generates problems for read-only compressed archives, as
> rasql() will want to write the uncompressed file into the archive
directory.
> So, the whole process does need a bit of thought.
> 
> If you were to write your own wrapper program, that looked in the database
> for the offsets, you would have to know to decompress the file before
calling
> ra() or whatever, to process the data.  But that should be straight
forward.
> 
> Carter
> 
> 
> On Nov 25, 2012, at 4:51 PM, Dave Edelman <dedelman at iname.com>
> wrote:
> 
> > I've just started to experiment with rasqltimeindex and after a bit of
> > tinkering I have time index information for about three years of flow
> > information files. While I was verifying the accuracy of the data, I
> > discovered two interesting facts:
> >
> > . Argus clients are able to provide accurate starting offsets for
> > records in a compressed archive but they are unable to use the offset
> > values to subsequently retrieve records or record ranges. This is not
> > a problem if the file is uncompressed and then read using the record
> offsets.
> > . A read with a starting and ending offset does not return the final
record.
> >
> > ra -r snk.2012.02.05.22.gz -s +offset -L 10
> >                  StartTime      Flgs  Proto            SrcAddr  Sport
Dir
> > DstAddr  Dport  TotPkts   TotBytes State       Offset
> > Sun 2012-02-05 22:59:59.740 Ne           tcp      xxx.xx.105.63.53890
->
> > xx.xx.70.5.657           1         60   REQ     18887684
> > Sun 2012-02-05 22:59:59.816 Ne           tcp     xxx.xxx.137.13.34313
->
> > xxx.xx.65.222.8000          1         60   REQ     18887792
> > Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2977
->
> > xxx.xx.139.178.80            1         48   REQ     18887900
> > Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2978
->
> > xx.xx.210.24.80            1         48   REQ     18888008
> > Sun 2012-02-05 22:59:59.924 Ne           tcp     xx.xxx.105.212.50657
->
> > xxx.xx.70.5.657           1         60   REQ     18888116
> >
> > ra -r snk.2012.02.05.22.gz::18887900:18888116 -s +offset
> >
> > gunzip snk.2012.02.05.22.gz
> >
> > ra -r snk.2012.02.05.22::18887900:18888116 -s +offset
> >                  StartTime      Flgs  Proto            SrcAddr  Sport
Dir
> > DstAddr  Dport  TotPkts   TotBytes State       Offset
> > Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2977
->
> > xxx.xx.139.178.80            1         48   REQ     18887900
> > Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2978
->
> > xx.xx.210.24.80            1         48   REQ     18888008
> >
> > I still have a bit more tinkering to do on the rasql end of things,
> > I'll keep the list informed of any progress.
> >
> > --Dave
> >
> >
> >
> >
> >
> >
> 
> 
> * Carter Bullard <carter at qosient.com>
> * Issuer: "VeriSign - Unverified