Argus Client Problems Reading with Record Offsets

[Carter Bullard]

Hey Dave,
rasql() and rasqlinsert() both have specific support for uncompressing files
that maybe compressed and indexed in the archive, so using rasql() to read
the compressed and indexed files should work fine.

   rasql() -t startTime-endTime

If the file that contains the data in the time range is compressed, it will
be uncompressed at the end of the command.

If you look at the Filename table that rasqltimeindex() generates, you can
see that the ".gz" extension is there, and rasql() will see the extension, and
decompress the file, before reading the data.  It leaves the file decompressed
after its done.  This was designed into rasql(), thinking that if you hit the file once,
you would probably hit it again.  

All you need is a cron script that runs through the archive and recompresses
any uncompressed files lying around.  I think we have find() scripts that do
that pretty well, if you need one.

This strategy generates problems for read-only compressed archives, as rasql()
will want to write the uncompressed file into the archive directory.  So, the
whole process does need a bit of thought.

If you were to write your own wrapper program, that looked in the database
for the offsets, you would have to know to decompress the file before calling
ra() or whatever, to process the data.  But that should be straight forward.

Carter


On Nov 25, 2012, at 4:51 PM, Dave Edelman <dedelman at iname.com> wrote:

> I've just started to experiment with rasqltimeindex and after a bit of
> tinkering I have time index information for about three years of flow
> information files. While I was verifying the accuracy of the data, I
> discovered two interesting facts:
> 
> . Argus clients are able to provide accurate starting offsets for records in
> a compressed archive but they are unable to use the offset values to
> subsequently retrieve records or record ranges. This is not a problem if the
> file is uncompressed and then read using the record offsets.
> . A read with a starting and ending offset does not return the final record.
> 
> ra -r snk.2012.02.05.22.gz -s +offset -L 10
>                  StartTime      Flgs  Proto            SrcAddr  Sport   Dir
> DstAddr  Dport  TotPkts   TotBytes State       Offset
> Sun 2012-02-05 22:59:59.740 Ne           tcp      xxx.xx.105.63.53890     ->
> xx.xx.70.5.657           1         60   REQ     18887684
> Sun 2012-02-05 22:59:59.816 Ne           tcp     xxx.xxx.137.13.34313     ->
> xxx.xx.65.222.8000          1         60   REQ     18887792
> Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2977      ->
> xxx.xx.139.178.80            1         48   REQ     18887900
> Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2978      ->
> xx.xx.210.24.80            1         48   REQ     18888008
> Sun 2012-02-05 22:59:59.924 Ne           tcp     xx.xxx.105.212.50657     ->
> xxx.xx.70.5.657           1         60   REQ     18888116
> 
> ra -r snk.2012.02.05.22.gz::18887900:18888116 -s +offset
> 
> gunzip snk.2012.02.05.22.gz
> 
> ra -r snk.2012.02.05.22::18887900:18888116 -s +offset
>                  StartTime      Flgs  Proto            SrcAddr  Sport   Dir
> DstAddr  Dport  TotPkts   TotBytes State       Offset
> Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2977      ->
> xxx.xx.139.178.80            1         48   REQ     18887900
> Sun 2012-02-05 22:59:59.860 Ne           tcp     xxx.xx.153.150.2978      ->
> xx.xx.210.24.80            1         48   REQ     18888008
> 
> I still have a bit more tinkering to do on the rasql end of things, I'll
> keep the list informed of any progress. 
> 
> --Dave
> 
> 
> 
> 
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20121126/b3314d50/attachment.bin>