Rasplit compression support

[Carter Bullard]

Hey Matt,
A future work is to integrate compression for all clients writing  
argus data,
so that should be in argus-3.0.4+.

But if you are running rasplit() as a daemon, reading argus data from  
argi or a radium()
collecting from a set of argi, and you would like to compress the file  
after
you know that all the data has arrived, you should use the program  
rastream().

This is still under development, but it is very stable so far, so you  
should be able to
use it in a production environment.  I need to update the  
documentation, when we
are all happy that it does what its designed to do.

rastream() differs from rasplit() only in that rastream() will run a  
script against an output
file after it closes the file.  When using time based spliting,  
rastream() needs to be told
how long to wait after the time boundary elapses before it closes the  
output file and
runs the script.  You specify the wait time using the "-B secs" option.

The value for the -B option depends on how well your argus probes are  
in sync, and how
often they generate status records.  You need to wait long enough for  
all the records that
need to go into the file arrive.  For me, all my probes generate  
status records every 5 seconds,
and I keep them in sync, so my rastream() -B option can be around  
15s.  30s is not a bad
number.

If you are reading netflow records, then you can forget about this  
scheme, as the time
delay for receiving a specific record can be, what, an hour?  While  
rastream()  can
handle a "-B 3600s" directive, its not a good solution, and I don't  
recommend it.

The script is specified using the "-f /path/script.sh" command line  
option.  Any script will do.
rastream() calls the script like this "foptionprovided -r /full/path/ 
name/to/output/file ",
so be sure and call rastream() with the complete pathname for your  
script, and the
script you provide needs to be able to support a "-r /path/to/argus/ 
data/file" option.

There is a sample rastream.sh in the ./support/Config directory.  All  
it does is compress
the file.  You can do anything you like, but test it out for a while.   
rastream() queues
script requests, so there is only one request outstanding at a time,  
so if you have
dozens of files that close at the same time, rastream() will handle  
the load pretty
well.  There is error reporting, so you can discovery that things are  
not right through
syslog().

The manpage for rastream() is just a copy of rasplit.1, so it needs to  
be updated to be
of any use, sorry!!!

If you run rasplit() as a daemon, then running rastream() is trivial.   
This is how I run it:

     rastream -S argus.source -M time 5m -B 30s -f /path/to/my/script - 
w /data/archive/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S -d

For some, all you need to do is add "-B 30s -f /path/to/your/script"  
to your call to rasplit()
and all should work.

If you have any luck, send email!!!!!

Carter



On Nov 4, 2009, at 1:52 PM, Matt Sheridan wrote:

> Hi Carter, all –
>
> I have been impressed with the argus product, thank you for your work.
>
> One consideration I had - possibly for product enhancement - would  
> be for rasplit to incorporate a feature/flag for compressing a split  
> file when it is written.  The functionality of %variable%  
> directories is excellent, but would be even better served if  
> compression was implemented immediately to a split file. It seems  
> difficult to use the variable directories, when they become  
> intricate – and the files need to be found and compressed. Obviously  
> a solution can be scripted – but many of us are not terribly  
> proficient there ;).
>
> Any thoughts on such a feature, or is there something already in  
> place I can use?
>
> Thanks for your help.
>
> Matt

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20091104/1be0abc6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20091104/1be0abc6/attachment.bin>