Best way to grab summary data from last generated file by rastream

[Carter Bullard]

Hey Jesse, Paul,
I understand the statement, "undocumented -f option to rastream", but the option is in the manpage rastream.1.

SYNOPSIS
       rastream [[-M splitmode] [splitmode options]] [-f file processing script -B secs] [raoptions]

Now I will admit that the "file processing script" isn't well described, but undocumented?
I do take exception ;O)

OK, this is the text that is in the updated manpage, that is not yet in the distribution:

       -f program
           Post processing program. rastream, will execute this program / shell script just after
           closing  the output file, passing the full path to the closed output file as a parame-
           ter, using this convention:

              program -r /full/path/to/closed/file

           This allows you to post-process the output file in an automated fashion.

           Generally, this program can do anything you like, such as aggregating  and  correcting
           flow  records,  labeling  records  for semantic enhancement, indexing the files, using
           programs like rasqltimeindex(), and compressing the files.  Traditionally, the program
           has  been  a shell-script, perl program, or php script, so that it can be easily modi-
           fied, on the fly, but it can be any executable  that  can  handle  the  "-r  filename"
           parameter  convention.   The  program should provides its own accountability and error
           logging, so that you know that things are working as you expect.

           rastream must have a path to the program, the program must be executable, and rastream
           must have permission to run the program for this strategy to be successful.

           An example rastream.sh is provided in the ./support/Config directory.

So, Paul, good you're using rastream().   Take a look at the example script, rastream.sh,
in ./support/Config, and modify it to do what you want.  If you have any problems, grab the
latest clients from the development site to see if that fixes your version of rastream(),
and if not holler on the list.

Remember, the " -B secs " option will cause rastream() to reject flows that are older
than " theCurrentTime - Bsecs ", so be sure and chose a " -B secs " value that makes
sense for your flow data sources.  

If you're using netflow, don't use this mechanism, as netflow sources will send records
that are older, and this rule will toss records, regardless of what time you set it for  !!!

Hope all is most excellent,

Carter

On May 7, 2013, at 10:53 AM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Hi Paul,
> 
> You're looking for the (as best as I can tell by looking at -h and the qosient man pages) undocumented '-f' option to rastream...With -f, rastream will execute the specified script providing the current filename as an argument...For example:
> 
> /usr/local/bin/rastream -M time 5m -B 10s -S ${OUR_IP}:${OUR_PORT} -w /nsm/argus/data/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S -f /usr/local/bin/argus_postprocess.bash -d
> 
> This will wait 10 seconds after the 5 minute mark and then execute /usr/local/bin/argus_postprocess.bash with an argument of the filename...
> 
> There's a sample script in ./support/Config/rastream.sh 
> 
> You might also check out http://nsmwiki.org/Argus , as this is one of the only wiki's I know dedicated to argus (although some sections of it could use updating)... :)
> 
> Does that help?
> 
> Cheers,
> 
> Jesse
> 
> 
> 
> 
> On Tue, May 7, 2013 at 9:52 AM, Paul Halliday <paul.halliday at gmail.com> wrote:
> I have rastream processing on hard 5 minute boundaries and I would like to create summary data after it closes each file. 
> 
> flow-capture had a nice option that would let you call an external program after it finished spooling a file; do I have an option like this with argus?
> 
> I can script it, just curious if there is something built-in.
> 
> thanks.
> 
> -- 
> Paul Halliday
> http://www.pintumbler.org/
> 
> 
> 
> -- 
> Jesse Bowling
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130507/18d026cb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130507/18d026cb/attachment.bin>