Best way to grab summary data from last generated file by rastream

[Paul Halliday]

You folks are very helpful. Thanks!

On Tue, May 7, 2013 at 12:27 PM, Carter Bullard <carter at qosient.com> wrote:

> Hey Jesse, Paul,
> I understand the statement, "undocumented -f option to rastream", but the
> option is in the manpage rastream.1.
>
> SYNOPSIS
>        rastream [[-M splitmode] [splitmode options]] [-f file processing
> script -B secs] [raoptions]
>
> Now I will admit that the "file processing script" isn't well described,
> but undocumented?
> I do take exception ;O)
>
> OK, this is the text that is in the updated manpage, that is not yet in
> the distribution:
>
>        -f program
>            Post processing program. rastream, will execute this program /
> shell script just after
>            closing  the output file, passing the full path to the closed
> output file as a parame-
>            ter, using this convention:
>
>               program -r /full/path/to/closed/file
>
>            This allows you to post-process the output file in an automated
> fashion.
>
>            Generally, this program can do anything you like, such as
> aggregating  and  correcting
>            flow  records,  labeling  records  for semantic enhancement,
> indexing the files, using
>            programs like rasqltimeindex(), and compressing the files.
>  Traditionally, the program
>            has  been  a shell-script, perl program, or php script, so that
> it can be easily modi-
>            fied, on the fly, but it can be any executable  that  can
>  handle  the  "-r  filename"
>            parameter  convention.   The  program should provides its own
> accountability and error
>            logging, so that you know that things are working as you expect.
>
>            rastream must have a path to the program, the program must be
> executable, and rastream
>            must have permission to run the program for this strategy to be
> successful.
>
>            An example rastream.sh is provided in the ./support/Config
> directory.
>
> So, Paul, good you're using rastream().   Take a look at the example
> script, rastream.sh,
> in ./support/Config, and modify it to do what you want.  If you have any
> problems, grab the
> latest clients from the development site to see if that fixes your version
> of rastream(),
> and if not holler on the list.
>
> Remember, the " -B secs " option will cause rastream() to reject flows
> that are older
> than " theCurrentTime - Bsecs ", so be sure and chose a " -B secs " value
> that makes
> sense for your flow data sources.
>
> If you're using netflow, don't use this mechanism, as netflow sources will
> send records
> that are older, and this rule will toss records, regardless of what time
> you set it for  !!!
>
> Hope all is most excellent,
>
> Carter
>
> On May 7, 2013, at 10:53 AM, Jesse Bowling <jessebowling at gmail.com> wrote:
>
> Hi Paul,
>
> You're looking for the (as best as I can tell by looking at -h and the
> qosient man pages) undocumented '-f' option to rastream...With -f, rastream
> will execute the specified script providing the current filename as an
> argument...For example:
>
> /usr/local/bin/rastream -M time 5m -B 10s -S ${OUR_IP}:${OUR_PORT} -w
> /nsm/argus/data/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S -f
> /usr/local/bin/argus_postprocess.bash -d
>
> This will wait 10 seconds after the 5 minute mark and then execute
> /usr/local/bin/argus_postprocess.bash with an argument of the filename...
>
> There's a sample script in ./support/Config/rastream.sh
>
> You might also check out http://nsmwiki.org/Argus , as this is one of the
> only wiki's I know dedicated to argus (although some sections of it could
> use updating)... :)
>
> Does that help?
>
> Cheers,
>
> Jesse
>
>
>
>
> On Tue, May 7, 2013 at 9:52 AM, Paul Halliday <paul.halliday at gmail.com>wrote:
>
>> I have rastream processing on hard 5 minute boundaries and I would like
>> to create summary data after it closes each file.
>>
>> flow-capture had a nice option that would let you call an external
>> program after it finished spooling a file; do I have an option like this
>> with argus?
>>
>> I can script it, just curious if there is something built-in.
>>
>> thanks.
>>
>> --
>> Paul Halliday
>> http://www.pintumbler.org/
>
>
>
>
> --
> Jesse Bowling
>
>
>


-- 
Paul Halliday
http://www.pintumbler.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130507/b861773e/attachment.html>