Quick question - multiple files for ra tools

[Carter Bullard]

Hey Craig,
Take a look at the combination, rasql() and rasqltimeindex().  It indexes the start and stop byte offsets for every second in each file you give it.  Rasql() uses the tables that rasqltimeindex() generates to do very fast time based data access.

There is a lot of 'magic' to make it look simple.  I run rasqltimeindex() from rastream(), every 5 minutes, on my complete archive(s), and reference the databases in my ~/.rarc file, so that rasql() can find the indexes.

   rasql -t -373500s+180s

Or something like this if your rarc isn't set up.

   rasql -t 2012/12/12.12:30:15+3m -r mysql://user@host/database

This reads every record during this 3 minute time range from the files that were indexed in the database 'database' on host 'host'.
I think this is what you are thinking about?

Carter


On Jan 14, 2013, at 3:24 PM, John Gerth <gerth at graphics.stanford.edu> wrote:

> Take a look at the options for "ra" with "man ra". Virtually all of these are applicable
> to most of the argus tools due to the comprehensive design of the argus client libraries.
> 
> For your specific question  "-r ...." is a list of files,  "-R ..." asks for recursion
> 
> In constructing a GUI to deal with displaying time ranges of flows, you're going to want
> to look at Carter's recent additions for data management of flow archives. He's got some
> nifty ways of doing exactly the indexing you want and even for compressed archives.
> I'll step aside here and let those who know more chime in.
> 
> 
> John Gerth      gerth at graphics.stanford.edu  Gates 378   (650) 725-3273 fax 725-6949
> 
> On 1/14/2013 10:05 AM, Craig Merchant wrote:
>> Is it possible to feed ra tools a comma-separated list of files to use or is it limited to either a single file or recursing through an entire
>> directory structure?
>> 
>> 
>> 
>> We are eventually going to build a GUI front-end to Argus in Splunk.  Given the volume of data we’re dealing with, I don’t want Argus to recurse
>> through days/weeks of flow data if the search is only spanning a few minutes or hours.  If I put the epoch time value in the file name, it should be
>> pretty trivial to generate the list of files that span the time period I want to search.
>> 
>> 
>> 
>> If that isn’t supported, it would be great if rasplit or rabins could invoke ratimerange each time they write a file to some kind of index file that
>> other ra clients could point to so that when recursing through the directory structure, they only open files that contain records within the specified
>> time range.
>> 
>> 
>> 
>> Thanks.
>> 
>> 
>> C
>