database indexing support additions

[Carter Bullard]

Gentle people,
Very sorry, the link to the descriptions is:
  
   http://qosient.com/argus/archives.shtml

Carter

On Apr 8, 2010, at 9:06 PM, Carter Bullard wrote:

> Gentle people,
> As promised, I'm finishing up the first of the tools to improve archive
> management and performance.  This first set of tools provides very fast time
> based searching of argus data,  when the data is stored in a native file system.
> We do this through MySQL based time indexes for all the files in the archive,
> using the program, rasqltimeindex().  You access the argus data using
> the program rasql() with a "-t timerange" command line option.
> 
> I've described them briefly here:
> 
>   http://qosient.com/archives.shtml
> 
> The methods used in rasqltimeindex() are pretty straightforward.  We process
> a given file and calculate the byte offsets for each second in the file where an
> argus record spans that second.    
> 
>   rasqltimeindex -r argus.file -w mysql://user@localhost/database
> 
> This creates a Seconds, Filename and Probes table, and inserts the derived
> data of where in the flle the specific seconds start and stop.  To do the fast
> lookups you use rasql()  with a "-t timerangespecifiction" option and a
> database to read from:
> 
>   rasql -t -8h+1m -r mysql://user@localhost/database
>   rasql -t 2010/02/12.10:14:30+60s -r mysql://user@localhost/database
> 
> 
> If you are building native file system archives, you could use rastream() to
> write argus data into the archive structure, and in your rastream.sh shell
> script, call rasqltimeindex().
> 
> I will have support in the next round for you to use databases on machines
> other than the localhost, and to have the files it references reside on other
> machines, so you could have the database on one machine, and the files
> that it references on another machine.
> 
> I'm also working on utilities to delete the files from the index when you
> remove them from your repository, and also a tool so you can move the
> files, say to another machine, or to another disk.
> 
> I'd like to know if this functionality should be an option to rasqltimeindex() or
> standalone programs, like rasqlmv() and rasqlrm() or something like that.
> 
> Please give this utility a trial run, and send feedback !!!  Of course, 
> if you have any questions, opinions, suggestions, whatever, don't hesitate to
> send them to this list!!!
> 
> 
> Carter
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100408/0b87bfa4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100408/0b87bfa4/attachment.bin>