database indexing support additions

[Carter Bullard]

Gentle people,
As promised, I'm finishing up the first of the tools to improve archive
management and performance.  This first set of tools provides very fast time
based searching of argus data,  when the data is stored in a native file system.
We do this through MySQL based time indexes for all the files in the archive,
using the program, rasqltimeindex().  You access the argus data using
the program rasql() with a "-t timerange" command line option.

I've described them briefly here:

   http://qosient.com/archives.shtml

The methods used in rasqltimeindex() are pretty straightforward.  We process
a given file and calculate the byte offsets for each second in the file where an
argus record spans that second.    

   rasqltimeindex -r argus.file -w mysql://user@localhost/database

This creates a Seconds, Filename and Probes table, and inserts the derived
data of where in the flle the specific seconds start and stop.  To do the fast
lookups you use rasql()  with a "-t timerangespecifiction" option and a
database to read from:

   rasql -t -8h+1m -r mysql://user@localhost/database
   rasql -t 2010/02/12.10:14:30+60s -r mysql://user@localhost/database


If you are building native file system archives, you could use rastream() to
write argus data into the archive structure, and in your rastream.sh shell
script, call rasqltimeindex().

I will have support in the next round for you to use databases on machines
other than the localhost, and to have the files it references reside on other
machines, so you could have the database on one machine, and the files
that it references on another machine.

I'm also working on utilities to delete the files from the index when you
remove them from your repository, and also a tool so you can move the
files, say to another machine, or to another disk.

I'd like to know if this functionality should be an option to rasqltimeindex() or
standalone programs, like rasqlmv() and rasqlrm() or something like that.

Please give this utility a trial run, and send feedback !!!  Of course, 
if you have any questions, opinions, suggestions, whatever, don't hesitate to
send them to this list!!!


Carter



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100408/c814f991/attachment.bin>