Recursive Read Problem with rasqlinsert

[Carter Bullard]

Hey Dave,
When you use the -R option, the ra* program will recursively descend into the file system,
and list every regular file it finds, excluding files that start with a '.'.  If you set the "-D2" option
before the "-R' option on the command line, you'll see every file that is being processed.

We then sort the file names, because descending a Unix file system doesn't
mean you're doing it in any order.  We do a simple strcmp() but with the 
standard archive naming strategy, we get the files in chronological order.

For most aggregators, such rabins(), racluster(), ratop(), rasqlinsert(), etc...
getting records in chronological order can be important, as each can be configured
to hold data for fixed time limits, and time for these tools is mostly based on the data
records timestamps.  This is true except for some queue timeouts that use
actual processing time.

One possibility for the bug is that your file system strategy may be generating a
sorted list that is not in chronological order, like getting dec-2012 before mar-2012,
just as an example.  

Another potential problem, could be, that one of the ra* program threads has
died during operation, such as the output thread, and while the input and the
analytic are working, no records are being output ???

Best way to catch the former is to run with "-D2" to see what files are being
processed, and in what order.  For the later is to run under gdb() to see if
one of the threads hits a snag.

Carter

On Oct 5, 2013, at 3:21 PM, "David Edelman" <dedelman at iname.com> wrote:

> This is the first of two problems that I am seeing with rasqlinsert reading its input from argus files.
> 
> 
> In this case I have a set of files stored in in the typical archive/YYYY/MM/DD/file_YYYY_MM_DD_HH.gz hierarchy. If I cd to archive/YYYY and attempt to use rasqlinsert  -R* go through  that set of files, I don’t end up with much of anything in the database. On the other hand, if I use a set of nested Bash for loops sequencing through the month and day values and use rasqlinsert –R $M/$D/* all seems to be fine.
> 
> 
> Has anyone else seen this problem?  The second problem is in the next email to avoid subject creep.
> 
> 
> --Dave
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/b33055c0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/b33055c0/attachment.bin>