rabins issue, maybe order related

[Mark E. Mallett]

On Wed, Jun 20, 2012 at 01:29:38PM -0400, Carter Bullard wrote:
> Hey  Mark,
> Yes, all ra* programs will take "-R ." file names and sort them to figure
> out what files to open in what order.    I don't think the -r filenames are
> a part of that algorithm, so you can push files in front of others by using
> -r instead of -R.
> 
> Currently, rabins() would like its input to be in some time order, as it uses the
> stream to estimate what the starting bin time is, and how many bins to process.
> rabins() will discard input records that are before its notion of the "epoch" of time.
> If you use "-D 2", rabins() will print out an error message when it throws
> a record away.  Something like this:

I ran it with -D all the way up to 8, but didn't see anything.


> You can solve this problem by giving rabins() a time filter that spans  the
> time of your processing.  This tell rabins to preallocate bins for the time
> span, and you are guaranteed that your data will be processed, regardless
> of what order it comes in.  Since your doing day bins, you can probably
> get away with an entire year as your filter " -t -12M+12M ".

In fact I know the time range (or rather the scripts know it), so that's
a great solution.  I'll try that.

Thanks,
-mm-