Rewrite of rastream

[Carter Bullard]

Hey Matt,
Just for completeness, the purpose of the "-B <secs> " option, is
to give rastream() an " all clear " time period.  The time should
take into account the ARGUS_FLOW_STATUS_INTERVAL of all the sources,
and time drift, and collection pipeline processing delay. 

Because rastream() can be collecting from multiple sources, (1…x)
I recommend that <secs> be, at least:

   (max(ARGUS_FLOW_STATUS_INTERVAL.1, …, ARGUS_FLOW_STATUS_INTERVAL.x) + 5)

rastream() could figure this out itself, as the ARGUS_FLOW_STATUS_INTERVAL
is reported downstream through the " man " records, but because you can
filter out " man " records in complex collection infrastructure, 
its not necessarily a reliable configuration strategy.

I personally use +10, just in case the system is supporting a lot of
tasks that kick in right on the hour, to avoid task collision at key
times.  The extra 5 secs, probably, isn't necessary, or for some
systems, +30 could be more appropriate.  All depends.

Carter



On Jun 24, 2013, at 11:02 AM, Matt Brown <matthewbrown at gmail.com> wrote:

> Thanks Carter.  I'll take a closer look.
> 
> I was and am concerned because argus() and rastream() are and always were running on the same box (with the same time source).
> 
> 
> Thanks,
> 
> Matt
> 
> 
> On Jun 24, 2013, at 10:30 AM, Carter Bullard <carter at qosient.com> wrote:
> 
>> Hey Matt,
>> If you run rastream() with debug level set to 2, you'll get error
>> messages when you reject late records.  Something like:
>> 
>>     "RaSendArgusRecord() rejecting late record secs ….."
>> 
>> Depending on your source, you will either get no messages or a
>> lot of them.
>> 
>> A good way to test, is to run rasplit() for a while, which doesn't
>> reject late records, and check the last modified dates on the file
>> to see if you are writing after the cutoff times.
>> 
>> Remember, if your argus data source's clock is way off, then you
>> will have problems.  We use the timestamp in the record to make
>> the decision as to where the record is going to go, but we're using
>> the local time where rastream() is running to make the decision to
>> close the file and reject late records.  So if your clocks aren't
>> sync'd up, then your " -B <sec> " may be inadequate.
>> 
>> 
>> Carter
>> 
>> 
>> On Jun 21, 2013, at 2:13 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>> 
>>> Carter,
>>> 
>>> The modified version(s) no longer cause the issue of the "excess" data file.
>>> 
>>> What should I do to test that data isn't being dropped as you originally indicated might occur?
>>> 
>>> 
>>> Thanks!
>>> 
>>> Matt
>>> 
>>> On Jun 17, 2013, at 2:40 PM, Carter Bullard <carter at qosient.com> wrote:
>>> 
>>>> Gentle men,
>>>> Any improvements in rastream()?
>>>> Carter
>>>> 
>>>> 
>>>> On Jun 10, 2013, at 8:44 PM, David Edelman <dedelman at iname.com> wrote:
>>>> 
>>>>> Carter,
>>>>>  
>>>>> I think that one of the files is not the one you expected to send.
>>>>>  
>>>>> --Dave
>>>>>  
>>>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On Behalf OfCarter Bullard
>>>>> Sent: Monday, June 10, 2013 4:47 PM
>>>>> To: Matt Brown
>>>>> Cc: argus-info at lists.andrew.cmu.edu
>>>>> Subject: Re: [ARGUS] Rewrite of rastream
>>>>>  
>>>>> Hey Matt,
>>>>> Give these two files a try.  This version of rastream.c does the record
>>>>> rejection, and the argus_util.c is needed to make debug message for -d
>>>>> daemons to go to syslog.  So, ……, put rastream.c in ./examples/rastream,
>>>>> and argus_util.c in ./common.  make again, and give it a try.
>>>>>  
>>>>> Should give you something that seems to work, but it may be working
>>>>> because its throwing data away.  Give this a test run, and then, lets
>>>>> try to fix your issue.
>>>>>  
>>>>> Carter 
>>>>> 
>>>> 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130624/bb6c9b09/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130624/bb6c9b09/attachment.bin>