possible radium issue

[Phillip G Deneault]

Sorry for the late reply, I tested it today and was successful.  Only one 
srcid per file now.

I will agree, I didn't notice a problem with rastream.  This only seemed 
to occur in rasplit.

Thanks!
Phil

On Thu, 9 Jul 2009, Carter Bullard wrote:

> Hey Phillip,
> I refreshed the argus-clients-3.0.2.beta.9 with fixes for the rasplit() 
> problem.
> rastream() is ok on this one right now.  Grab the image and give it a quick 
> try.
>
> http://qosient.com/argus/dev/argus-clients-3.0.2.beta.9.tar.gz
>
> Carter
>
> On Jul 9, 2009, at 1:57 PM, Phillip Deneault wrote:
>
>> I can confirm every file being generated has multiple srcid's in them 
>> despite having the $srcid criteria in the rasplit lines in all three time 
>> slices.
>> 
>> Thanks,
>> Phil
>> 
>> Carter Bullard wrote:
>>> The tell tale symptom would be that you have slices with multiple
>>> srcid's in them, even though your output filename has the
>>> $srcid directive in it.
>>> I'm working this now, if this is your problem, then I hope to have
>>> a fix very soon.
>>> Carter
>>> On Jul 9, 2009, at 11:36 AM, Phillip Deneault wrote:
>>>> Ummm, I think so?  You know your code best. :-)
>>>> 
>>>> If I'm reading this right, you have a pool of file handles to use.  You 
>>>> believe the pool is working fine except that its ignoring the srcid when 
>>>> choose which entry in the pool to use?  If so, yes, that is very possibly 
>>>> what could be happening.
>>>> 
>>>> Phil
>>>> 
>>>> Carter Bullard wrote:
>>>>> OK!!!  Progress.
>>>>> So, if I'm reading this right, rasplit() seems to be ignoring the srcid 
>>>>> on
>>>>> some outputs?
>>>>> Rasplit() does cache the output files, and tries to be smart about open
>>>>> file descriptors, so its not reading a record, opening the output file,
>>>>> writing a single record, and closing the file.  We keep a list of open
>>>>> files per srcid, when the "$srcid" keywork is in the output filename.
>>>>> It must be that we're confusing which file to use?
>>>>> Does this seem to describe where we are?
>>>>> Carter
>>>>> On Jul 9, 2009, at 11:18 AM, Phillip Deneault wrote:
>>>>>> Carter Bullard wrote:
>>>>>>> One thing to check is whether rasplit() is generating a file somewhere 
>>>>>>> else
>>>>>>> in your file system, say if the "srcid" is screwy, or if time goes to 
>>>>>>> zero.  Are your dates in the file name looking alright?
>>>>>> 
>>>>>> More progress.  The time boundaries look ok, but if I compare all file 
>>>>>> sizes of that time period, one is always WAY bigger than the other ones 
>>>>>> and it seems to be randomly picked as to which file, in which $srcid 
>>>>>> directory it will be.
>>>>>> 
>>>>>>> I would recommend that you add a few more directories in your target 
>>>>>>> path.
>>>>>>> Unix has a bad performance issue when the files/directory get above 
>>>>>>> say
>>>>>>> 200 or so.  Thats why I add a %Y/%m/%d for the slices, so that the 
>>>>>>> file
>>>>>>> count doesn't get too  high.
>>>>>> 
>>>>>> This isn't so much of a problem since my retention on these files is 
>>>>>> relatively small, only a day or two.  They are being processed into 
>>>>>> other things and then removed.
>>>>>>