possible radium issue

[Phillip Deneault]

I can confirm every file being generated has multiple srcid's in them 
despite having the $srcid criteria in the rasplit lines in all three 
time slices.

Thanks,
Phil

Carter Bullard wrote:
> The tell tale symptom would be that you have slices with multiple
> srcid's in them, even though your output filename has the
> $srcid directive in it.
> 
> I'm working this now, if this is your problem, then I hope to have
> a fix very soon.
> 
> Carter
> 
> On Jul 9, 2009, at 11:36 AM, Phillip Deneault wrote:
> 
>> Ummm, I think so?  You know your code best. :-)
>>
>> If I'm reading this right, you have a pool of file handles to use.  
>> You believe the pool is working fine except that its ignoring the 
>> srcid when choose which entry in the pool to use?  If so, yes, that is 
>> very possibly what could be happening.
>>
>> Phil
>>
>> Carter Bullard wrote:
>>> OK!!!  Progress.
>>> So, if I'm reading this right, rasplit() seems to be ignoring the 
>>> srcid on
>>> some outputs?
>>> Rasplit() does cache the output files, and tries to be smart about open
>>> file descriptors, so its not reading a record, opening the output file,
>>> writing a single record, and closing the file.  We keep a list of open
>>> files per srcid, when the "$srcid" keywork is in the output filename.
>>> It must be that we're confusing which file to use?
>>> Does this seem to describe where we are?
>>> Carter
>>> On Jul 9, 2009, at 11:18 AM, Phillip Deneault wrote:
>>>> Carter Bullard wrote:
>>>> > One thing to check is whether rasplit() is generating a file 
>>>> somewhere else
>>>>> in your file system, say if the "srcid" is screwy, or if time goes 
>>>>> to zero.  Are your dates in the file name looking alright?
>>>>
>>>> More progress.  The time boundaries look ok, but if I compare all 
>>>> file sizes of that time period, one is always WAY bigger than the 
>>>> other ones and it seems to be randomly picked as to which file, in 
>>>> which $srcid directory it will be.
>>>>
>>>>> I would recommend that you add a few more directories in your 
>>>>> target path.
>>>>> Unix has a bad performance issue when the files/directory get above 
>>>>> say
>>>>> 200 or so.  Thats why I add a %Y/%m/%d for the slices, so that the 
>>>>> file
>>>>> count doesn't get too  high.
>>>>
>>>> This isn't so much of a problem since my retention on these files is 
>>>> relatively small, only a day or two.  They are being processed into 
>>>> other things and then removed.
>>>>
>>>> Phil
>>>>
>>
>> -- 
>> --------------------------------------------------------------------
>>  WPI Information Technology will never ask for your password and
>> you should never give it.  http://www.wpi.edu/+infosec/phishing.html
>> --------------------------------------------------------------------
>> Phil Deneault                               Network Security Officer
>> deneault at wpi.edu                                Information Security
>> http://www.wpi.edu/~deneault/        Worcester Polytechnic Institute
>> --------------------------------------------------------------------
>>
> 
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York  10022
> 
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
> 
> 
> 

-- 
--------------------------------------------------------------------
   WPI Information Technology will never ask for your password and
you should never give it.  http://www.wpi.edu/+infosec/phishing.html
--------------------------------------------------------------------
Phil Deneault                               Network Security Officer
deneault at wpi.edu                                Information Security
http://www.wpi.edu/~deneault/        Worcester Polytechnic Institute
--------------------------------------------------------------------