possible radium issue

[Phillip Deneault]

Ummm, I think so?  You know your code best. :-)

If I'm reading this right, you have a pool of file handles to use.  You 
believe the pool is working fine except that its ignoring the srcid when 
choose which entry in the pool to use?  If so, yes, that is very 
possibly what could be happening.

Phil

Carter Bullard wrote:
> OK!!!  Progress.
> So, if I'm reading this right, rasplit() seems to be ignoring the srcid on
> some outputs?
> 
> Rasplit() does cache the output files, and tries to be smart about open
> file descriptors, so its not reading a record, opening the output file,
> writing a single record, and closing the file.  We keep a list of open
> files per srcid, when the "$srcid" keywork is in the output filename.
> It must be that we're confusing which file to use?
> 
> Does this seem to describe where we are?
> 
> Carter
> 
> On Jul 9, 2009, at 11:18 AM, Phillip Deneault wrote:
> 
>> Carter Bullard wrote:
>> > One thing to check is whether rasplit() is generating a file 
>> somewhere else
>>> in your file system, say if the "srcid" is screwy, or if time goes to 
>>> zero.  Are your dates in the file name looking alright?
>>
>> More progress.  The time boundaries look ok, but if I compare all file 
>> sizes of that time period, one is always WAY bigger than the other 
>> ones and it seems to be randomly picked as to which file, in which 
>> $srcid directory it will be.
>>
>>> I would recommend that you add a few more directories in your target 
>>> path.
>>> Unix has a bad performance issue when the files/directory get above say
>>> 200 or so.  Thats why I add a %Y/%m/%d for the slices, so that the file
>>> count doesn't get too  high.
>>
>> This isn't so much of a problem since my retention on these files is 
>> relatively small, only a day or two.  They are being processed into 
>> other things and then removed.
>>
>> Phil
>>
> 
> 
> 
> 

-- 
--------------------------------------------------------------------
   WPI Information Technology will never ask for your password and
you should never give it.  http://www.wpi.edu/+infosec/phishing.html
--------------------------------------------------------------------
Phil Deneault                               Network Security Officer
deneault at wpi.edu                                Information Security
http://www.wpi.edu/~deneault/        Worcester Polytechnic Institute
--------------------------------------------------------------------