possible radium issue

[Carter Bullard]

Hey Phillip,
yes, I have a fix now for rasplit(), and I'll need to implement it for  
rastream() as well.
need to let to test for a while, but I should have it up by late this  
afternoon, EDT.

Carter

On Jul 9, 2009, at 1:57 PM, Phillip Deneault wrote:

> I can confirm every file being generated has multiple srcid's in  
> them despite having the $srcid criteria in the rasplit lines in all  
> three time slices.
>
> Thanks,
> Phil
>
> Carter Bullard wrote:
>> The tell tale symptom would be that you have slices with multiple
>> srcid's in them, even though your output filename has the
>> $srcid directive in it.
>> I'm working this now, if this is your problem, then I hope to have
>> a fix very soon.
>> Carter
>> On Jul 9, 2009, at 11:36 AM, Phillip Deneault wrote:
>>> Ummm, I think so?  You know your code best. :-)
>>>
>>> If I'm reading this right, you have a pool of file handles to  
>>> use.  You believe the pool is working fine except that its  
>>> ignoring the srcid when choose which entry in the pool to use?  If  
>>> so, yes, that is very possibly what could be happening.
>>>
>>> Phil
>>>
>>> Carter Bullard wrote:
>>>> OK!!!  Progress.
>>>> So, if I'm reading this right, rasplit() seems to be ignoring the  
>>>> srcid on
>>>> some outputs?
>>>> Rasplit() does cache the output files, and tries to be smart  
>>>> about open
>>>> file descriptors, so its not reading a record, opening the output  
>>>> file,
>>>> writing a single record, and closing the file.  We keep a list of  
>>>> open
>>>> files per srcid, when the "$srcid" keywork is in the output  
>>>> filename.
>>>> It must be that we're confusing which file to use?
>>>> Does this seem to describe where we are?
>>>> Carter
>>>> On Jul 9, 2009, at 11:18 AM, Phillip Deneault wrote:
>>>>> Carter Bullard wrote:
>>>>> > One thing to check is whether rasplit() is generating a file  
>>>>> somewhere else
>>>>>> in your file system, say if the "srcid" is screwy, or if time  
>>>>>> goes to zero.  Are your dates in the file name looking alright?
>>>>>
>>>>> More progress.  The time boundaries look ok, but if I compare  
>>>>> all file sizes of that time period, one is always WAY bigger  
>>>>> than the other ones and it seems to be randomly picked as to  
>>>>> which file, in which $srcid directory it will be.
>>>>>
>>>>>> I would recommend that you add a few more directories in your  
>>>>>> target path.
>>>>>> Unix has a bad performance issue when the files/directory get  
>>>>>> above say
>>>>>> 200 or so.  Thats why I add a %Y/%m/%d for the slices, so that  
>>>>>> the file
>>>>>> count doesn't get too  high.
>>>>>
>>>>> This isn't so much of a problem since my retention on these  
>>>>> files is relatively small, only a day or two.  They are being  
>>>>> processed into other things and then removed.
>>>>>
>>>>> Phil
>>>>>
>>>
>>> -- 
>>> --------------------------------------------------------------------
>>> WPI Information Technology will never ask for your password and
>>> you should never give it.  http://www.wpi.edu/+infosec/phishing.html
>>> --------------------------------------------------------------------
>>> Phil Deneault                               Network Security Officer
>>> deneault at wpi.edu                                Information Security
>>> http://www.wpi.edu/~deneault/        Worcester Polytechnic Institute
>>> --------------------------------------------------------------------
>>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York  10022
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>
> -- 
> --------------------------------------------------------------------
>  WPI Information Technology will never ask for your password and
> you should never give it.  http://www.wpi.edu/+infosec/phishing.html
> --------------------------------------------------------------------
> Phil Deneault                               Network Security Officer
> deneault at wpi.edu                                Information Security
> http://www.wpi.edu/~deneault/        Worcester Polytechnic Institute
> --------------------------------------------------------------------
>

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090709/c05cae4e/attachment.bin>