rastream and %T in -w

Carter Bullard carter at qosient.com
Thu May 9 13:31:09 EDT 2013


Hey Matt,
OK, a bug !!!!  I maybe able to reproduce this.  Let me check later today.
Carter

On May 9, 2013, at 12:42 PM, Matt Brown <matthewbrown at gmail.com> wrote:

> Yes, that would make more sense.  Check this out:
> 
> # rasplit -S 127.0.0.1:561 -M time 1d -w /var/opt/argus/%m-%d/argus_%T
> # cd 05-09/
> # ls -al
> total 12
> drwxr-xr-x. 2 root root 4096 May  9 12:36 .
> drwxr-xr-x. 4 root root 4096 May  9 12:36 ..
> -rw-r--r--. 1 root root 3260 May  9 12:36 argus_00:00:00
> # cd ..
> # rm -rf 05-09/
>  
> # rastream -B 1s -S 127.0.0.1:561 -M time 1d -w /var/opt/argus/%m-%d/argus_%T
> # cd 05-09/
> # ls -al
> total 12
> drwxr-xr-x. 2 root root 4096 May  9 12:38 .
> drwxr-xr-x. 4 root root 4096 May  9 12:38 ..
> -rw-r--r--. 1 root root  256 May  9 12:38 argus_01:00:00
> # cd ..
> # rm -rf 05-09/
>  
> Is this reproducible with 3.0.7.8?
> 
> 
> Thanks!
> 
> Matt
>  
>  
> 
> On May 9, 2013, at 11:45 AM, Carter Bullard <carter at qosient.com> wrote:
> 
>> Hey Matt,
>> Yes, the time used to generate the file will be rounded to the
>> time resolution you specify,  but I would suspect that the value 
>> should be 00:00:00, not 01:00:00.
>> 
>> rasplit() and rastream() share the same filename generation
>> logic.  If you take a file of argus data, on this specific machine,
>> does it also generate the 01:00:00 in the filenames it creates ?
>> Just one record should cause the situation/bug (if its a bug), so
>> don't need much data.
>> 
>> Carter
>> 
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E. 57th Street Suite 12D
>> New York, New York 10022
>> 
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>> 
>> 
>> 
>> On May 9, 2013, at 10:34 AM, Matt Brown <matthewbrown at gmail.com> wrote:
>> 
>>> Re-looped the list.
>>> 
>>> I had grabbed the *-latest.tar.gz off the site the other day.
>>> 
>>> Upgrading to 3.0.7.8 does not solve the problem with the file naming scheme.
>>> 
>>> I hadn't previously answered, but the times are the local TZ in the records and are not UTC in the filenames; simply always 01:00:00 regardless of when the file is created/process is started.
>>> 
>>> 
>>> strace shows nothing useful at all.  I did verify that the md5 of /etc/localtime and the correct timezone file are the same.
>>> ...
>>> 
>>> Ah hah... `-M 1d`, so resolution of the time format strings is to the day, not the given time format string.  I do find this a little odd.  Carter, what do you think?
>>> 
>>> "Problem" resolved.
>>> 
>>> 
>>> Thanks,
>>> 
>>> Matt
>>> 
>>> 
>>> 
>>> 
>>> On May 8, 2013, at 11:14 PM, Carter Bullard <carter at qosient.com> wrote:
>>> 
>>>> Hey Matt,
>>>> Are the times off by a fixed amount of time each time, like the offset to GMT ?
>>>> We did have a bug in time processing that should be fixed in argus-clients-3.0.7.8
>>>> which is on the server at:
>>>>    http://qosient.com/argus/dev/argus-clients-3.0.7.8.tar.gz
>>>> 
>>>> If you grab that and the problem goes away, I would think that it was the tm_gmoffset
>>>> bug that is biting.
>>>> 
>>>> Carter 
>>>> 
>>>> On May 8, 2013, at 9:21 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>> 
>>>>> Sorry Carter.  I did use `-B 15s`
>>>>> 
>>>>> I will attempt to dump something with `strace` with harder evidence of the problem.
>>>>> 
>>>>> %Y-%m-%d do equate to the strftime() values, but %T, %H, %M, and %S do not.
>>>>> 
>>>>> Do you have any suggestions for further troubleshoot the problem?
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, May 8, 2013 at 7:37 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>> Hey Matt,
>>>>> Don't forget you need a "-B secs" option for rastream() to work.
>>>>> If you are reading argus data, set it to 2x your flow status interval.
>>>>> I haven't used a "-", but it shouldn't make a difference.  I use '.'
>>>>> Carter
>>>>> 
>>>>> On May 8, 2013, at 7:25 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>> 
>>>>>> Hello Carter,
>>>>>> 
>>>>>> Thanks for writing back quickly.
>>>>>> 
>>>>>> 
>>>>>> If I start rastream as follows:
>>>>>> rastream -S 127.0.0.1:561 -M time 1d -w /var/opt/argus//%Y-%m-%d/argus_%T
>>>>>> 
>>>>>> the generated file is named:
>>>>>> /var/opt/argus//%Y-%m-%d/argus_01:00:00
>>>>>> 
>>>>>> 
>>>>>> As is the case with %H %M and %S == 01 00 and 00
>>>>>> 
>>>>>> I pulled these variables from the man page of strftime() http://linux.die.net/man/3/strftime
>>>>>> 
>>>>>> 
>>>>>> I've finally got around to implementing argus in a real way to complement the project flow-inspector, which presents flow data via a web interface using a few d3.js visualizations.  The commit that extends support for the data source of an "rasqlinserted" argus DB can be reviewed: https://github.com/constcast/flow-inspector/commit/e800598c3481c8ec6a44b103d98906668f612546.  It would be great to have an ra* client that would BLPOP() data into a redis queue.  A python script takes in a few IPFIX information elements about the flows and inserts them into a backend DB (mysql, oracle, or mongo).  I've been going back and forth with Lothar Braun who has been quite responsive.
>>>>>> 
>>>>>> 
>>>>>> Thanks again for your help,
>>>>>> 
>>>>>> Matt Brown
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wed, May 8, 2013 at 3:49 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> Hey Matt,
>>>>>> Not sure, from your description, what is up.
>>>>>> So, your calling rastream() against a file or a stream?
>>>>>> Parameters ?
>>>>>> 
>>>>>> Since rastream() gets its time from the records, are those correct?
>>>>>> 
>>>>>> Carter
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On May 8, 2013, at 1:52 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>>> 
>>>>>> > Hello all,
>>>>>> >
>>>>>> > With 3.0.6.2 I am seeing something odd with rastream's -w.
>>>>>> >
>>>>>> > It appears to not deal with %T %H %M or %S properly, not returning
>>>>>> > now(), but starting with 01:00:00 and 01 00 00 respectively.
>>>>>> >
>>>>>> > Why is this?
>>>>>> >
>>>>>> >
>>>>>> > Unfortunately gmane's search function seems to not be functioning.
>>>>>> >
>>>>>> >
>>>>>> > Any assistance is appreciated.
>>>>>> >
>>>>>> >
>>>>>> > Thanks,
>>>>>> >
>>>>>> > Matt Brown
>>>>>> >
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130509/64821f51/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130509/64821f51/attachment.bin>


More information about the argus mailing list