little endian bug in 2.x record parsing in new clients

[Jason Carr]

Carter,

For me, rasplit only wrote one file for 5 minutes after giving rasplit 
the -R x.x.x.x:561 option.  It doesn't exit after the 5 minutes anymore, 
not sure why though.  Is there a way to do that?

I still have the issue of radium not letting go of the file when it's 
been moved.

- Jason

On 3/12/09 12:51 PM, Carter Bullard wrote:
> Hey Jason,
> Hmmmm, so I can't replicate your problem here on any machine.
> If you can run rasplit() under gdb, (and you can, of course generate
> files with a shorter granularity), maybe you will catch the rasplit()
> exiting. Be sure and ./configure and compile with these tag files
> in the root directory:
>
> % touch .devel .debug
> % ./configure;make clean;make
> % gdb bin/rasplit
>
> For rasplit() do you have write permission to the entire directory
> structure that you are trying to write to?
>
> For rastream(), you do need the $srcid in the path, at least right now
> you do, so you should add it.
>
> rastream ....... -w /data/argus/core/\$srcid/%Y/%m/%d/argus.........
>
> Carter
>
>
>
> On Mar 12, 2009, at 1:38 AM, Jason Carr wrote:
>
>> I'm running all of this on a x86_64 Linux box.
>>
>> When I run rasplit like this:
>>
>> # rasplit -S argus-source:561 -M time 5m -w
>> /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>
>> It only creates one 5 minute file and then exits. Can I make it just
>> keep listening and keep on writing the 5 minute files?
>>
>> rastream segfaults when I run it:
>>
>> # rastream -S argus-source:561 -M time 5m -w
>> '/data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S' -B 15s -f
>> ~/argus/test.sh
>> rastream[5979]: 01:34:40.205530 output string requires $srcid
>> Segmentation fault
>>
>> We've historically always used radium to write to a file and move the
>> file every 5 minutes. radium has always just recreates the output file
>> once it's been moved. It is a little bit of a hack and if
>> rastream/rasplit already does it, that would be awesome. I'll use
>> either at this point :)
>>
>> - Jason
>>
>>
>> Carter Bullard wrote:
>>> Hey Jason,
>>> I'm answering on the mailing list, so we capture the thread.
>>>
>>> Glad to hear that your argus-2.x data on your Bivio box is
>>> working well for you. Sorry about your radium() file behavior.
>>>
>>> I still can't duplicate your error, but I'll try again later today.
>>> What kind of machine is your radium() running on?
>>>
>>> To get rasplit to generate the files run:
>>> rasplit -S radium -M time 5m \
>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>
>>> you can run it as a daemon with the "-d" option. Some like to run
>>> it from a script that respawns it if it dies, but its been a really
>>> stable
>>> program.
>>>
>>> The reason you want to use rasplit(), is that it will actually clip
>>> argus records to ensure that all the argus data for a particular time
>>> frame is only in the specific timed file. So if a record starts at
>>> 12:04:12.345127 and ends at 12:06:12.363131, rasplit will clip
>>> that record into 2 records with timestamps:
>>> 12:04:12.345127 - 12:05:00.000000
>>> 12:05:00.000000 - 12:06:12.363131
>>>
>>> This makes it so you can process, graph the files, and you don't
>>> have to worry about whether you've got all the data or not.
>>>
>>> Argus does a good job keeping its output records in time
>>> order, but if you're processing multiple streams of data, the
>>> time order can get a bit skewed. So that when you pull the file
>>> out from underneath radium() you can still get records that
>>> have data that should be in another file.
>>>
>>> To compress the data, have a cron job compress the files say
>>> an hour later.
>>>
>>> If you want to experiment, use rastream(), which is rasplit() with
>>> the addition feature to run a shell script against files after they
>>> close.
>>> You need to tell rastream() how long to wait before running the script,
>>> so you know that all the records for a particular time period are "in".
>>> If you are collecting netflow records, that time period could be hours
>>> and in some cases days (so don't use rastream for netflow data).
>>>
>>> rastream -S radium -M time 5m \
>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S \
>>> -B 15s -f /usr/local/bin/rastream.sh
>>>
>>> So the "-B 15s" option indicates that rastream() should close and
>>> process its files 15 seconds after a 5 minute boundary is passed.
>>>
>>> The sample rastream.sh in ./support/Config compresses files. You
>>> can do anything you like in there, but do test ;o)
>>>
>>> The shell script will be passed a single parameter, the full pathname
>>> of the file that is being closed.
>>>
>>> I have not done extensive testing on this specific version of
>>> rastream(),
>>> but something like it has been running for years in a number of sites,
>>> so please give it a try, and if you have any problems, holler!!!
>>>
>>> Carter
>>>
>>>
>>> On Mar 11, 2009, at 12:18 AM, Jason Carr wrote:
>>>
>>>> Pretty simple file:
>>>>
>>>> RADIUM_DAEMON=no
>>>> RADIUM_MONITOR_ID=4
>>>> RADIUM_MAR_STATUS_INTERVAL=60
>>>> RADIUM_ARGUS_SERVER=bivio-01.iso.cmu.local:561
>>>> RADIUM_ACCESS_PORT=561
>>>> RADIUM_BIND_IP=127.0.0.1
>>>> RADIUM_OUTPUT_FILE=/data/argus/var/core.out
>>>>
>>>> radium is ran like this: ./radium
>>>>
>>>> Does rasplit split out files into a directory structure like this:
>>>>
>>>> /data/argus/core/2009/03/03/22/
>>>>
>>>> With 5 minute files that are gzipped inside of that directory for the
>>>> entire hour.
>>>>
>>>> Carter Bullard wrote:
>>>>> Hey Jason,
>>>>> So I can't reproduce your situation on any of my machines here.
>>>>> How are you running radium()? Do you have a /etc/radium.conf file?
>>>>> If so can you send it?
>>>>>
>>>>> Carter
>>>>>
>>>>> On Mar 10, 2009, at 1:21 PM, Jason Carr wrote:
>>>>>
>>>>>> Good news, it seemed to have fixed my problem, thank you very much.
>>>>>>
>>>>>> Bad news, there seems to be a problem with writing to the output
>>>>>> file.
>>>>>>
>>>>>> Did the way files are written to change in this version? Right now
>>>>>> we're telling radium to write to one file and then every 5 minutes
>>>>>> move
>>>>>> the file. Previously, radium would recreate the file but now it does
>>>>>> not and lsof showed that the file was still open by radium.
>>>>>>
>>>>>> - Jason
>>>>>>
>>>>>> Carter Bullard wrote:
>>>>>>> There is a good chance that it will. But the Bivio part is still an
>>>>>>> unknown, so if it doesn't we'll still have work to do.
>>>>>>> Carter
>>>>>>>
>>>>>>> On Mar 6, 2009, at 1:48 PM, Jason Carr wrote:
>>>>>>>
>>>>>>>> Carter,
>>>>>>>>
>>>>>>>> Do you think that this would be a fix for my problem? I'll try it
>>>>>>>> out
>>>>>>>> regardless... :)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> Carter Bullard wrote:
>>>>>>>>> Gentle people,
>>>>>>>>> I have found a bug in the little-endian processing of 2.x data in
>>>>>>>>> the
>>>>>>>>> argus-clients-3.0.2 code, so I have uploaded a new
>>>>>>>>> argus-clients-3.0.2.beta.2.tar.gz file to the dev directory.
>>>>>>>>>
>>>>>>>>> ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.2.beta.2.tar.gz
>>>>>>>>>
>>>>>>>>> I have tested reading data from as many argus files as I have,
>>>>>>>>> regardless
>>>>>>>>> of version, and it seems to work, ......
>>>>>>>>>
>>>>>>>>> I also fixed problems with racluster() results having short
>>>>>>>>> counts.
>>>>>>>>> I have gone through all the clients and revised their file closing
>>>>>>>>> strategies,
>>>>>>>>> and so ....
>>>>>>>>>
>>>>>>>>> This does not fix the "-lz" issue for some ./configure runs.......
>>>>>>>>>
>>>>>>>>> For those having problems, please grab this set of code for
>>>>>>>>> testing.
>>>>>>>>> And don't hesitate to report any problems you find!!!!!
>>>>>>>>>
>>>>>>>>> Thanks!!!!!
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Carter Bullard
>>>>>>> CEO/President
>>>>>>> QoSient, LLC
>>>>>>> 150 E 57th Street Suite 12D
>>>>>>> New York, New York 10022
>>>>>>>
>>>>>>> +1 212 588-9133 Phone
>>>>>>> +1 212 588-9134 Fax
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E 57th Street Suite 12D
>>>>> New York, New York 10022
>>>>>
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York 10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>