little endian bug in 2.x record parsing in new clients

[Jason Carr]

I'm running all of this on a x86_64 Linux box.

When I run rasplit like this:

# rasplit -S argus-source:561 -M time 5m -w
/data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S

It only creates one 5 minute file and then exits.  Can I make it just
keep listening and keep on writing the 5 minute files?

rastream segfaults when I run it:

# rastream -S argus-source:561 -M time 5m -w
'/data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S' -B 15s -f
~/argus/test.sh
rastream[5979]: 01:34:40.205530 output string requires $srcid
Segmentation fault

We've historically always used radium to write to a file and move the
file every 5 minutes.  radium has always just recreates the output file
once it's been moved.  It is a little bit of a hack and if
rastream/rasplit already does it, that would be awesome.  I'll use
either at this point :)

- Jason


Carter Bullard wrote:
> Hey Jason,
> I'm answering on the mailing list, so we capture the thread.
> 
> Glad to hear that your argus-2.x data on your Bivio box is
> working well for you.  Sorry about your radium() file behavior.
> 
> I still can't duplicate your error, but I'll try again later today.
> What kind of machine is your radium() running on?
> 
> To get rasplit to generate the files run:
>    rasplit -S radium -M time 5m \
>        -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
> 
> you can run it as a daemon with the "-d" option.  Some like to run
> it from a script that respawns it if it dies, but its been a really stable
> program.
> 
> The reason you want to use rasplit(), is that it will actually clip
> argus records to ensure that all the argus data for a particular time
> frame is only in the specific timed file.   So if a record starts at
> 12:04:12.345127 and ends at 12:06:12.363131, rasplit will clip
> that record into 2 records with timestamps:
>    12:04:12.345127 - 12:05:00.000000
>    12:05:00.000000 - 12:06:12.363131
> 
> This makes it so you can process, graph the files, and you don't
> have to worry about whether you've got all the data or not.
> 
> Argus does a good job keeping its output records in time
> order, but if you're processing multiple streams of data, the
> time order can get a bit skewed.  So that when you pull the file
> out from  underneath radium() you can still get records that
> have data that should be in another file.
> 
> To compress the data, have a cron job compress the files say
> an hour later.
> 
> If you want to experiment, use rastream(), which is rasplit() with
> the addition feature to run a shell script against files after they close.
> You need to tell rastream() how long to wait before running the script,
> so you know that all the records for a particular time period are "in".
> If you are collecting netflow records, that time period could be hours
> and in some cases days (so don't use rastream for netflow data).
> 
>    rastream -S radium -M time 5m \
>        -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S \
>        -B 15s -f /usr/local/bin/rastream.sh
> 
> So the "-B 15s" option indicates that rastream() should close and
> process its files 15 seconds after a 5 minute boundary is passed.
> 
> The sample rastream.sh in ./support/Config compresses files.  You
> can do anything you like in there, but do test ;o)
> 
> The shell script will be passed a single parameter, the full pathname
> of the file that is being closed.
> 
> I have not done extensive testing on this specific version of rastream(),
> but something like it has been running for years in a number of sites,
> so please give it a try, and if you have any problems, holler!!!
> 
> Carter
> 
> 
> On Mar 11, 2009, at 12:18 AM, Jason Carr wrote:
> 
>> Pretty simple file:
>>
>> RADIUM_DAEMON=no
>> RADIUM_MONITOR_ID=4
>> RADIUM_MAR_STATUS_INTERVAL=60
>> RADIUM_ARGUS_SERVER=bivio-01.iso.cmu.local:561
>> RADIUM_ACCESS_PORT=561
>> RADIUM_BIND_IP=127.0.0.1
>> RADIUM_OUTPUT_FILE=/data/argus/var/core.out
>>
>> radium is ran like this: ./radium
>>
>> Does rasplit split out files into a directory structure like this:
>>
>> /data/argus/core/2009/03/03/22/
>>
>> With 5 minute files that are gzipped inside of that directory for the
>> entire hour.
>>
>> Carter Bullard wrote:
>>> Hey Jason,
>>> So I can't reproduce your situation on any of my machines here.
>>> How are you running radium()?  Do you have a /etc/radium.conf file?
>>> If so can you send it?
>>>
>>> Carter
>>>
>>> On Mar 10, 2009, at 1:21 PM, Jason Carr wrote:
>>>
>>>> Good news, it seemed to have fixed my problem, thank you very much.
>>>>
>>>> Bad news, there seems to be a problem with writing to the output file.
>>>>
>>>> Did the way files are written to change in this version?  Right now
>>>> we're telling radium to write to one file and then every 5 minutes move
>>>> the file.  Previously, radium would recreate the file but now it does
>>>> not and lsof showed that the file was still open by radium.
>>>>
>>>> - Jason
>>>>
>>>> Carter Bullard wrote:
>>>>> There is a good chance that it will.  But the Bivio part is still an
>>>>> unknown, so if it doesn't we'll still have work to do.
>>>>> Carter
>>>>>
>>>>> On Mar 6, 2009, at 1:48 PM, Jason Carr wrote:
>>>>>
>>>>>> Carter,
>>>>>>
>>>>>> Do you think that this would be a fix for my problem?  I'll try it
>>>>>> out
>>>>>> regardless... :)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Carter Bullard wrote:
>>>>>>> Gentle people,
>>>>>>> I have found a bug in the little-endian processing of 2.x data in
>>>>>>> the
>>>>>>> argus-clients-3.0.2 code, so I have uploaded a new
>>>>>>> argus-clients-3.0.2.beta.2.tar.gz file to the dev directory.
>>>>>>>
>>>>>>> ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.2.beta.2.tar.gz
>>>>>>>
>>>>>>> I have tested reading data from as many argus files as I have,
>>>>>>> regardless
>>>>>>> of version, and it seems to work, ......
>>>>>>>
>>>>>>> I also fixed problems with racluster() results having short counts.
>>>>>>> I have gone through all the clients and revised their file closing
>>>>>>> strategies,
>>>>>>> and so ....
>>>>>>>
>>>>>>> This does not fix the "-lz" issue for some ./configure runs.......
>>>>>>>
>>>>>>> For those having problems, please grab this set of code for testing.
>>>>>>> And don't hesitate to report any problems you find!!!!!
>>>>>>>
>>>>>>> Thanks!!!!!
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E 57th Street Suite 12D
>>>>> New York, New York  10022
>>>>>
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York  10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>>
>>
> 
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York  10022
> 
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
> 
> 
> 
>