little endian bug in 2.x record parsing in new clients

Mon Mar 23 13:17:35 EDT 2009

default is the Bivio's device name that is (I believe) only interpreted 
in the libpcap.  There is no /dev/default.  I can use -i s0.e0 instead 
which is a normal interface that is in ifconfig.  argus 2.6 doesn't seem 
to care that it's not a real device.  I looked at argus 3.0 and it seems 
like it requires that it's a real device and -i default will not work then.

I'll try it out without the -e and the -F /etc/argus.conf.  I'll change 
the -i to s0.e0.

On 3/23/09 12:54 PM, Carter Bullard wrote:
> Hey Jason,
> A few things you should change.
>
> All the things you have on the command line can be set in the argus.conf
> file, that would be a great place to set them, unless they change with
> every
> run.
>
> Don't use "-F /etc/argus.conf". Argus always reads the /etc/argus.conf, and
> by putting a "-F /etc/argus.conf" it is reading it twice, and thus
> setting the
> bind addresses twice, which means its opening twice, etc.... This is not
> good.
>
> You have the ARGUS_MONITOR_ID set in the argus.conf file, but you are
> changing it on the command line, and then you re-read the argus.conf file.
> What MONITOR_ID is actually being used would be interesting to know.
>
> I have no idea what "-i default" will do. Do you have a /dev/default
> device?
>
>
> On Mar 23, 2009, at 11:47 AM, Jason Carr wrote:
>
>> For argus:
>>
>> /usr/sbin/argus -P 561 -B 10.10.10.103 -e CPU-2c1 -i default -F
>> /etc/argus.conf
>>
>> argus.conf:
>> ARGUS_DAEMON=no
>> ARGUS_MAX_INSTANCES=1
>> ARGUS_SET_PID=no
>> ARGUS_PID_FILENAME=/var/run/argus.pid
>> ARGUS_MONITOR_ID=`hostname`
>> ARGUS_ACCESS_PORT=561
>> ARGUS_BIND_IP="127.0.0.1"
>> ARGUS_GO_PROMISCUOUS=yes
>> ARGUS_FLOW_STATUS_INTERVAL=60
>> ARGUS_MAR_STATUS_INTERVAL=300
>> ARGUS_GENERATE_RESPONSE_TIME_DATA=no
>> ARGUS_GENERATE_JITTER_DATA=no
>> ARGUS_GENERATE_MAC_DATA=no
>> ARGUS_CAPTURE_DATA_LEN=128
>> ARGUS_FILTER_OPTIMIZER=yes
>> ARGUS_FILTER=""
>>
>> argus runs on an internal IP address 10.10.10.103. The connection from
>> the client machine running rasplit comes from an external IP (lets
>> call that argus-sensor) port forwarded to 10.10.10.103. Lets just call
>> it argus-collector. argus-collector has the following configuration:
>>
>> ./rasplit -S argus-sensor:561 -M time 5m -w
>> /data/argus/core/%Y/%m/%d/%H/core.%Y.%m.%d.%H.%M.%S.argus
>
>
> The best way to talk about this is that argus "binds" to 10.10.10.103.
> That is much different than
> "argus runs on 10.10.10.103", especially in multi-processor/multi-core
> environments.
>
> I'm concerned about the port forwarding, why not just bind argus to the
> 172.19.41.41 address?
> I think when you are having problems, that all this complexity is just
> going
> to be in the way?
>
>>
>>
>>
>> So essentially, we're doing this:
>>
>>
>> +---------------+ +---------------+
>> | 10.10.10.103 |<->| 10.10.10.1 |
>> | argus TCP/561 | | port forward |
>> +---------------+ +---------------+
>> ^
>> | tcp
>> \/ 561
>> +---------------+ +---------------+
>> | 172.19.41.42 |<->| 172.19.41.41 |
>> | rasplit | | port forward |
>> +---------------+ +---------------+
>>
>> 172.19.41.41, 10.10.10.1, 10.10.10.103 are all the same physical unit
>> and are all PowerPC arcitecture. 10.10.10.1 and 172.19.41.41 is the
>> same machine.
>>
>> 172.19.41.42 is an x86_64 machine.
>>
>>
>> Hopefully this makes sense. If it doesn't let me know and I'll figure
>> something else out :)
>>
>>
>>
>> On 3/23/09 9:56 AM, Carter Bullard wrote:
>>> Hey Jason,
>>> You are printing the suser and duser fields (not sdata and ddata).
>>> The spacing is there because you are specifying the length to be
>>> 128, so its 128. If you want to parse the strings, change the field
>>> separator, using the "-c <char>" option (see the ra.1 man page).
>>>
>>> Regardless, depending on the encoding of the data buffer, we
>>> should be 'quoting' the strings if you're printing in ascii, so I'll
>>> take
>>> a look at that.
>>>
>>> The debug messages you sent indicate that the Bivio is closing
>>> the connection:
>>>
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242288 ArgusHandleDatum
>>>>> (0xadf1b0, 0x7c26a38) received closing Mar
>>>
>>> But, if radium() is crashing after that, then that is not good. It
>>> should recoverso I'll look into the memory problem.
>>>
>>> The kind of problems that you are describing, (other than the crashes)
>>> suggest that you may not have your system architected the way
>>> you would like. The only way to understand why you are having
>>> problems, is for you to completely describe what you have done.
>>> argus() running on X listening on port abcde with these options
>>> (argus.conf)
>>> radium() running on Y attaching to argus() on system X:port abcde,
>>> listening
>>> on port vwxyz, with these options (radium.conf). Clients running on Z
>>> attaching to radium on port vwyxz (etc).
>>>
>>> I need to see all command line options, to make sure that those are
>>> correct.
>>>
>>> When there is a problem, such as in this case, what syslog
>>> error/messages
>>> are generated by each component, etc....., need to be checked.
>>>
>>> And, if all is lost, if I can access the system, I can usually get it
>>> going
>>> in short time.
>>>
>>>
>>> Sorry for the delayed response,
>>>
>>> Carter
>>>
>>> On Mar 19, 2009, at 4:15 PM, Jason Carr wrote:
>>>
>>>> Now that I'm directly connecting to the argus server from the client,
>>>> rasplit from 3.0.2 works just fine. 3.0.2.beta.2 crashes.
>>>>
>>>> Does it take awhile for rasplit to close the file after it's been
>>>> completely written to? I've had times were ra only will read ~800
>>>> flows and exit because there was an error:
>>>>
>>>> ra[1576]: 09:29:10.582079 ArgusReadStreamSocket (0xf2b0) record length
>>>> is zero
>>>>
>>>> With the newer clients, we can't use the -s user -d s128:d128 like
>>>> before. Now I'm using -s ddata:128 -s sdata:128. This is interesting
>>>> because it looks like it still leaves 128 characters of space for the
>>>> source even if the source didn't need it. I also noticed that the data
>>>> fields don't have quotes around them any more. Can these be
>>>> enabled/disabled (maybe I had them enabled on my older box)?
>>>>
>>>> - Jason
>>>>
>>>> On 3/18/09 12:54 PM, Jason Carr wrote:
>>>>> Connecting directly yields this error after a minute or so of running:
>>>>>
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241519 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 1312 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241643 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 1448 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241778 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 2088 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241933 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 1448 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242006 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 880 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242168 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 1448 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242243 ArgusReadStreamSocket
>>>>> (0x7ba4010) read 1128 bytes
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242263 ArgusCopyRecordStruct
>>>>> (0x7ba4600) retn 0xf8007710
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242277 RaProcessRecord
>>>>> (0x7c06010, 0x7ba4600) returning
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242288 ArgusHandleDatum
>>>>> (0xadf1b0, 0x7c26a38) received closing Mar
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242300
>>>>> ArgusCloseInput(0x7ba4010) closing
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242310 ArgusWriteConnection:
>>>>> write(6, 0x49d7a9, 6)
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242330
>>>>> ArgusWriteConnection(0x7ba4010, 0x49d7a9, 6) returning 6
>>>>> radium[29333.5019cf4100000000]: 12:45:44.242368 ArgusOutputProcess()
>>>>> received mar 0xf8007710 totals 129200 count 0 remaining 0
>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242381
>>>>> ArgusCloseInput(0x7ba4010) done
>>>>> radium[29333.5019cf4100000000]: 12:45:44.242385 ArgusCopyRecordStruct
>>>>> (0xf8007710) retn 0xb2aa60
>>>>> radium[29333.5019cf4100000000]: 12:45:44.242448 ArgusWriteOutSocket
>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 128
>>>>> radium[29333.50d9d15700000000]: 12:45:44.932325
>>>>> ArgusConnectRemote(0x7ba4010) starting
>>>>> radium[29333.50d9d15700000000]: 12:45:44.932362 Trying 172.19.41.41
>>>>> port
>>>>> 561 Expecting Argus records
>>>>> radium[29333.50d9d15700000000]: 12:45:44.932659 connected
>>>>> radium[29333.50d9d15700000000]: 12:45:44.932675 ArgusGetServerSocket
>>>>> (0x7ba4010) returning 6
>>>>> radium[29333.50d9d15700000000]: 12:45:44.944349 ArgusReadConnection()
>>>>> read 16 bytes
>>>>> *** glibc detected *** radium: double free or corruption (out):
>>>>> 0x0000000000b3e900 ***
>>>>> ======= Backtrace: =========
>>>>> /lib/libc.so.6[0x7f6b0709508a]
>>>>> /lib/libc.so.6(cfree+0x8c)[0x7f6b07098c1c]
>>>>> radium[0x44c8d9]
>>>>> radium[0x4579c8]
>>>>> radium[0x453511]
>>>>> radium[0x4602e0]
>>>>> /lib/libpthread.so.0[0x7f6b073893f7]
>>>>> /lib/libc.so.6(clone+0x6d)[0x7f6b070f8b3d]
>>>>>
>>>>>
>>>>> On 3/18/09 1:50 AM, Jason Carr wrote:
>>>>>> The Bivio unit has 12 cores, each has a, for a lack of a better
>>>>>> phrase,
>>>>>> virtual machine, that is connected to the head node via an internal
>>>>>> network. I just added an iptables rule to forward the connection from
>>>>>> radium on the x86_64 box to the Bivio argus server.
>>>>>>
>>>>>> If that works long term I'll try out the various rasplit and rastream
>>>>>> stuff and let you know how it turns out.
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> - Jason
>>>>>>
>>>>>> On 3/16/09 2:31 PM, Carter Bullard wrote:
>>>>>>> Jason,
>>>>>>> Very confusing, why do you run client software on the bivio?
>>>>>>> Cater
>>>>>>>
>>>>>>> On Mar 16, 2009, at 2:02 PM, Jason Carr wrote:
>>>>>>>
>>>>>>>> Yep, sorry I meant -S host:port.
>>>>>>>>
>>>>>>>> What I'm trying to do is to store the incoming data stream on disk.
>>>>>>>> Currently we have the files up into the directory structure like
>>>>>>>> this:
>>>>>>>>
>>>>>>>> /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>
>>>>>>>> And split it up every 5 minutes. So we get 12 5 minute files in
>>>>>>>> each
>>>>>>>> hour. This works out well for us since they are much smaller files
>>>>>>>> than running through an hour or days worth of data.
>>>>>>>>
>>>>>>>> -D 4 output is this on radium:
>>>>>>>>
>>>>>>>> *a ton of other lines*
>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.865875 ArgusWriteOutSocket
>>>>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 328
>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866297
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x4ff34010) read 400 bytes
>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866321
>>>>>>>> ArgusCopyRecordStruct
>>>>>>>> (0x4ff34600) retn 0x4801f310
>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866336 RaProcessRecord
>>>>>>>> (0x4ff96010, 0x4ff34600) returning
>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866373
>>>>>>>> ArgusCopyRecordStruct
>>>>>>>> (0x4801f310) retn 0xb06310
>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866392 ArgusGenerateRecord
>>>>>>>> (0xb06310, 0) old len 400 new len 380
>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866411 ArgusWriteOutSocket
>>>>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 400
>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.868431
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x4ff34010) read 100 bytes
>>>>>>>>
>>>>>>>> I noticed that the file wasn't moved when that happened. I'm
>>>>>>>> starting
>>>>>>>> to think it has nothing to do with the file moving.
>>>>>>>>
>>>>>>>> I upgraded the bivio software from "argus-clients-3.0.2" to
>>>>>>>> "argus-clients-3.0.2.beta.2" and it's consistently crashing now.
>>>>>>>> 3.0.2
>>>>>>>> never crashed previously.
>>>>>>>>
>>>>>>>> *** glibc detected *** radium: double free or corruption (out):
>>>>>>>> 0x3350a008 ***
>>>>>>>> ======= Backtrace: =========
>>>>>>>> /lib/libc.so.6[0xfd4a964]
>>>>>>>> /lib/libc.so.6(__libc_free+0xc8)[0xfd4ded8]
>>>>>>>> radium[0x1004d500]
>>>>>>>> radium[0x10055ac4]
>>>>>>>> radium[0x10062ebc]
>>>>>>>> /lib/libpthread.so.0[0xfe55994]
>>>>>>>> /lib/libc.so.6(__clone+0x84)[0xfdb7794]
>>>>>>>>
>>>>>>>> - Jason
>>>>>>>>
>>>>>>>> On 3/16/09 12:50 PM, Carter Bullard wrote:
>>>>>>>>> Hey Jason,
>>>>>>>>> Its not "-R host:port" its "-S host:port". I'm sure that was a
>>>>>>>>> slip?
>>>>>>>>> So things look like they are working for rasplit() then, and with
>>>>>>>>> your Bivio box.
>>>>>>>>>
>>>>>>>>> So I cannot replicate your problem on any machine, and I've
>>>>>>>>> tested it on 8 different machines now. I do not recommend
>>>>>>>>> radium writing out to a file when there are so many other ways
>>>>>>>>> to manage the data streams. Why are you doing this? or rather
>>>>>>>>> What are you trying to do?
>>>>>>>>>
>>>>>>>>> There are a lot of ways to misconfigure a radium. Having it
>>>>>>>>> connect to itself is the common bad thing, and we try to catch
>>>>>>>>> this, but there are clever ways of getting around that ;o)
>>>>>>>>>
>>>>>>>>> Usually radium doesn't update a moved file, because there
>>>>>>>>> are no new records coming in. It doesn't know to do anything
>>>>>>>>> until a data record shows up, so more than likely your server
>>>>>>>>> connection is poorly configured.
>>>>>>>>>
>>>>>>>>> The best way to see what's going on, is to run with something
>>>>>>>>> like "-D 4" to see the debug information, to assure that you
>>>>>>>>> are attaching properly, that data is coming in, and that it
>>>>>>>>> see's that the file is gone.
>>>>>>>>>
>>>>>>>>> So run with -D 4 and then delete the file, and see what it sez.
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mar 16, 2009, at 11:36 AM, Jason Carr wrote:
>>>>>>>>>
>>>>>>>>>> Carter,
>>>>>>>>>>
>>>>>>>>>> For me, rasplit only wrote one file for 5 minutes after giving
>>>>>>>>>> rasplit
>>>>>>>>>> the -R x.x.x.x:561 option. It doesn't exit after the 5 minutes
>>>>>>>>>> anymore, not sure why though. Is there a way to do that?
>>>>>>>>>>
>>>>>>>>>> I still have the issue of radium not letting go of the file when
>>>>>>>>>> it's
>>>>>>>>>> been moved.
>>>>>>>>>>
>>>>>>>>>> - Jason
>>>>>>>>>>
>>>>>>>>>> On 3/12/09 12:51 PM, Carter Bullard wrote:
>>>>>>>>>>> Hey Jason,
>>>>>>>>>>> Hmmmm, so I can't replicate your problem here on any machine.
>>>>>>>>>>> If you can run rasplit() under gdb, (and you can, of course
>>>>>>>>>>> generate
>>>>>>>>>>> files with a shorter granularity), maybe you will catch the
>>>>>>>>>>> rasplit()
>>>>>>>>>>> exiting. Be sure and ./configure and compile with these tag
>>>>>>>>>>> files
>>>>>>>>>>> in the root directory:
>>>>>>>>>>>
>>>>>>>>>>> % touch .devel .debug
>>>>>>>>>>> % ./configure;make clean;make
>>>>>>>>>>> % gdb bin/rasplit
>>>>>>>>>>>
>>>>>>>>>>> For rasplit() do you have write permission to the entire
>>>>>>>>>>> directory
>>>>>>>>>>> structure that you are trying to write to?
>>>>>>>>>>>
>>>>>>>>>>> For rastream(), you do need the $srcid in the path, at least
>>>>>>>>>>> right
>>>>>>>>>>> now
>>>>>>>>>>> you do, so you should add it.
>>>>>>>>>>>
>>>>>>>>>>> rastream ....... -w
>>>>>>>>>>> /data/argus/core/\$srcid/%Y/%m/%d/argus.........
>>>>>>>>>>>
>>>>>>>>>>> Carter
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mar 12, 2009, at 1:38 AM, Jason Carr wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'm running all of this on a x86_64 Linux box.
>>>>>>>>>>>>
>>>>>>>>>>>> When I run rasplit like this:
>>>>>>>>>>>>
>>>>>>>>>>>> # rasplit -S argus-source:561 -M time 5m -w
>>>>>>>>>>>> /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>>>>>
>>>>>>>>>>>> It only creates one 5 minute file and then exits. Can I make it
>>>>>>>>>>>> just
>>>>>>>>>>>> keep listening and keep on writing the 5 minute files?
>>>>>>>>>>>>
>>>>>>>>>>>> rastream segfaults when I run it:
>>>>>>>>>>>>
>>>>>>>>>>>> # rastream -S argus-source:561 -M time 5m -w
>>>>>>>>>>>> '/data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S' -B 15s -f
>>>>>>>>>>>> ~/argus/test.sh
>>>>>>>>>>>> rastream[5979]: 01:34:40.205530 output string requires $srcid
>>>>>>>>>>>> Segmentation fault
>>>>>>>>>>>>
>>>>>>>>>>>> We've historically always used radium to write to a file and
>>>>>>>>>>>> move
>>>>>>>>>>>> the
>>>>>>>>>>>> file every 5 minutes. radium has always just recreates the
>>>>>>>>>>>> output
>>>>>>>>>>>> file
>>>>>>>>>>>> once it's been moved. It is a little bit of a hack and if
>>>>>>>>>>>> rastream/rasplit already does it, that would be awesome.
>>>>>>>>>>>> I'll use
>>>>>>>>>>>> either at this point :)
>>>>>>>>>>>>
>>>>>>>>>>>> - Jason
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>>> I'm answering on the mailing list, so we capture the thread.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Glad to hear that your argus-2.x data on your Bivio box is
>>>>>>>>>>>>> working well for you. Sorry about your radium() file behavior.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I still can't duplicate your error, but I'll try again later
>>>>>>>>>>>>> today.
>>>>>>>>>>>>> What kind of machine is your radium() running on?
>>>>>>>>>>>>>
>>>>>>>>>>>>> To get rasplit to generate the files run:
>>>>>>>>>>>>> rasplit -S radium -M time 5m \
>>>>>>>>>>>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>>>>>>
>>>>>>>>>>>>> you can run it as a daemon with the "-d" option. Some like to
>>>>>>>>>>>>> run
>>>>>>>>>>>>> it from a script that respawns it if it dies, but its been a
>>>>>>>>>>>>> really
>>>>>>>>>>>>> stable
>>>>>>>>>>>>> program.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The reason you want to use rasplit(), is that it will actually
>>>>>>>>>>>>> clip
>>>>>>>>>>>>> argus records to ensure that all the argus data for a
>>>>>>>>>>>>> particular
>>>>>>>>>>>>> time
>>>>>>>>>>>>> frame is only in the specific timed file. So if a record
>>>>>>>>>>>>> starts at
>>>>>>>>>>>>> 12:04:12.345127 and ends at 12:06:12.363131, rasplit will clip
>>>>>>>>>>>>> that record into 2 records with timestamps:
>>>>>>>>>>>>> 12:04:12.345127 - 12:05:00.000000
>>>>>>>>>>>>> 12:05:00.000000 - 12:06:12.363131
>>>>>>>>>>>>>
>>>>>>>>>>>>> This makes it so you can process, graph the files, and you
>>>>>>>>>>>>> don't
>>>>>>>>>>>>> have to worry about whether you've got all the data or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Argus does a good job keeping its output records in time
>>>>>>>>>>>>> order, but if you're processing multiple streams of data, the
>>>>>>>>>>>>> time order can get a bit skewed. So that when you pull the
>>>>>>>>>>>>> file
>>>>>>>>>>>>> out from underneath radium() you can still get records that
>>>>>>>>>>>>> have data that should be in another file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To compress the data, have a cron job compress the files say
>>>>>>>>>>>>> an hour later.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you want to experiment, use rastream(), which is rasplit()
>>>>>>>>>>>>> with
>>>>>>>>>>>>> the addition feature to run a shell script against files after
>>>>>>>>>>>>> they
>>>>>>>>>>>>> close.
>>>>>>>>>>>>> You need to tell rastream() how long to wait before running
>>>>>>>>>>>>> the
>>>>>>>>>>>>> script,
>>>>>>>>>>>>> so you know that all the records for a particular time period
>>>>>>>>>>>>> are
>>>>>>>>>>>>> "in".
>>>>>>>>>>>>> If you are collecting netflow records, that time period
>>>>>>>>>>>>> could be
>>>>>>>>>>>>> hours
>>>>>>>>>>>>> and in some cases days (so don't use rastream for netflow
>>>>>>>>>>>>> data).
>>>>>>>>>>>>>
>>>>>>>>>>>>> rastream -S radium -M time 5m \
>>>>>>>>>>>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S \
>>>>>>>>>>>>> -B 15s -f /usr/local/bin/rastream.sh
>>>>>>>>>>>>>
>>>>>>>>>>>>> So the "-B 15s" option indicates that rastream() should close
>>>>>>>>>>>>> and
>>>>>>>>>>>>> process its files 15 seconds after a 5 minute boundary is
>>>>>>>>>>>>> passed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The sample rastream.sh in ./support/Config compresses
>>>>>>>>>>>>> files. You
>>>>>>>>>>>>> can do anything you like in there, but do test ;o)
>>>>>>>>>>>>>
>>>>>>>>>>>>> The shell script will be passed a single parameter, the full
>>>>>>>>>>>>> pathname
>>>>>>>>>>>>> of the file that is being closed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have not done extensive testing on this specific version of
>>>>>>>>>>>>> rastream(),
>>>>>>>>>>>>> but something like it has been running for years in a
>>>>>>>>>>>>> number of
>>>>>>>>>>>>> sites,
>>>>>>>>>>>>> so please give it a try, and if you have any problems,
>>>>>>>>>>>>> holler!!!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 11, 2009, at 12:18 AM, Jason Carr wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pretty simple file:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> RADIUM_DAEMON=no
>>>>>>>>>>>>>> RADIUM_MONITOR_ID=4
>>>>>>>>>>>>>> RADIUM_MAR_STATUS_INTERVAL=60
>>>>>>>>>>>>>> RADIUM_ARGUS_SERVER=bivio-01.iso.cmu.local:561
>>>>>>>>>>>>>> RADIUM_ACCESS_PORT=561
>>>>>>>>>>>>>> RADIUM_BIND_IP=127.0.0.1
>>>>>>>>>>>>>> RADIUM_OUTPUT_FILE=/data/argus/var/core.out
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> radium is ran like this: ./radium
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does rasplit split out files into a directory structure like
>>>>>>>>>>>>>> this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /data/argus/core/2009/03/03/22/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With 5 minute files that are gzipped inside of that directory
>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>> entire hour.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>>>>> So I can't reproduce your situation on any of my machines
>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>> How are you running radium()? Do you have a /etc/radium.conf
>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>> If so can you send it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mar 10, 2009, at 1:21 PM, Jason Carr wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Good news, it seemed to have fixed my problem, thank you
>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>> much.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Bad news, there seems to be a problem with writing to the
>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Did the way files are written to change in this version?
>>>>>>>>>>>>>>>> Right
>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>> we're telling radium to write to one file and then every 5
>>>>>>>>>>>>>>>> minutes
>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>> the file. Previously, radium would recreate the file but
>>>>>>>>>>>>>>>> now it
>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>> not and lsof showed that the file was still open by radium.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Jason
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>>> There is a good chance that it will. But the Bivio part is
>>>>>>>>>>>>>>>>> still an
>>>>>>>>>>>>>>>>> unknown, so if it doesn't we'll still have work to do.
>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mar 6, 2009, at 1:48 PM, Jason Carr wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Carter,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do you think that this would be a fix for my problem?
>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>> try it
>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>> regardless... :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>>>>> Gentle people,
>>>>>>>>>>>>>>>>>>> I have found a bug in the little-endian processing of
>>>>>>>>>>>>>>>>>>> 2.x
>>>>>>>>>>>>>>>>>>> data in
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> argus-clients-3.0.2 code, so I have uploaded a new
>>>>>>>>>>>>>>>>>>> argus-clients-3.0.2.beta.2.tar.gz file to the dev
>>>>>>>>>>>>>>>>>>> directory.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.2.beta.2.tar.gz
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have tested reading data from as many argus files as I
>>>>>>>>>>>>>>>>>>> have,
>>>>>>>>>>>>>>>>>>> regardless
>>>>>>>>>>>>>>>>>>> of version, and it seems to work, ......
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I also fixed problems with racluster() results having
>>>>>>>>>>>>>>>>>>> short
>>>>>>>>>>>>>>>>>>> counts.
>>>>>>>>>>>>>>>>>>> I have gone through all the clients and revised their
>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> closing
>>>>>>>>>>>>>>>>>>> strategies,
>>>>>>>>>>>>>>>>>>> and so ....
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This does not fix the "-lz" issue for some ./configure
>>>>>>>>>>>>>>>>>>> runs.......
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For those having problems, please grab this set of code
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> testing.
>>>>>>>>>>>>>>>>>>> And don't hesitate to report any problems you find!!!!!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks!!!!!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Carter Bullard
>>>>>>> CEO/President
>>>>>>> QoSient, LLC
>>>>>>> 150 E 57th Street Suite 12D
>>>>>>> New York, New York 10022
>>>>>>>
>>>>>>> +1 212 588-9133 Phone
>>>>>>> +1 212 588-9134 Fax
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York 10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>