little endian bug in 2.x record parsing in new clients
Carter Bullard
carter at qosient.com
Mon Mar 30 09:11:52 EDT 2009
Hey Jason,
So, where are we? Are you working? Is there a chance for me to log
onto this thing to figure it out?
Carter
On Mar 23, 2009, at 10:54 PM, Jason Carr wrote:
> There is a call inside of the new argus-3.0 code that their older
> pcap doesn't implement until their next OS release so I'm stuck on
> 2.6.
>
> On 3/23/09 8:06 PM, Carter Bullard wrote:
>> Hey Jason,
>> If argus-2.x could read the interface, argus-3.0 should be able to
>> do it
>> as well,
>> since they both should be using the same libpcap library. If it can't
>> open the
>> interface, then maybe it linked to the wrong libpcap?
>>
>> argus-3.0 can open many interfaces at a time, the ARGUS_INTERFACE
>> entries
>> in the new ./support/Config/argus.conf file will guide you on what
>> to do.
>>
>> Carter
>>
>> On Mar 23, 2009, at 1:55 PM, Jason Carr wrote:
>>
>>> Also, default uses all interfaces available to each machine. For
>>> example, we have two 10G interfaces that have two halves of our
>>> core.
>>> We were using the default interface for this reason. Since we
>>> can't do
>>> that I think we can get argus to read two interfaces. If not, we'll
>>> run two argoose (plural argus...) and hopefully rasplit can read
>>> multiple streams. Any recommendations for that?
>>>
>>> On 3/23/09 12:54 PM, Carter Bullard wrote:
>>>> Hey Jason,
>>>> A few things you should change.
>>>>
>>>> All the things you have on the command line can be set in the
>>>> argus.conf
>>>> file, that would be a great place to set them, unless they change
>>>> with
>>>> every
>>>> run.
>>>>
>>>> Don't use "-F /etc/argus.conf". Argus always reads the
>>>> /etc/argus.conf, and
>>>> by putting a "-F /etc/argus.conf" it is reading it twice, and thus
>>>> setting the
>>>> bind addresses twice, which means its opening twice, etc.... This
>>>> is not
>>>> good.
>>>>
>>>> You have the ARGUS_MONITOR_ID set in the argus.conf file, but you
>>>> are
>>>> changing it on the command line, and then you re-read the
>>>> argus.conf
>>>> file.
>>>> What MONITOR_ID is actually being used would be interesting to
>>>> know.
>>>>
>>>> I have no idea what "-i default" will do. Do you have a /dev/
>>>> default
>>>> device?
>>>>
>>>>
>>>> On Mar 23, 2009, at 11:47 AM, Jason Carr wrote:
>>>>
>>>>> For argus:
>>>>>
>>>>> /usr/sbin/argus -P 561 -B 10.10.10.103 -e CPU-2c1 -i default -F
>>>>> /etc/argus.conf
>>>>>
>>>>> argus.conf:
>>>>> ARGUS_DAEMON=no
>>>>> ARGUS_MAX_INSTANCES=1
>>>>> ARGUS_SET_PID=no
>>>>> ARGUS_PID_FILENAME=/var/run/argus.pid
>>>>> ARGUS_MONITOR_ID=`hostname`
>>>>> ARGUS_ACCESS_PORT=561
>>>>> ARGUS_BIND_IP="127.0.0.1"
>>>>> ARGUS_GO_PROMISCUOUS=yes
>>>>> ARGUS_FLOW_STATUS_INTERVAL=60
>>>>> ARGUS_MAR_STATUS_INTERVAL=300
>>>>> ARGUS_GENERATE_RESPONSE_TIME_DATA=no
>>>>> ARGUS_GENERATE_JITTER_DATA=no
>>>>> ARGUS_GENERATE_MAC_DATA=no
>>>>> ARGUS_CAPTURE_DATA_LEN=128
>>>>> ARGUS_FILTER_OPTIMIZER=yes
>>>>> ARGUS_FILTER=""
>>>>>
>>>>> argus runs on an internal IP address 10.10.10.103. The
>>>>> connection from
>>>>> the client machine running rasplit comes from an external IP (lets
>>>>> call that argus-sensor) port forwarded to 10.10.10.103. Lets
>>>>> just call
>>>>> it argus-collector. argus-collector has the following
>>>>> configuration:
>>>>>
>>>>> ./rasplit -S argus-sensor:561 -M time 5m -w
>>>>> /data/argus/core/%Y/%m/%d/%H/core.%Y.%m.%d.%H.%M.%S.argus
>>>>
>>>>
>>>> The best way to talk about this is that argus "binds" to
>>>> 10.10.10.103.
>>>> That is much different than
>>>> "argus runs on 10.10.10.103", especially in multi-processor/multi-
>>>> core
>>>> environments.
>>>>
>>>> I'm concerned about the port forwarding, why not just bind argus
>>>> to the
>>>> 172.19.41.41 address?
>>>> I think when you are having problems, that all this complexity is
>>>> just
>>>> going
>>>> to be in the way?
>>>>
>>>>>
>>>>>
>>>>>
>>>>> So essentially, we're doing this:
>>>>>
>>>>>
>>>>> +---------------+ +---------------+
>>>>> | 10.10.10.103 |<->| 10.10.10.1 |
>>>>> | argus TCP/561 | | port forward |
>>>>> +---------------+ +---------------+
>>>>> ^
>>>>> | tcp
>>>>> \/ 561
>>>>> +---------------+ +---------------+
>>>>> | 172.19.41.42 |<->| 172.19.41.41 |
>>>>> | rasplit | | port forward |
>>>>> +---------------+ +---------------+
>>>>>
>>>>> 172.19.41.41, 10.10.10.1, 10.10.10.103 are all the same physical
>>>>> unit
>>>>> and are all PowerPC arcitecture. 10.10.10.1 and 172.19.41.41 is
>>>>> the
>>>>> same machine.
>>>>>
>>>>> 172.19.41.42 is an x86_64 machine.
>>>>>
>>>>>
>>>>> Hopefully this makes sense. If it doesn't let me know and I'll
>>>>> figure
>>>>> something else out :)
>>>>>
>>>>>
>>>>>
>>>>> On 3/23/09 9:56 AM, Carter Bullard wrote:
>>>>>> Hey Jason,
>>>>>> You are printing the suser and duser fields (not sdata and
>>>>>> ddata).
>>>>>> The spacing is there because you are specifying the length to be
>>>>>> 128, so its 128. If you want to parse the strings, change the
>>>>>> field
>>>>>> separator, using the "-c <char>" option (see the ra.1 man page).
>>>>>>
>>>>>> Regardless, depending on the encoding of the data buffer, we
>>>>>> should be 'quoting' the strings if you're printing in ascii, so
>>>>>> I'll
>>>>>> take
>>>>>> a look at that.
>>>>>>
>>>>>> The debug messages you sent indicate that the Bivio is closing
>>>>>> the connection:
>>>>>>
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242288
>>>>>>>> ArgusHandleDatum
>>>>>>>> (0xadf1b0, 0x7c26a38) received closing Mar
>>>>>>
>>>>>> But, if radium() is crashing after that, then that is not good.
>>>>>> It
>>>>>> should recoverso I'll look into the memory problem.
>>>>>>
>>>>>> The kind of problems that you are describing, (other than the
>>>>>> crashes)
>>>>>> suggest that you may not have your system architected the way
>>>>>> you would like. The only way to understand why you are having
>>>>>> problems, is for you to completely describe what you have done.
>>>>>> argus() running on X listening on port abcde with these options
>>>>>> (argus.conf)
>>>>>> radium() running on Y attaching to argus() on system X:port
>>>>>> abcde,
>>>>>> listening
>>>>>> on port vwxyz, with these options (radium.conf). Clients
>>>>>> running on Z
>>>>>> attaching to radium on port vwyxz (etc).
>>>>>>
>>>>>> I need to see all command line options, to make sure that those
>>>>>> are
>>>>>> correct.
>>>>>>
>>>>>> When there is a problem, such as in this case, what syslog
>>>>>> error/messages
>>>>>> are generated by each component, etc....., need to be checked.
>>>>>>
>>>>>> And, if all is lost, if I can access the system, I can usually
>>>>>> get it
>>>>>> going
>>>>>> in short time.
>>>>>>
>>>>>>
>>>>>> Sorry for the delayed response,
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>> On Mar 19, 2009, at 4:15 PM, Jason Carr wrote:
>>>>>>
>>>>>>> Now that I'm directly connecting to the argus server from the
>>>>>>> client,
>>>>>>> rasplit from 3.0.2 works just fine. 3.0.2.beta.2 crashes.
>>>>>>>
>>>>>>> Does it take awhile for rasplit to close the file after it's
>>>>>>> been
>>>>>>> completely written to? I've had times were ra only will read
>>>>>>> ~800
>>>>>>> flows and exit because there was an error:
>>>>>>>
>>>>>>> ra[1576]: 09:29:10.582079 ArgusReadStreamSocket (0xf2b0) record
>>>>>>> length
>>>>>>> is zero
>>>>>>>
>>>>>>> With the newer clients, we can't use the -s user -d s128:d128
>>>>>>> like
>>>>>>> before. Now I'm using -s ddata:128 -s sdata:128. This is
>>>>>>> interesting
>>>>>>> because it looks like it still leaves 128 characters of space
>>>>>>> for the
>>>>>>> source even if the source didn't need it. I also noticed that
>>>>>>> the
>>>>>>> data
>>>>>>> fields don't have quotes around them any more. Can these be
>>>>>>> enabled/disabled (maybe I had them enabled on my older box)?
>>>>>>>
>>>>>>> - Jason
>>>>>>>
>>>>>>> On 3/18/09 12:54 PM, Jason Carr wrote:
>>>>>>>> Connecting directly yields this error after a minute or so of
>>>>>>>> running:
>>>>>>>>
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241519
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 1312 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241643
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 1448 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241778
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 2088 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.241933
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 1448 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242006
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 880 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242168
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 1448 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242243
>>>>>>>> ArgusReadStreamSocket
>>>>>>>> (0x7ba4010) read 1128 bytes
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242263
>>>>>>>> ArgusCopyRecordStruct
>>>>>>>> (0x7ba4600) retn 0xf8007710
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242277 RaProcessRecord
>>>>>>>> (0x7c06010, 0x7ba4600) returning
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242288
>>>>>>>> ArgusHandleDatum
>>>>>>>> (0xadf1b0, 0x7c26a38) received closing Mar
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242300
>>>>>>>> ArgusCloseInput(0x7ba4010) closing
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242310
>>>>>>>> ArgusWriteConnection:
>>>>>>>> write(6, 0x49d7a9, 6)
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242330
>>>>>>>> ArgusWriteConnection(0x7ba4010, 0x49d7a9, 6) returning 6
>>>>>>>> radium[29333.5019cf4100000000]: 12:45:44.242368
>>>>>>>> ArgusOutputProcess()
>>>>>>>> received mar 0xf8007710 totals 129200 count 0 remaining 0
>>>>>>>> radium[29333.e096c4076b7f0000]: 12:45:44.242381
>>>>>>>> ArgusCloseInput(0x7ba4010) done
>>>>>>>> radium[29333.5019cf4100000000]: 12:45:44.242385
>>>>>>>> ArgusCopyRecordStruct
>>>>>>>> (0xf8007710) retn 0xb2aa60
>>>>>>>> radium[29333.5019cf4100000000]: 12:45:44.242448
>>>>>>>> ArgusWriteOutSocket
>>>>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 128
>>>>>>>> radium[29333.50d9d15700000000]: 12:45:44.932325
>>>>>>>> ArgusConnectRemote(0x7ba4010) starting
>>>>>>>> radium[29333.50d9d15700000000]: 12:45:44.932362 Trying
>>>>>>>> 172.19.41.41
>>>>>>>> port
>>>>>>>> 561 Expecting Argus records
>>>>>>>> radium[29333.50d9d15700000000]: 12:45:44.932659 connected
>>>>>>>> radium[29333.50d9d15700000000]: 12:45:44.932675
>>>>>>>> ArgusGetServerSocket
>>>>>>>> (0x7ba4010) returning 6
>>>>>>>> radium[29333.50d9d15700000000]: 12:45:44.944349
>>>>>>>> ArgusReadConnection()
>>>>>>>> read 16 bytes
>>>>>>>> *** glibc detected *** radium: double free or corruption (out):
>>>>>>>> 0x0000000000b3e900 ***
>>>>>>>> ======= Backtrace: =========
>>>>>>>> /lib/libc.so.6[0x7f6b0709508a]
>>>>>>>> /lib/libc.so.6(cfree+0x8c)[0x7f6b07098c1c]
>>>>>>>> radium[0x44c8d9]
>>>>>>>> radium[0x4579c8]
>>>>>>>> radium[0x453511]
>>>>>>>> radium[0x4602e0]
>>>>>>>> /lib/libpthread.so.0[0x7f6b073893f7]
>>>>>>>> /lib/libc.so.6(clone+0x6d)[0x7f6b070f8b3d]
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/18/09 1:50 AM, Jason Carr wrote:
>>>>>>>>> The Bivio unit has 12 cores, each has a, for a lack of a
>>>>>>>>> better
>>>>>>>>> phrase,
>>>>>>>>> virtual machine, that is connected to the head node via an
>>>>>>>>> internal
>>>>>>>>> network. I just added an iptables rule to forward the
>>>>>>>>> connection
>>>>>>>>> from
>>>>>>>>> radium on the x86_64 box to the Bivio argus server.
>>>>>>>>>
>>>>>>>>> If that works long term I'll try out the various rasplit and
>>>>>>>>> rastream
>>>>>>>>> stuff and let you know how it turns out.
>>>>>>>>>
>>>>>>>>> Thanks again,
>>>>>>>>>
>>>>>>>>> - Jason
>>>>>>>>>
>>>>>>>>> On 3/16/09 2:31 PM, Carter Bullard wrote:
>>>>>>>>>> Jason,
>>>>>>>>>> Very confusing, why do you run client software on the bivio?
>>>>>>>>>> Cater
>>>>>>>>>>
>>>>>>>>>> On Mar 16, 2009, at 2:02 PM, Jason Carr wrote:
>>>>>>>>>>
>>>>>>>>>>> Yep, sorry I meant -S host:port.
>>>>>>>>>>>
>>>>>>>>>>> What I'm trying to do is to store the incoming data stream
>>>>>>>>>>> on
>>>>>>>>>>> disk.
>>>>>>>>>>> Currently we have the files up into the directory
>>>>>>>>>>> structure like
>>>>>>>>>>> this:
>>>>>>>>>>>
>>>>>>>>>>> /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>>>>
>>>>>>>>>>> And split it up every 5 minutes. So we get 12 5 minute
>>>>>>>>>>> files in
>>>>>>>>>>> each
>>>>>>>>>>> hour. This works out well for us since they are much smaller
>>>>>>>>>>> files
>>>>>>>>>>> than running through an hour or days worth of data.
>>>>>>>>>>>
>>>>>>>>>>> -D 4 output is this on radium:
>>>>>>>>>>>
>>>>>>>>>>> *a ton of other lines*
>>>>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.865875
>>>>>>>>>>> ArgusWriteOutSocket
>>>>>>>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 328
>>>>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866297
>>>>>>>>>>> ArgusReadStreamSocket
>>>>>>>>>>> (0x4ff34010) read 400 bytes
>>>>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866321
>>>>>>>>>>> ArgusCopyRecordStruct
>>>>>>>>>>> (0x4ff34600) retn 0x4801f310
>>>>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.866336
>>>>>>>>>>> RaProcessRecord
>>>>>>>>>>> (0x4ff96010, 0x4ff34600) returning
>>>>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866373
>>>>>>>>>>> ArgusCopyRecordStruct
>>>>>>>>>>> (0x4801f310) retn 0xb06310
>>>>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866392
>>>>>>>>>>> ArgusGenerateRecord
>>>>>>>>>>> (0xb06310, 0) old len 400 new len 380
>>>>>>>>>>> radium[19615.50999a4000000000]: 13:34:16.866411
>>>>>>>>>>> ArgusWriteOutSocket
>>>>>>>>>>> (0xacd0a0, 0xacec40) 0 records waiting. returning 400
>>>>>>>>>>> radium[19615.e096fd4f7e7f0000]: 13:34:16.868431
>>>>>>>>>>> ArgusReadStreamSocket
>>>>>>>>>>> (0x4ff34010) read 100 bytes
>>>>>>>>>>>
>>>>>>>>>>> I noticed that the file wasn't moved when that happened. I'm
>>>>>>>>>>> starting
>>>>>>>>>>> to think it has nothing to do with the file moving.
>>>>>>>>>>>
>>>>>>>>>>> I upgraded the bivio software from "argus-clients-3.0.2" to
>>>>>>>>>>> "argus-clients-3.0.2.beta.2" and it's consistently
>>>>>>>>>>> crashing now.
>>>>>>>>>>> 3.0.2
>>>>>>>>>>> never crashed previously.
>>>>>>>>>>>
>>>>>>>>>>> *** glibc detected *** radium: double free or corruption
>>>>>>>>>>> (out):
>>>>>>>>>>> 0x3350a008 ***
>>>>>>>>>>> ======= Backtrace: =========
>>>>>>>>>>> /lib/libc.so.6[0xfd4a964]
>>>>>>>>>>> /lib/libc.so.6(__libc_free+0xc8)[0xfd4ded8]
>>>>>>>>>>> radium[0x1004d500]
>>>>>>>>>>> radium[0x10055ac4]
>>>>>>>>>>> radium[0x10062ebc]
>>>>>>>>>>> /lib/libpthread.so.0[0xfe55994]
>>>>>>>>>>> /lib/libc.so.6(__clone+0x84)[0xfdb7794]
>>>>>>>>>>>
>>>>>>>>>>> - Jason
>>>>>>>>>>>
>>>>>>>>>>> On 3/16/09 12:50 PM, Carter Bullard wrote:
>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>> Its not "-R host:port" its "-S host:port". I'm sure that
>>>>>>>>>>>> was a
>>>>>>>>>>>> slip?
>>>>>>>>>>>> So things look like they are working for rasplit() then,
>>>>>>>>>>>> and
>>>>>>>>>>>> with
>>>>>>>>>>>> your Bivio box.
>>>>>>>>>>>>
>>>>>>>>>>>> So I cannot replicate your problem on any machine, and I've
>>>>>>>>>>>> tested it on 8 different machines now. I do not recommend
>>>>>>>>>>>> radium writing out to a file when there are so many other
>>>>>>>>>>>> ways
>>>>>>>>>>>> to manage the data streams. Why are you doing this? or
>>>>>>>>>>>> rather
>>>>>>>>>>>> What are you trying to do?
>>>>>>>>>>>>
>>>>>>>>>>>> There are a lot of ways to misconfigure a radium. Having it
>>>>>>>>>>>> connect to itself is the common bad thing, and we try to
>>>>>>>>>>>> catch
>>>>>>>>>>>> this, but there are clever ways of getting around that ;o)
>>>>>>>>>>>>
>>>>>>>>>>>> Usually radium doesn't update a moved file, because there
>>>>>>>>>>>> are no new records coming in. It doesn't know to do
>>>>>>>>>>>> anything
>>>>>>>>>>>> until a data record shows up, so more than likely your
>>>>>>>>>>>> server
>>>>>>>>>>>> connection is poorly configured.
>>>>>>>>>>>>
>>>>>>>>>>>> The best way to see what's going on, is to run with
>>>>>>>>>>>> something
>>>>>>>>>>>> like "-D 4" to see the debug information, to assure that
>>>>>>>>>>>> you
>>>>>>>>>>>> are attaching properly, that data is coming in, and that it
>>>>>>>>>>>> see's that the file is gone.
>>>>>>>>>>>>
>>>>>>>>>>>> So run with -D 4 and then delete the file, and see what
>>>>>>>>>>>> it sez.
>>>>>>>>>>>>
>>>>>>>>>>>> Carter
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 16, 2009, at 11:36 AM, Jason Carr wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Carter,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For me, rasplit only wrote one file for 5 minutes after
>>>>>>>>>>>>> giving
>>>>>>>>>>>>> rasplit
>>>>>>>>>>>>> the -R x.x.x.x:561 option. It doesn't exit after the 5
>>>>>>>>>>>>> minutes
>>>>>>>>>>>>> anymore, not sure why though. Is there a way to do that?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I still have the issue of radium not letting go of the
>>>>>>>>>>>>> file
>>>>>>>>>>>>> when
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> been moved.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Jason
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/12/09 12:51 PM, Carter Bullard wrote:
>>>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>>>> Hmmmm, so I can't replicate your problem here on any
>>>>>>>>>>>>>> machine.
>>>>>>>>>>>>>> If you can run rasplit() under gdb, (and you can, of
>>>>>>>>>>>>>> course
>>>>>>>>>>>>>> generate
>>>>>>>>>>>>>> files with a shorter granularity), maybe you will catch
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> rasplit()
>>>>>>>>>>>>>> exiting. Be sure and ./configure and compile with these
>>>>>>>>>>>>>> tag
>>>>>>>>>>>>>> files
>>>>>>>>>>>>>> in the root directory:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> % touch .devel .debug
>>>>>>>>>>>>>> % ./configure;make clean;make
>>>>>>>>>>>>>> % gdb bin/rasplit
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For rasplit() do you have write permission to the entire
>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>> structure that you are trying to write to?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For rastream(), you do need the $srcid in the path, at
>>>>>>>>>>>>>> least
>>>>>>>>>>>>>> right
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>> you do, so you should add it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rastream ....... -w
>>>>>>>>>>>>>> /data/argus/core/\$srcid/%Y/%m/%d/argus.........
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 12, 2009, at 1:38 AM, Jason Carr wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm running all of this on a x86_64 Linux box.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I run rasplit like this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # rasplit -S argus-source:561 -M time 5m -w
>>>>>>>>>>>>>>> /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It only creates one 5 minute file and then exits. Can I
>>>>>>>>>>>>>>> make it
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> keep listening and keep on writing the 5 minute files?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rastream segfaults when I run it:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # rastream -S argus-source:561 -M time 5m -w
>>>>>>>>>>>>>>> '/data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S' -B
>>>>>>>>>>>>>>> 15s -f
>>>>>>>>>>>>>>> ~/argus/test.sh
>>>>>>>>>>>>>>> rastream[5979]: 01:34:40.205530 output string requires
>>>>>>>>>>>>>>> $srcid
>>>>>>>>>>>>>>> Segmentation fault
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We've historically always used radium to write to a
>>>>>>>>>>>>>>> file and
>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> file every 5 minutes. radium has always just recreates
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>> once it's been moved. It is a little bit of a hack and
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>> rastream/rasplit already does it, that would be awesome.
>>>>>>>>>>>>>>> I'll use
>>>>>>>>>>>>>>> either at this point :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Jason
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>>>>>> I'm answering on the mailing list, so we capture the
>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Glad to hear that your argus-2.x data on your Bivio
>>>>>>>>>>>>>>>> box is
>>>>>>>>>>>>>>>> working well for you. Sorry about your radium() file
>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I still can't duplicate your error, but I'll try
>>>>>>>>>>>>>>>> again later
>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>> What kind of machine is your radium() running on?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To get rasplit to generate the files run:
>>>>>>>>>>>>>>>> rasplit -S radium -M time 5m \
>>>>>>>>>>>>>>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> you can run it as a daemon with the "-d" option. Some
>>>>>>>>>>>>>>>> like to
>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>> it from a script that respawns it if it dies, but its
>>>>>>>>>>>>>>>> been a
>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>> program.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The reason you want to use rasplit(), is that it will
>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>> clip
>>>>>>>>>>>>>>>> argus records to ensure that all the argus data for a
>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>> frame is only in the specific timed file. So if a
>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>> starts at
>>>>>>>>>>>>>>>> 12:04:12.345127 and ends at 12:06:12.363131, rasplit
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> clip
>>>>>>>>>>>>>>>> that record into 2 records with timestamps:
>>>>>>>>>>>>>>>> 12:04:12.345127 - 12:05:00.000000
>>>>>>>>>>>>>>>> 12:05:00.000000 - 12:06:12.363131
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This makes it so you can process, graph the files,
>>>>>>>>>>>>>>>> and you
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>> have to worry about whether you've got all the data
>>>>>>>>>>>>>>>> or not.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Argus does a good job keeping its output records in
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>> order, but if you're processing multiple streams of
>>>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> time order can get a bit skewed. So that when you
>>>>>>>>>>>>>>>> pull the
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> out from underneath radium() you can still get
>>>>>>>>>>>>>>>> records that
>>>>>>>>>>>>>>>> have data that should be in another file.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To compress the data, have a cron job compress the
>>>>>>>>>>>>>>>> files say
>>>>>>>>>>>>>>>> an hour later.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you want to experiment, use rastream(), which is
>>>>>>>>>>>>>>>> rasplit()
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> the addition feature to run a shell script against
>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>> close.
>>>>>>>>>>>>>>>> You need to tell rastream() how long to wait before
>>>>>>>>>>>>>>>> running
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> script,
>>>>>>>>>>>>>>>> so you know that all the records for a particular time
>>>>>>>>>>>>>>>> period
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> "in".
>>>>>>>>>>>>>>>> If you are collecting netflow records, that time period
>>>>>>>>>>>>>>>> could be
>>>>>>>>>>>>>>>> hours
>>>>>>>>>>>>>>>> and in some cases days (so don't use rastream for
>>>>>>>>>>>>>>>> netflow
>>>>>>>>>>>>>>>> data).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> rastream -S radium -M time 5m \
>>>>>>>>>>>>>>>> -w /data/argus/core/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S \
>>>>>>>>>>>>>>>> -B 15s -f /usr/local/bin/rastream.sh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So the "-B 15s" option indicates that rastream() should
>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> process its files 15 seconds after a 5 minute
>>>>>>>>>>>>>>>> boundary is
>>>>>>>>>>>>>>>> passed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The sample rastream.sh in ./support/Config compresses
>>>>>>>>>>>>>>>> files. You
>>>>>>>>>>>>>>>> can do anything you like in there, but do test ;o)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The shell script will be passed a single parameter,
>>>>>>>>>>>>>>>> the full
>>>>>>>>>>>>>>>> pathname
>>>>>>>>>>>>>>>> of the file that is being closed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have not done extensive testing on this specific
>>>>>>>>>>>>>>>> version of
>>>>>>>>>>>>>>>> rastream(),
>>>>>>>>>>>>>>>> but something like it has been running for years in a
>>>>>>>>>>>>>>>> number of
>>>>>>>>>>>>>>>> sites,
>>>>>>>>>>>>>>>> so please give it a try, and if you have any problems,
>>>>>>>>>>>>>>>> holler!!!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mar 11, 2009, at 12:18 AM, Jason Carr wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Pretty simple file:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> RADIUM_DAEMON=no
>>>>>>>>>>>>>>>>> RADIUM_MONITOR_ID=4
>>>>>>>>>>>>>>>>> RADIUM_MAR_STATUS_INTERVAL=60
>>>>>>>>>>>>>>>>> RADIUM_ARGUS_SERVER=bivio-01.iso.cmu.local:561
>>>>>>>>>>>>>>>>> RADIUM_ACCESS_PORT=561
>>>>>>>>>>>>>>>>> RADIUM_BIND_IP=127.0.0.1
>>>>>>>>>>>>>>>>> RADIUM_OUTPUT_FILE=/data/argus/var/core.out
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> radium is ran like this: ./radium
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does rasplit split out files into a directory
>>>>>>>>>>>>>>>>> structure
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /data/argus/core/2009/03/03/22/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> With 5 minute files that are gzipped inside of that
>>>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>>>>> entire hour.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>>>> Hey Jason,
>>>>>>>>>>>>>>>>>> So I can't reproduce your situation on any of my
>>>>>>>>>>>>>>>>>> machines
>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>> How are you running radium()? Do you have a
>>>>>>>>>>>>>>>>>> /etc/radium.conf
>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>> If so can you send it?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mar 10, 2009, at 1:21 PM, Jason Carr wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Good news, it seemed to have fixed my problem,
>>>>>>>>>>>>>>>>>>> thank you
>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>> much.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Bad news, there seems to be a problem with writing
>>>>>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Did the way files are written to change in this
>>>>>>>>>>>>>>>>>>> version?
>>>>>>>>>>>>>>>>>>> Right
>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>> we're telling radium to write to one file and then
>>>>>>>>>>>>>>>>>>> every 5
>>>>>>>>>>>>>>>>>>> minutes
>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>> the file. Previously, radium would recreate the
>>>>>>>>>>>>>>>>>>> file but
>>>>>>>>>>>>>>>>>>> now it
>>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>> not and lsof showed that the file was still open by
>>>>>>>>>>>>>>>>>>> radium.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Jason
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>>>>>> There is a good chance that it will. But the Bivio
>>>>>>>>>>>>>>>>>>>> part is
>>>>>>>>>>>>>>>>>>>> still an
>>>>>>>>>>>>>>>>>>>> unknown, so if it doesn't we'll still have work
>>>>>>>>>>>>>>>>>>>> to do.
>>>>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mar 6, 2009, at 1:48 PM, Jason Carr wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Carter,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Do you think that this would be a fix for my
>>>>>>>>>>>>>>>>>>>>> problem?
>>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>>> try it
>>>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>>> regardless... :)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Carter Bullard wrote:
>>>>>>>>>>>>>>>>>>>>>> Gentle people,
>>>>>>>>>>>>>>>>>>>>>> I have found a bug in the little-endian
>>>>>>>>>>>>>>>>>>>>>> processing of
>>>>>>>>>>>>>>>>>>>>>> 2.x
>>>>>>>>>>>>>>>>>>>>>> data in
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> argus-clients-3.0.2 code, so I have uploaded a
>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>> argus-clients-3.0.2.beta.2.tar.gz file to the dev
>>>>>>>>>>>>>>>>>>>>>> directory.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.2.beta.2.tar.gz
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I have tested reading data from as many argus
>>>>>>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>> have,
>>>>>>>>>>>>>>>>>>>>>> regardless
>>>>>>>>>>>>>>>>>>>>>> of version, and it seems to work, ......
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I also fixed problems with racluster() results
>>>>>>>>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>>> short
>>>>>>>>>>>>>>>>>>>>>> counts.
>>>>>>>>>>>>>>>>>>>>>> I have gone through all the clients and revised
>>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>>> closing
>>>>>>>>>>>>>>>>>>>>>> strategies,
>>>>>>>>>>>>>>>>>>>>>> and so ....
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This does not fix the "-lz" issue for some ./
>>>>>>>>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>>>>>> runs.......
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For those having problems, please grab this set
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> testing.
>>>>>>>>>>>>>>>>>>>>>> And don't hesitate to report any problems you
>>>>>>>>>>>>>>>>>>>>>> find!!!!!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks!!!!!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Carter Bullard
>>>>>>>>>> CEO/President
>>>>>>>>>> QoSient, LLC
>>>>>>>>>> 150 E 57th Street Suite 12D
>>>>>>>>>> New York, New York 10022
>>>>>>>>>>
>>>>>>>>>> +1 212 588-9133 Phone
>>>>>>>>>> +1 212 588-9134 Fax
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Carter Bullard
>>>>>> CEO/President
>>>>>> QoSient, LLC
>>>>>> 150 E 57th Street Suite 12D
>>>>>> New York, New York 10022
>>>>>>
>>>>>> +1 212 588-9133 Phone
>>>>>> +1 212 588-9134 Fax
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Carter Bullard
>>>> CEO/President
>>>> QoSient, LLC
>>>> 150 E 57th Street Suite 12D
>>>> New York, New York 10022
>>>>
>>>> +1 212 588-9133 Phone
>>>> +1 212 588-9134 Fax
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>>
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
More information about the argus
mailing list