Rewrite of rastream
David Edelman
dedelman at iname.com
Sat Jun 8 20:08:36 EDT 2013
The last time that I tried to use 'not broadcast' the error message seemed to indicate it was looking at layer 2 rather than layer 3
If you added all of the variants from recount it would cover my need and more. I'd love to be able to populate one of my rasql tables with flows between unicast ipv4 addresses.
--Dave
-----Original Message-----
From: Carter Bullard [mailto:carter at qosient.com]
Sent: Saturday, June 08, 2013 2:08 PM
To: David Edelman
Cc: Terry Burton; argus-info at lists.andrew.cmu.edu
Subject: Re: [ARGUS] Rewrite of rastream
Hey Dave,
So we have multicast and broadcast. Local broadcast, right now, is relative to the sensor. Do these work as expected?
Adding unicast for ipV4 is easy. I think V6 is straight forward, other than I think there is local and non-local unicast ? What else would be good ?
racount, with the -M addr option reports on a lot of different IP address types. Should I add all of them?
Carter
On Jun 7, 2013, at 11:45 PM, "David Edelman" <dedelman at iname.com> wrote:
> Carter,
>
> I would love to have a layer-3 filter for unicast so that I could say ra - unicast and not net 10.0.0.0/24 to look for transit traffic. I would think that src and dst could be modifiers.
> layer 3 broadcast and multicast would be nice as well.
>
> --Dave
>
>
>> ----- Original Message -----
>> From: Carter Bullard
>> Sent: 05/31/13 03:22 PM
>> To: Terry Burton
>> Subject: Re: [ARGUS] Rewrite of rastream
>>
>> Hey Terry,
>> Thanks for the suggestion, it fits in great with what we were trying
>> to do in the color version of ratop(), which was how to identify
>> large numbers of different things, for coloring and ordering of
>> objects on the display. If we can color something, surely we can
>> figure out a way to filter, formulate, analyate (is that a word?),
>> etc …. with large configuration data.
>>
>> In 3.0.7.x we have the rarc variable RA_LOCAL, which is the basic
>> technology of rafilteraddr() and ralabel(), that allows all clients
>> to ingest at least one big list of addresses, and assign them to a
>> variable. Right now, that is LOCAL. Only the new ratop() uses this
>> variable right now, so my thought is that your suggestion would extend this feature to all clients ?
>>
>> Cool, just need to understand how to configure it, and how to specify
>> what to do with the data….. In ratop(), you can use the LOCAL
>> addresses to pick colors, and you can use the local designation as a
>> hint for flow direction, when you don't know, see RA_LOCAL_DIRECTION
>> in the new sample rarc file, ./support/Config/rarc.
>>
>> So you are thinking of giving all ra* clients a large address filtering capability?
>> How do we turn that on, in a general way? Do you want to do something
>> like…..
>>
>> ra - src $LOCAL and dst not $LOCAL
>>
>> as a filter? I'm in favor of improving the filters in any way, so
>> that's a fair request, although a little difficult to do with our
>> current compiler strategy, but not impossible….
>>
>> If you can figure out a way to assign a fixed variable, and provide
>> fixed configuration in the rarc file, (like the direction support) we
>> can do something really quickly.
>>
>> The other way to think about this problem of general identification,
>> is to use ralabel() to label the flows with the state you are looking
>> for, and then use standard label greping filters to do the actions
>> your interested in, in a pipeline. Radium is a labeler, so it can
>> mark your flows so your clients can figure it out later with regular expressions.
>>
>> Lots of options, so if you can help to define this I'll put it in….
>>
>> Thanks !!!!
>> Carter
>>
>>
>>
>> On May 31, 2013, at 7:41 AM, Terry Burton <tez at terryburton.co.uk> wrote:
>>
>>> On 30 May 2013 16:33, Carter Bullard <carter at qosient.com> wrote:
>>>> OK, so I've found a pretty bad bug in rastream() and rasqlinsert(),
>>>> based on your descriptions, but I'm sorry that what I've found is
>>>> not supportive of your thesis. This problem needs to be fixed, but
>>>> existing implementations can continue if you are not experiencing problems.
>>> <...snip...>
>>>> Since I'm rewriting it based on rasplit(), now would be the time to
>>>> put some new feature requests if there are any, and any discussions
>>>> as to what sucks and what works in rastream() would be great !!!!!
>>>
>>> Hi Carter,
>>>
>>> Since rasplit/rastream is fresh in your mind and you're soliciting
>>> feature requests...
>>>
>>> We monitor a sparsely populated /16 in which all authorised devices
>>> are pre-registered in a database. We perform near real-time analysis
>>> of the data using rafilteraddr to detect both when unregistered
>>> devices are connected to the network and when external hosts contact
>>> dark addresses during scanning/fingerprinting. We can then deploy
>>> appropriate responses (firewalling, shutting the switch port, ...)
>>>
>>> I would find it very helpful if the functionality of rafilteraddr
>>> were rolled into these clients (and perhaps the raclients in
>>> general) in such a way that when the address specification file is
>>> modified and indicated by the user (e.g. using SIGHUP, SIGUSR1, etc)
>>> the client can continue to process the data uninterrupted without
>>> having to restart some element of the real-time data processing chain.
>>>
>>>
>>> All the best,
>>>
>>> Terry
>>>
>>>
>>>> On May 29, 2013, at 3:23 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> Hey Matt,
>>>> Hmmmmmmmm, well, we need to get on the same page. Based on the last
>>>> set of information, I don't think that you are running the correct
>>>> code, as you are reporting segfault information without symbols,
>>>> which means you're running the wrong set of software. If you are
>>>> running the wrong software there, not sure that you are running the
>>>> right software anywhere. I can give
>>>> racluster()
>>>> bad filenames, files without permissions, etc…. and it doesn't segfault…..
>>>>
>>>> So we have to clean things up, then we can hypothesis what the real
>>>> problem is.
>>>>
>>>> Now, I have been digging into rastream() and rasqlinsert() code and
>>>> I've found an issue that may be related, but I need to have some
>>>> certainty with regard to your setup, before I take the leap to say
>>>> that I have a strategy…...
>>>>
>>>> So, we need to understand what is going on with your rastream() script.
>>>> One thing to do is to simplify the script, have it do some
>>>> extensive logging to a file in /tmp, or /var/tmp or somewhere you
>>>> like, so we can see what its doing…… Like printing out its
>>>> environment, what it thinks is its root directory, its path,
>>>> etc…...
>>>>
>>>> Carter
>>>>
>>>>
>>>> On May 29, 2013, at 1:43 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Hello carter,
>>>>
>>>> Thanks for replying.
>>>>
>>>> Rastream and racluster are 3.0.7.10.
>>>>
>>>>
>>>> Are you able to reproduce the issue when using the -d switch with rastream?
>>>> I can reproduce it on other systems.
>>>>
>>>> What do you think of my theory?
>>>>
>>>>
>>>> Hope this helps,
>>>>
>>>> Matt
>>>>
>>>>
>>>> On May 29, 2013, at 1:34 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>> Hey Matt,
>>>> So where are we on this? Its clear that you are not running the
>>>> version of code that you think you are running, so we need to clear
>>>> that up before proceeding any further ?
>>>>
>>>> Carter
>>>>
>>>>
>>>> On May 22, 2013, at 4:07 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Sorry about that, I was trying to be verbose in case you spotted
>>>> something interesting that indicated a clear problem.
>>>>
>>>> Thanks for getting back to me.
>>>>
>>>>
>>>> 1) racluster segfaulting:
>>>>
>>>> racluster is executed via the script assigned by the -f parameter
>>>> of the rastream instance (/usr/local/bin/rastream.sh).
>>>>
>>>> When racluster segfaults, its reporting the following to syslog:
>>>> May 22 01:00:18 ny-sentinel kernel: racluster[5558]: segfault at 46
>>>> ip
>>>> 002e1877 sp bfb54e10 error 4 in libc-2.12.so[281000+190000] May 22
>>>> 02:00:18 ny-sentinel racluster[6044]: 02:00:18.586910 open
>>>> '/var/opt/argus/2013-05-22/argus_01:00:00': No such file or
>>>> directory May 22 02:00:18 ny-sentinel kernel: racluster[6044]:
>>>> segfault at 46 ip
>>>> 0020d877 sp bf854d50 error 4 in libc-2.12.so[1ad000+190000] May 22
>>>> 03:00:18 ny-sentinel racluster[6520]: 03:00:18.534326 open
>>>> '/var/opt/argus/2013-05-22/argus_02:00:00': No such file or
>>>> directory May 22 03:00:18 ny-sentinel kernel: racluster[6520]:
>>>> segfault at 46 ip
>>>> 0019c877 sp bfaf2b30 error 4 in libc-2.12.so[13c000+190000] May 22
>>>> 06:00:18 ny-sentinel racluster[8015]: 06:00:18.786627 open
>>>> '/var/opt/argus/2013-05-22/argus_05:00:00': No such file or
>>>> directory May 22 06:00:18 ny-sentinel kernel: racluster[8015]:
>>>> segfault at 46 ip
>>>> 002a2877 sp bfa74270 error 4 in libc-2.12.so[242000+190000] May 22
>>>> 07:00:18 ny-sentinel racluster[8517]: 07:00:18.386497 open
>>>> '/var/opt/argus/2013-05-22/argus_06:00:00': No such file or
>>>> directory May 22 07:00:18 ny-sentinel kernel: racluster[8517]:
>>>> segfault at 46 ip
>>>> 0073c877 sp bfaeee10 error 4 in libc-2.12.so[6dc000+190000] May 22
>>>> 08:00:18 ny-sentinel racluster[9006]: 08:00:18.591348 open
>>>> '/var/opt/argus/2013-05-22/argus_07:00:00': No such file or
>>>> directory May 22 08:00:18 ny-sentinel kernel: racluster[9006]:
>>>> segfault at 46 ip
>>>> 0023e877 sp bfe6f110 error 4 in libc-2.12.so[1de000+190000] May 22
>>>> 10:00:19 ny-sentinel racluster[10009]: 10:00:19.991831 open
>>>> '/var/opt/argus/2013-05-22/argus_09:00:00': No such file or
>>>> directory May 22 10:00:19 ny-sentinel kernel: racluster[10009]:
>>>> segfault at 46 ip 00627877sp bfd4cf50 error 4 in
>>>> libc-2.12.so[5c7000+190000] May 22 13:00:20 ny-sentinel
>>>> racluster[11556]: 13:00:20.585851 open
>>>> '/var/opt/argus/2013-05-22/argus_12:00:00': No such file or
>>>> directory May 22 13:00:20 ny-sentinel kernel: racluster[11556]:
>>>> segfault at 46 ip
>>>> 002bb877 sp bfda4480 error 4 in libc-2.12.so[25b000+190000] May 22
>>>> 14:00:20 ny-sentinel racluster[12106]: 14:00:20.585760 open
>>>> '/var/opt/argus/2013-05-22/argus_13:00:00': No such file or
>>>> directory May 22 14:00:20 ny-sentinel kernel: racluster[12106]:
>>>> segfault at 46 ip
>>>> 00c3b877 sp bfbebb40 error 4 in libc-2.12.so[bdb000+190000] May 22
>>>> 14:00:20 ny-sentinel racluster[12109]: 14:00:20.997232 open
>>>> '/var/opt/argus/2013-05-22/argus_13:00:00': No such file or
>>>> directory May 22 14:00:20 ny-sentinel kernel: racluster[12109]:
>>>> segfault at 46 ip
>>>> 001ba877 sp bf838800 error 4 in libc-2.12.so[15a000+190000] May 22
>>>> 15:00:19 ny-sentinel racluster[12619]: 15:00:19.898971 open
>>>> '/var/opt/argus/2013-05-22/argus_14:00:00': No such file or
>>>> directory May 22 15:00:19 ny-sentinel kernel: racluster[12619]:
>>>> segfault at 46 ip
>>>> 001d4877 sp bf865ab0 error 4 in libc-2.12.so[174000+190000]
>>>>
>>>> Note that there is no easily identifiable pattern in which files
>>>> cause the segfault and which do not, but the problem occurs with every file.
>>>>
>>>> I can only guess that it is segfaulting because it can't open
>>>> '/var/opt/argus/N/argus_N' (No such file or directory).
>>>> It's worth noting that in the instance mentioned in the previous
>>>> email, '/var/opt/argus/2013-05-21/argus_10:00:00' did exist at the
>>>> time racluster is executed. I am concluding this because after
>>>> racluster is executed, gzip compresses '/var/opt/argus/2013-05-21/argus_10:00:00' successfully.
>>>>
>>>> I don't believe the segfault is the problem.
>>>>
>>>>
>>>> 2) Can I run racluster against the file and get good results?
>>>>
>>>> Yes, I can run racluster against both the gzipped resultant file,
>>>> and the file that is generated by rastream that wasn't gzipped.
>>>>
>>>>
>>>> 3) Multiple client versions:
>>>>
>>>> When I mention modifying which racluster is executed by the shell
>>>> script (executed by rastream's -f argument), I simply was
>>>> attempting to have it use the racluster binary that I had compiled
>>>> with support for debugging and symbols. My reasoning was so that it
>>>> would produce more verbose output to syslog, which it didn't.
>>>>
>>>> There are only two sets of binaries located on the system. One that
>>>> has been compiled with debugging support and one that hasn't:
>>>> # find / -iname racluster
>>>> /root/argus-clients-3.0.7.10/bin/racluster #debugging support
>>>> /usr/local/bin/racluster #no debugging support
>>>>
>>>> /usr/local/bin/rastream is executed and is writing the files I am
>>>> concerned with (and is causing the problem).
>>>> /usr/local/bin/racluster is currently executed by
>>>> /usr/local/bin/rastream.sh due to its location in the $PATH (and is
>>>> the one that is segfaulting occasionally).
>>>>
>>>>
>>>> 4) How I see the problem:
>>>>
>>>> I don't want to make any assumptions, or create a red herring,
>>>> since I am not familiar with the code of rastream, but I believe
>>>> that the following is
>>>> happening:
>>>>
>>>> 1) `/usr/local/bin/rastream -d -S 127.0.0.1:561 -B 15s -M time 1h
>>>> -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh`
>>>> is executed, and then (due to -d) the daemonizing method is called
>>>> and forks another rastream process and starts writing to
>>>> /var/opt/argus/Y-m-d/argus_T1. (which is good)
>>>> 2) At `time 1h`, `/usr/local/bin/rastream.sh` is forked, without
>>>> blocking `rastream` from writing. (good)
>>>> 3) While #2 occurs, rastream is _still_ writing to
>>>> /var/opt/argus/Y-m-d/argus_T1 (the file from the previous `time 1h`
>>>> time span, not the new `time 1h` time span). (which is bad)
>>>> 4) `/usr/local/bin/rastream.sh` executes `racluster -M replace -r
>>>> $FILES` against the file /var/opt/argus/Y-m-d/argus_T1. (bad)
>>>> 5) `/usr/local/bin/rastream.sh` executes `gzip $FILES` and is able
>>>> to add
>>>> /var/opt/argus/Y-m-d/argus_T1 (bad)
>>>> 6) `rastream` begins writing /var/opt/argus/Y-m-d/argus_T2 (the new
>>>> `time 1h` file) (bad)
>>>>
>>>> So, the daemonized fork of rastream doesn't start writing to the
>>>> new file
>>>> (/var/opt/argus/Y-m-d/argus_T2) until _after_ it executes the -f
>>>> designated script `/usr/local/bin/rastream.sh`.
>>>> Instead, what should be happening is: the daemonized fork of
>>>> rastream should start writing to the new file, and _then_ execute
>>>> the -f designated script `/usr/local/bin/rastream.sh`.
>>>>
>>>> It would seem that the way that rastream is daemonized affects the
>>>> later -f logic.
>>>>
>>>>
>>>> Hope this helps.
>>>>
>>>>
>>>> Thanks for your time,
>>>>
>>>> Matt
>>>>
>>>>
>>>>
>>>> On May 22, 2013, at 10:45 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>> Hey Matt,
>>>> Hmmm, so your emails are waaayyyyy too dense for dealing with bugs
>>>> / problems.
>>>> Lets focus on one thing at a time.
>>>>
>>>> Your racluster() is segfaulting. Why is it doing that? Can you run
>>>> racluster()
>>>> against the file and get good results ? Do you have multiple
>>>> versions of the clients installed on your machine, so that there is
>>>> ambiguity in which
>>>> racluster()
>>>> or rastream() is actually running? We should clear that up before
>>>> proceeding any further.
>>>>
>>>> Carter
>>>>
>>>>
>>>> On May 21, 2013, at 1:31 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Hello Carter,
>>>>
>>>>
>>>>
>>>> Thanks for getting back to me quickly.
>>>>
>>>>
>>>>
>>>> I've tried the following to attempt to troubleshoot the problem,
>>>> but it doesn't appear that anything's outstanding.
>>>>
>>>>
>>>>
>>>> Apologies for the verbosity…
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> # cd ./argus-clients-3.0.7.10
>>>> # touch .devel .debug
>>>> # ./configure
>>>> # make clean && make
>>>> # kill -2 $(pgrep rastream)
>>>> # ./bin/rastream -d -F /etc/rastream.conf -S 127.0.0.1:561 -B 15s
>>>> -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f
>>>> /usr/local/bin/rastream.sh -D4
>>>> rastream[29500.c0567db7]: 2013-05-21 10:54:31.455541 ArgusNewQueue
>>>> () returning 0x8609548
>>>> rastream[29500.c0567db7]: 2013-05-21 10:54:31.455612
>>>> ArgusNewHashTable
>>>> (65536) returning 0x8609bc0
>>>> rastream[29500.c0567db7]: 2013-05-21 10:54:31.455632 ArgusNewList
>>>> () returning 0x860a010
>>>> rastream[29500.c0567db7]: 2013-05-21 10:54:31.455695 ArgusNewQueue
>>>> () returning 0x8611878
>>>> rastream[29500]: 2013-05-21 10:54:31.455729 started
>>>>
>>>>
>>>>
>>>> syslog reports:
>>>> May 21 10:54:31 ny-sentinel rastream[29500]: 2013-05-21
>>>> 10:54:31.455729 started May 21 11:00:18 ny-sentinel
>>>> racluster[29544]: 2013-05-21 11:00:18.635754 open
>>>> '/var/opt/argus/2013-05-21/argus_10:00:00': No such file or
>>>> directory May 21 11:00:18 ny-sentinel kernel: racluster[29544]:
>>>> segfault at 46 ip
>>>> 001d9877 sp bfb9f200 error 4 in libc-2.12.so[179000+190000]
>>>>
>>>>
>>>>
>>>> With /usr/local/bin/rastream.sh executing `racluster -M replace -r
>>>> $FILES`, using the racluster in $PATH (as it is originally).
>>>>
>>>>
>>>>
>>>> Note that '/var/opt/argus/2013-05-21/argus_10:00:00' did exist at
>>>> the time [but that doesn't mean that racluster can read it ?].
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> This is interesting:
>>>> I modifed /usr/local/bin/rastream.sh to execute
>>>> `[path]/argus-clients-3.0.7.10/bin/racluster -D4 -M replace -r
>>>> $FILES` instead of just using the $PATHed racluster…
>>>>
>>>>
>>>>
>>>> # kill -2 $(pgrep rastream)
>>>> # ./argus-clients-3.0.7.10/bin/rastream -d -F /etc/rastream.conf -S
>>>> 127.0.0.1:561 -B 15s -M time 10m -w
>>>> /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh -D4
>>>> rastream[30131.c0f67cb7]: 2013-05-21 11:26:53.528909 ArgusNewQueue
>>>> () returning 0x9e77550
>>>> rastream[30131.c0f67cb7]: 2013-05-21 11:26:53.528982
>>>> ArgusNewHashTable
>>>> (65536) returning 0x9e77bc8
>>>> rastream[30131.c0f67cb7]: 2013-05-21 11:26:53.529008 ArgusNewList
>>>> () returning 0x9e78018
>>>> rastream[30131.c0f67cb7]: 2013-05-21 11:26:53.529072 ArgusNewQueue
>>>> () returning 0x9e7f880
>>>> rastream[30131]: 2013-05-21 11:26:53.529102 started
>>>>
>>>>
>>>>
>>>> syslog reports:
>>>> May 21 11:26:53 ny-sentinel rastream[30131]: 2013-05-21
>>>> 11:26:53.529102 started
>>>>
>>>>
>>>>
>>>> …and nothing from racluster or anything else at the 10 minute mark,
>>>> even though the rotation (and problem) still occur:
>>>>
>>>>
>>>>
>>>> # ll --full-time /var/opt/argus/2013-05-21/ | tail -n 5 -rw-r--r--.
>>>> 1 root 159764 2013-05-21 11:30:59.466174549 -0400
>>>> argus_11:20:00
>>>> -rw-r--r--. 1 root 384117 2013-05-21 11:30:17.619561000 -0400
>>>> argus_11:20:00.gz -rw-r--r--. 1 root 181472 2013-05-21
>>>> 11:41:00.721865476 -0400
>>>> argus_11:30:00
>>>> -rw-r--r--. 1 root 394783 2013-05-21 11:40:17.511247000 -0400
>>>> argus_11:30:00.gz -rw-r--r--. 1 root 1421440 2013-05-21
>>>> 11:48:55.328587722 -0400
>>>> argus_11:40:00
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Reverting rastream.sh to execute racluster in the $PATH (as
>>>> original) and changing back to 'time 1h' as:
>>>> # kill -2 $(pgrep rastream)
>>>> # ./argus-clients-3.0.7.10/bin/rastream -d -F /etc/rastream.conf -S
>>>> 127.0.0.1:561 -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T
>>>> -f /usr/local/bin/rastream.sh -D4
>>>>
>>>>
>>>>
>>>> results in syslog:
>>>> May 21 12:05:37 ny-sentinel rastream[30905]: 2013-05-21
>>>> 12:05:37.950988 started
>>>>
>>>>
>>>>
>>>> …and nothing from racluster (why and how?)), but the problem still exists:
>>>>
>>>>
>>>>
>>>> # ll --full-time /var/opt/argus/2013-05-21/ | tail -n 3 -rw-r--r--.
>>>> 1 root 145612 2013-05-21 13:01:00.562634873 -0400
>>>> argus_12:00:00
>>>> -rw-r--r--. 1 root 2303564 2013-05-21 13:00:18.216161000 -0400
>>>> argus_12:00:00.gz -rw-r--r--. 1 root 315520 2013-05-21
>>>> 13:02:35.126692699 -0400
>>>> argus_13:00:00
>>>>
>>>>
>>>>
>>>> It's quite obvious that the size of the argus data has increased a
>>>> very large amount (due to the debugging).
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Does anything here indicate the problem?
>>>>
>>>>
>>>>
>>>> Is there anything else that I can do to further troubleshoot the issue?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here's some freaky stuff:
>>>>
>>>>
>>>>
>>>> querying the file argus_12:00:00 for ltime doesn't correlate with
>>>> the last modified time of the file:
>>>>
>>>>
>>>>
>>>> # ra -nr argus_12\:00\:00 -s stime ltime -w - | rasort -m stime -s
>>>> stime ltime | head -n 3 StartTime LastTime
>>>> 2013-05-21 12:59:03.164195 2013-05-21 12:59:06.417385
>>>> 2013-05-21 12:59:03.369926 2013-05-21 12:59:03.501171
>>>>
>>>>
>>>>
>>>> # ra -nr argus_12\:00\:00 -s stime ltime -w - | rasort -m ltime -s
>>>> stime ltime | tail -n 2
>>>> 2013-05-21 12:59:17.117235 2013-05-21 13:00:00.000000
>>>> 2013-05-21 12:59:28.896645 2013-05-21 13:00:00.000000
>>>>
>>>>
>>>>
>>>> # ra -nr argus_12\:00\:00 -s stime ltime | head -n 3 StartTime
>>>> LastTime
>>>> 2013-05-21 12:05:38.036319 2013-05-21 09:52:57.532792 # this is weird.
>>>> 2013-05-21 12:59:27.970584 2013-05-21 12:59:36.325653
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> querying the file argus_12:00:00.gz:
>>>>
>>>>
>>>>
>>>> sort by stime:
>>>> # ra -nr argus_12\:00\:00.gz -s stime ltime -w - | rasort -m stime
>>>> -s stime ltime | head -n 3 StartTime LastTime
>>>> 2013-05-21 11:59:44.064216 2013-05-21 12:34:17.996512
>>>> 2013-05-21 11:59:44.509320 2013-05-21 11:59:59.639822
>>>>
>>>>
>>>>
>>>> # ra -nr argus_12\:00\:00.gz -s stime ltime -w - | rasort -m stime
>>>> -s stime ltime | tail -n 2
>>>> 2013-05-21 12:59:02.980166 2013-05-21 12:59:02.981078
>>>> 2013-05-21 12:59:02.980171 2013-05-21 12:59:02.981120
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> sort by ltime:
>>>> # ra -nr argus_12\:00\:00.gz -s ltime -w - | rasort -m ltime -s
>>>> stime ltime
>>>> | head -n 3
>>>> StartTime LastTime
>>>> 2013-05-21 11:59:46.214566 2013-05-21 11:59:46.314022
>>>> 2013-05-21 11:59:51.860594 2013-05-21 11:59:51.887852
>>>>
>>>>
>>>>
>>>> # ra -nr argus_12\:00\:00.gz -s ltime -w - | rasort -m stime ltime
>>>> -s stime ltime | tail -n 2
>>>> 2013-05-21 12:00:00.000000 2013-05-21 13:00:00.000000
>>>> 2013-05-21 12:38:12.914597 2013-05-21 13:00:00.000000
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks again,
>>>>
>>>>
>>>>
>>>> Matt
>>>>
>>>> On May 21, 2013, at 10:49 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>> Yes you have to compile in debug support. We should turn on symbols, etc...
>>>> as well. In the clients root directory, try this:
>>>>
>>>> % touch .devel .debug
>>>> % ./configure
>>>> % make clean; make
>>>>
>>>> And use the resulting binaries for the testing.
>>>> Carter
>>>>
>>>> On May 21, 2013, at 10:08 AM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Carter,
>>>>
>>>> Do settings need to be changed at compile time in order to get
>>>> debugging features? I have started with -D4 and only the start time
>>>> and forked time are logged. I have confirmed that rastream is appending to the proper file.
>>>>
>>>>
>>>> Just to let you know *latest* is still 3.0.7.9, not 3.0.7.10.
>>>>
>>>>
>>>> Thanks very much!
>>>>
>>>> Matt
>>>>
>>>> On May 21, 2013, at 9:55 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>> Try the newest client distribution, just to see if any of the
>>>> changes we made to date affect the problem. This may be a tough one
>>>> to debug, but if I can recreate the issue, then we're on our way to happiness.
>>>>
>>>> We'll need you to turn on debug on your daemon, the output should
>>>> go to the syslog, so testing that briefly may help a bit. Run your
>>>> rastream with the "-d -D4" options, and see if you get any debug info in your syslog.
>>>>
>>>> Carter
>>>>
>>>> On May 21, 2013, at 9:48 AM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Good morning Carter!
>>>>
>>>>
>>>>
>>>> I wrote an init script and had some issues when starting rastream
>>>> with the script targetting ~/.rarc even if I set $ARGUSHOME or
>>>> $HOME within the script (on CentOS) (due to service's use of /bin/env).
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> So, the line I use within the init script is:
>>>>
>>>>
>>>>
>>>> /usr/local/bin/rastream -d -F /etc/rastream.conf -S 127.0.0.1:561
>>>> -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f
>>>> /usr/local/bin/rastream.sh
>>>>
>>>>
>>>>
>>>> The same problem with timing occurs as had previously occur when
>>>> relying on the full ~/.rarc.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is an etherpad with relevant
>>>> information:https://etherpad.mozilla.org/dmoSdfQ9H4
>>>>
>>>>
>>>>
>>>> # rastream --version
>>>> Rastream Version 3.0.7.9
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks very much Carter!
>>>>
>>>>
>>>>
>>>> Matt
>>>>
>>>> On May 21, 2013, at 6:44 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>>
>>>> Hey Matt,
>>>> So.......any chance you are setting the timezine in a rarc file
>>>> somewhere, like /etc/rarc or in your yme directory ?
>>>> What version are you running ?
>>>>
>>>> Carter
>>>>
>>>> On May 20, 2013, at 2:42 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>
>>>> Hello All:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I am having a problem with rastream that's manifested itself when
>>>> using the -f "shell script executor" argument to rotate files at 'time 1h'.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> If I run rastream as a daemon, then the script seems to run before
>>>> the "hour" is over (and the "hour" is over at the incorrect time):
>>>> # rastream -d -S 127.0.0.1:561 -B 15s -M time 1h -w
>>>> /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh
>>>>
>>>>
>>>>
>>>> A few hours' files look like:
>>>>
>>>>
>>>>
>>>> # ls --full-time /var/opt/argus/2013-05-18 total 3728 -rw-r--r--. 1
>>>> root 18752 2013-05-18 01:00:59.556839459 -0400 argus_00:00:00
>>>> -rw-r--r--. 1 root 160607 2013-05-18 01:00:17.793286000 -0400
>>>> argus_00:00:00.gz -rw-r--r--. 1 root 12068 2013-05-18
>>>> 02:00:59.619364943 -0400 argus_01:00:00 -rw-r--r--. 1 root 163409
>>>> 2013-05-18 02:00:17.943700000 -0400 argus_01:00:00.gz -rw-r--r--. 1
>>>> root 9032 2013-05-1803:01:00.579907536 -0400 argus_02:00:00
>>>> -rw-r--r--. 1 root 122920 2013-05-18 03:00:17.834317000 -0400
>>>> argus_02:00:00.gz -rw-r--r--. 1 root 22092 2013-05-18
>>>> 04:01:00.698357771 -0400 argus_03:00:00 -rw-r--r--. 1 root 122002
>>>> 2013-05-18 04:00:17.835675000 -0400 argus_03:00:00.gz -rw-r--r--. 1
>>>> root 17704 2013-05-18 05:01:00.450618851 -0400 argus_04:00:00
>>>> -rw-r--r--. 1 root 133212 2013-05-18 05:00:17.742040000 -0400
>>>> argus_04:00:00.gz -rw-r--r--. 1 root 14592 2013-05-18
>>>> 06:00:54.886285774 -0400 argus_05:00:00 -rw-r--r--. 1 root 160523
>>>> 2013-05-18 06:00:17.562776000 -0400 argus_05:00:00.gz
>>>>
>>>>
>>>>
>>>> It looks like the gzipped file is last modified before the hour
>>>> file, which leads me to believe that rastream isn't finished
>>>> writing to the argus file before -f[] is executed.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> If I run rastream as follows, I have no problem:
>>>> # nohup rastream -S 127.0.0.1:561 -B 15s -M time 1h -w
>>>> /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh &
>>>>
>>>>
>>>>
>>>> A few hours' files look like:
>>>>
>>>>
>>>>
>>>> # ls --full-time /var/opt/argus/2013-05-20 total 5372 -rw-r--r--. 1
>>>> root 217245 2013-05-20 10:00:17.573908000 -0400 argus_09:00:00.gz
>>>> -rw-r--r--. 1 root 6377 2013-05-2011:00:17.762140000 -0400
>>>> argus_10:00:00.gz -rw-r--r--. 1 root 9269
>>>> 2013-05-2011:38:08.879810000 -0400 argus_11:00:00.gz -rw-r--r--. 1
>>>> root 313 2013-05-2013:00:17.170958000 -0400 argus_12:00:00.gz
>>>> -rw-r--r--. 1 root 8965 2013-05-2014:00:17.540889000 -0400
>>>> argus_13:00:00.gz
>>>>
>>>>
>>>>
>>>> (early day over there)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I have verified that the system time is correct and ntpd is running
>>>> properly.
>>>>
>>>>
>>>>
>>>> Clients are 3.0.7.9.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Carter, do you have any ideas?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> Matt
>
>
More information about the argus
mailing list