Random question about Argus DSRs

Wed Aug 12 18:43:43 EDT 2009

Hey Harry,
rasplit() can split the individual records into, say a time based  
file.  That is what
its designed to do.  But it doesn't do what you are doing with rabins().

rabins() is aggregating all your records within the 1 hour period, and  
the output
you are generating is time sorted.  This is a good data reduction  
scheme, assuming
you're not interested in events that occur in the sub-hour range.   
But,  if you
were interested in whether any interruptions occurred in your long lived
flows, aggregating them over an hour, generally loses some of that  
info (but not all).

Most sites can't run rabins() with a time period of 1h. The get too  
many unique flows,
and their collection machines run out of memory.

Many sites generate your output using rasplit() and cron(), or  
rastream().  Both rasplit()
and rastream()  asynchronously write records to the output file, as  
the records arrive.
If you use rasplit(), you can use cron() to run a script to find the  
collection file, and
process it (aggregate, sort, filter, etc....).  This has its issues,  
as cron and rasplit are
unaware of each other.

Thus, ....,  rastream().   You tell it how long to wait (-B option),  
and the script to run against
the file.    In this case you would have a shell script that runs  
racluster().   So, x number
of seconds after the hour, rastream() would fire off a script to  
process your collection
file.  There is a sample rastream.sh in the support/Config directory.   
This would
generate the same output you're currently getting.

The benefit of doing it this way, is the hourly file is collecting  
records as they come in,
so that if you want to know what is happening, you can just read the  
file.  The way you're
doing it now, rabins() is holding the data for an hour in memory, and  
you can't get to it.
Also, if rabins() fails you lose potentially a complete hour of data.

OK, after all that.  I suspect that rabins() is the culprit, so I'll  
have to look into it tonight.
As a work around, I suggest the rastream() method.  I'll walk you  
through it, if its a
mystery.

Carter

On Aug 12, 2009, at 5:35 PM, Harry Bock wrote:

> That's a good question - all I really need to do is to split input  
> from an Argus server into 1 hour logs, can rasplit alone achieve that?
>
> On Wed, Aug 12, 2009 at 5:31 PM, Carter Bullard <carter at qosient.com>  
> wrote:
> So, in your hourly*10 file, we've got a record that thinks its  
> length is 20 bytes longer than
> it really is, and in your hourly*8 file, we've got an arp record  
> that thinks its length is
> 4 bytes shorter than it should be.
>
> Looks like it maybe rabins() that is generating the errors.  Do you  
> need to be running rabins()?
>
> Carter
>
>
>
> On Aug 12, 2009, at 3:49 PM, Harry Bock wrote:
>
>> Thanks a bunch Carter, I appreciate it :) I've been able to work  
>> around it for now, so take as much time as you need to debug.
>>
>> On Wed, Aug 12, 2009 at 3:48 PM, Carter Bullard  
>> <carter at qosient.com> wrote:
>> Looking at it now.  Looks like somebody is generating a record with
>> a bad length.  Not sure who yet, so give me a little time.  I'll  
>> get back to
>> you soon.
>>
>> Carter
>>
>>
>> On Aug 12, 2009, at 3:31 PM, Harry Bock wrote:
>>
>>> Argus server 3.0.1 beta 5, and clients 3.0.2 beta 11.
>>>
>>> On Wed, Aug 12, 2009 at 3:27 PM, Carter Bullard  
>>> <carter at qosient.com> wrote:
>>> So there is some header corruption in the files.
>>> What version of argus? and clients?
>>>
>>> Carter
>>>
>>> On Aug 12, 2009, at 3:25 PM, Harry Bock wrote:
>>>
>>>> They are collected via rabins piped into rasplit, like so:
>>>>
>>>> rabins -S server -M time 1h -B 20s -w - | rasplit -M time 1h -w  
>>>> "hourly.%Y%m%d-%H"
>>>>
>>>> On Wed, Aug 12, 2009 at 3:20 PM, Carter Bullard  
>>>> <carter at qosient.com> wrote:
>>>> Hey Harry,
>>>> Hmmmm, are these straight from argus or are they collected by  
>>>> radium, or rasplit, or ?
>>>>
>>>> Carter
>>>>
>>>> On Aug 12, 2009, at 3:14 PM, Harry Bock wrote:
>>>>
>>>>> Hey Carter,
>>>>>
>>>>> I've been getting some weirdness with some logs where the  
>>>>> ArgusMetricStruct DSR (and the ArgusTime DSR) for a particular  
>>>>> record is NULL, specifically with multicast flows.  I'm assuming  
>>>>> this is an expected situation, as ra* hande it just fine - but  
>>>>> what exactly does this indicate? What useful information can I  
>>>>> garner from such a record without metric/time data?
>>>>> ra prints out the following line for that transmission:
>>>>>                     v         udp    131.128.118.246.63956    <- 
>>>>> >    239.255.255.253.427                  INT
>>>>>
>>>>> I've uploaded an example log with such a record here: http://spanning-tree.org/~hbock/argus/NOTCRASH-hourly.20090812-02
>>>>>
>>>>> Thanks!
>>>>> Harry
>>>>>
>>>>> -- 
>>>>> Harry Bock
>>>>> Software Developer, Package Maintainer
>>>>> OSHEAN, Inc.
>>>>> Email: harry at oshean.org
>>>>> PGP Key ID: 546CC353
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> Harry Bock
>>>> Software Developer, Package Maintainer
>>>> OSHEAN, Inc.
>>>> Email: harry at oshean.org
>>>> PGP Key ID: 546CC353
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York  10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>>
>>>
>>>
>>> -- 
>>> Harry Bock
>>> Software Developer, Package Maintainer
>>> OSHEAN, Inc.
>>> Email: harry at oshean.org
>>> PGP Key ID: 546CC353
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York  10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>>
>>
>>
>> -- 
>> Harry Bock
>> Software Developer, Package Maintainer
>> OSHEAN, Inc.
>> Email: harry at oshean.org
>> PGP Key ID: 546CC353
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York  10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>
>
>
> -- 
> Harry Bock
> Software Developer, Package Maintainer
> OSHEAN, Inc.
> Email: harry at oshean.org
> PGP Key ID: 546CC353

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090812/e6d60727/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090812/e6d60727/attachment.bin>