Request for rabins to handle SIGHUP and reload the config?

Tue Jan 28 10:11:15 EST 2014

Hey Matt,
Well, if it really is six vs 0.5, then we should choose to flush
based solely on the ease of implementation of any generic
configuration change.

The real issue is that rabins() does / could do a lot of stuff.  It bins
things up, its an aggregator, its a labeler, it has an input and
an output filter, you can strip fields on input or output, 
you can actually run analytics within each bin just before the
output is flushed, it can / could provide its output stream to ports,
files, etc... functions that can all be modified on the fly if we
implement a generic "re-read the configuration" function.
So it can get pretty complicated.

Many of rabins() options are command-line only, so we may need
to create a rabins.conf file, or add rabins() specific options to the
.rarc file, which, in terms of a generic feature, can make the concept
of re-configuration even more complicated.  We already have one
rabins() specific configuration variable in the .rarc file, (see below)
and not sure that is the best idea.

If on the other hand, you only want to implement a bin size
change, we would only have to flush if the new bin size is smaller
than the old one, and the time boundary for the new bin size
has already passed, so we have to flush the current bin.  Practically,
that means that you could be getting data whose bin size was 1h,
and when you change the size to 5s, depending on the time, you
could get one set of data whose bin size is actually something like
32.5613 minutes.

OK, so on a signal, such as SIGHUP, you want us to reread
a configuration file, or all configuration files, which includes the
/etc/rarc, and the ~/.rarc file, and any future rabins() specific configuration
file, make changes and then keep processing, which may cause
us to flush the current bin, because its elapsed time is greater
than the new bin size.

Nice thing is that rabins() is designed to optionally send a start and
stop MAR record with each bin processed using the .rarc option
RA_GENERATE_BIN_MAR_RECORDS, which is by default set to "no".

Currently the MAR cause code used at the end of each bin is
ARGUS_STOP.  When we reconfigure, we would use the MAR
cause code of ARGUS_FLUSH, so that we know that we've flushed
a bin, rather than close a bin.  Just an accounting plus.

I think this is doable, but it will have to wait until after 3.0.8.

The fact that you can just crank up another rabins() with your
new configuration, using the same input as the one that is already
running, suggests that there may not be that many reasons
to change the configuration of a running rabins() on the fly.  Running
two in parallel, and switching from one to the other, maybe more
useful.

So, what do you think about that ?????

Carter

On Jan 28, 2014, at 9:12 AM, Matt Brown <matthewbrown at gmail.com> wrote:

> Immediately at SIGHUP?  Without letting the bin or buffer finish?  I don't know.  Honestly, I'd rather have the option.  But, it's six of one, half dozen of the other, right?
> 
> Up to you!
> 
> Matt
> 
> 
> 
> On Jan 28, 2014, at 8:56 AM, Carter Bullard <carter at qosient.com> wrote:
> 
>> I'm thinking that we should flush and start over regardless of the
>> configuration change.  Not very complicated, easy to describe predictable, etc....
>> Would that still do what you want ???
>> 
>> Carter
>> 
>> On Jan 28, 2014, at 8:54 AM, Matt Brown <matthewbrown at gmail.com> wrote:
>> 
>>> Carter,
>>> 
>>> Maybe this option (the "flush cache and use new bin size at N") can be passed at invocation?
>>> 
>>> That way, if someone has a bin of 24 hours, and a buffer of an hour, they can control how this new features works.
>>> 
>>> 
>>> What do you think?
>>> 
>>> 
>>> Thanks,
>>> 
>>> Matt
>>> 
>>> On Jan 27, 2014, at 2:47 PM, Carter Bullard <carter at qosient.com> wrote:
>>> 
>>>> Hey Matt,
>>>> So if you change the bin size and the hold time, if you can
>>>> wait for it to apply to the next time bin, then we don’t have
>>>> to do anything special.  But you may want the changes to apply
>>>> immediately, we’ll need to flush all the current bins and start
>>>> over.
>>>> 
>>>> Because there are a lot more options that can be changed,
>>>> we’ll have to consider how this new feature affects the strategy.
>>>> It would be easiest if we could just flush and start over again
>>>> regardless of the change, but there maybe an elegant way of
>>>> doing something more sophisticated……
>>>> 
>>>> But even with your limited example, if you’re running with 24
>>>> hour bins, and a “-B <secs>” option of 3600, and you
>>>> asynchronously change the bins size to, say 1s and a hold
>>>> time of 5s, then I doubt that you would be willing to
>>>> wait until tomorrow for the new bins to start ????
>>>> 
>>>> Once we figure out what we need to do, then should be no
>>>> problem.  It will have to wait until after we release 3.0.8
>>>> which should be in a few weeks, now that FloCon is over.
>>>> 
>>>> Carter
>>>> 
>>>> 
>>>> On Jan 27, 2014, at 2:08 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>> 
>>>>> Thanks for replying, Carter.
>>>>> 
>>>>> My intention is to specifically change the bin size (by way of
>>>>> modifying a config file).
>>>>> 
>>>>> I would think it makes sense to:
>>>>> - allow caches to expire within a max timeout (N seconds.... maybe the
>>>>> -B ?  maybe the bin size? maybe either as a user wishes?)
>>>>> - rolling active flows won't be a problem, right?
>>>>> - Any config changes would take effect after the set expiration.
>>>>> 
>>>>> Do these make sense to you?
>>>>> 
>>>>> I think it makes the most sense to have a single rule ("everything
>>>>> takes effect after the previous [buffer|bin|whatever] ends"), than to
>>>>> make seperate rules ("if the bin size changes, then, we will flush
>>>>> cache after the latest bin expires." and/or "if the output filter
>>>>> changes, it will be immediate or maybe not.").
>>>>> 
>>>>> But... you know what makes the most sense from what you've seen from
>>>>> users over the years.
>>>>> 
>>>>> 
>>>>> Thanks much,
>>>>> 
>>>>> Matt
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jan 23, 2014, at 8:01 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>> 
>>>>>> Hey Matt,
>>>>>> Yes, but there is a lot of behaviors we need to define.
>>>>>> Do we flush all of the pending data and its caches when we change the configuration, or do we just keep plugging along, and let the caches time out ???  If we change bin sizes, well need to flush everything, if its output filters do they take immediate affect, or after the flush.  There are a lot of issues to consider.
>>>>>> 
>>>>>> What are you thinking about and what would you like for it to do ??
>>>>>> 
>>>>>> Carter
>>>>>> 
>>>>>>> On Jan 23, 2014, at 3:38 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>>>>>>> 
>>>>>>> Hello all,
>>>>>>> 
>>>>>>> Carter:  is it possible to add SIGHUP handling to rabins() to reload
>>>>>>> the config file(s)?
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Matt
>>>>>>> 
>>>>> 
>>>> 
>>>> Carter Bullard
>>>> CEO/President
>>>> QoSient, LLC
>>>> 150 E 57th Street Suite 12D
>>>> New York, New York  10022
>>>> 
>>>> +1 212 588-9133 Phone
>>>> +1 212 588-9134 Fax
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140128/d7eebefd/attachment.bin>