[ARGUS] Argus flows to Kafka

[Phillip Deneault]

Hi Carter,

Its less about what new is wanted, and more just to optimize how to get
what we already have there.

Kafka is simply a message queue system.  In principle, Argus or radium
could directly send the logs to a Kafka 'topic' (maybe simply using the
defined collector id or something) that would be buffered for as long as
someone wanted their buffers to be.  Then other consumers can come along
and process the data.  In my use case, it allows me to run local Kafka
queues on my sensors which produce events as fast as it can, while letting
my consumers (who are far less tolerant to spikes in events) come along and
pick up those logs as they can.  Additionally, I'd like to avoid writing a
XML log to disk, only to have to tail that log in (and all that comes with
that), to convert it into JSON for Elastic.  If a tool can write directly
to a Kafka buffer, in any sort of structured format, Kafka can do all the
heavy lifting of managing the records from there and you can leave
downstream processing to something else.

And I'm personally intending to use ELK, but there are tons of applications
and processors out there that will happily process Kafka queues.  I know
you have been hesitant in the past to support lots and lots of output
formats and types and I would be too, since it would take away from the
core flow development, but it would be great for us who like Argus and want
to load data from directly into other tools.

Thanks,
Phil





On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard <carter at qosient.com> wrote:

> Hey Phil,
>
> There are a few groups that have done specific ML strategies using
> streaming argus data, Oak Ridge National Labs has an operational system,
> Situ, and a number of commercial entities have used argus as a part of
> their ML offerings, but I’m not sure if they are using Kafka … TensorFlow
> has always been the most common buzz word with these groups … since
> TensorFlow and Kafka are a common pair of terms, I think some of these
> companies are probably doing Kafka streaming and Argus data, but not sure
> that anyone will tell you ...
>
> Is there something that Kafka would need from radium, or the argus-clients
> that isn’t already there ....  Is there a specific thing that Kafka wants
> in its streaming pipeline ??
>
> Carter
>
> > On Sep 10, 2019, at 8:08 AM, Phillip Deneault <deneaulp at bc.edu> wrote:
> >
> > Is there an ra* tool, or is anyone aware of a 3rd party tool, that can
> process argus output directly into Kafka?  Final stop would be an ELK
> database, but using Kafka would be a better middle ground from a
> performance and maintenance point of view.
> >
> > Thanks,
> > Phil
> >
> >
> > _______________________________________________
> > argus mailing list
> > argus at qosient.com
> > https://pairlist1.pair.net/mailman/listinfo/argus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20190911/894a5428/attachment.html>