argus-3.0.3 on the server, new ARGUS_EVENT feature

[Carter Bullard]

Gentle people,
argus-3.0.3 is now an official development thread for argus.  It will become argus-3.0.4
when it is stable and released.  I'm going to try to avoid the endless alpha, beta
numbers, and just have 3.0.3.x until we have 3.0.4.

OK, argus-3.0.3.1 has many bug fixes, as reported on the list.  If I have provided patches,
they are in this release.  You can use these packages in production, but new features may
have bugs, so consider the odd releases as unstable, experimental whatever, but they
should work.

New features are now coming out, based on the list I put on the web site.  This has
Peter's argusarchive() programs and documentation, as an example.  And argus has
gotten some tiny performance enhancements, but no major architectural changes.

argus-3.0.3 will always be backward compatible with argus-3.x, so you can use
the new clients in your existing infrastructure.  argus-3.0.3 server may generate
data that argus-3.0.2 clients may not like, so be aware that upgrading should
be thoughtful.  With argus-3.0.3,  new features will be coming really fast,
so consider this a very dynamic set of software.

In this first wave of new features, argus gets the "Argus Events" functions, which allows
you to add additional data sources to your argus data stream.    The idea is that we want to
correlate lots of different data with network events to solve problems.  The easiest way to
do this, is to inject non-flow data into the flow data stream, so that as you collect, process
and archive argus data, supporting non-flow data is available.  Using this strategy,
the observation point for the non-flow data is the argus probe, so the data has the same
source identifiers and sync'd timestamps.  If you build archives, then the argus event
data is stored along with the flow data, so you can retrieve relevant data though local
scoping rules.

Argus can be configured to run programs/scripts or read files, such as those in /proc,
periodically, as specified in its argus.conf file.   Argus runs the ArgusEvent generator
as a thread, using 

It will take the output it receives as a buffer, optionally compress it (only if it reduces
the size of the output) and then form an ArgusEvent record, which is composed of
an ArgusRecordHeader,  an ArgusTransport DSR, so we can know who sent it,
a ArgusTime DSR, which has the starttime and endtime for running the program, and
a string that describes what the script is/was, and the buffer is placed in an ArgusData
DSR.  Argus then inserts the record into the argus output stream.

All the client programs are aware of ARGUS_EVENTs, and try to do the right thing with
them.  There is some work that needs to be done here.  A program called raevent() is
provided in the client distribution that prints the ARGUS_EVENT.  All other clients
basically ignore them at the moment.

The new argus configuration variables look like:
   ARGUS_EVENT_DATA="prog:/usr/local/bin/rasnmp:1m:compress"

The ./support/Config/argus.conf file has examples.

The initial example programs are rasnmp(), ravms(), and ralsof(), which can be found in
./bin of the argus distribution.  The concepts are that you can generate local stats, or
you can use argus to collect remote stats from other devices.   The examples show that
you can provide remote SNMP data, collected on whatever interval you want, local host
performance data, and user/process to flow data mappings (using lsof) so you can
attribute network flows to programs and users.  The examples are just bash or perl scripts
that generate ascii output structured as XML.  The XML schema could be improved upon,
for those interested.

The rasnmp() script is designed to gather the arp tables and counters from specific interfaces on
a switch in my enterprise.  It generates this type of output:

2010/02/09.10:03:34.379890:srcid=192.168.0.68:prog:/usr/local/bin/rasnmp
<ArgusEvent>
   <ArgusEventData Type = "SNMP Stats: /usr/bin/snmpwalk -Os -c xxxxxx -v 2c 192.168.0.1" >
      < Label = "ipNetToMediaPhysAddress.2.207.237.36.97 " Value = " STRING: 0:21:a0:ce:c:5" />
      < Label = "ipNetToMediaPhysAddress.3.192.168.0.3 " Value = " STRING: 0:f:b5:df:17:f8" />
      < Label = "ipNetToMediaPhysAddress.3.192.168.0.66 " Value = " STRING: 0:16:cb:ad:90:11" />
      < Label = "ipNetToMediaPhysAddress.3.192.168.0.68 " Value = " STRING: 0:23:32:2f:ac:9c" />
      < Label = "ipNetToMediaPhysAddress.3.192.168.0.164 " Value = " STRING: 0:b:db:5c:e5:7c" />
      < Label = "ipNetToMediaPhysAddress.3.192.168.0.202 " Value = " STRING: 0:9:5b:36:a:33" />
      < Label = "ipNetToMediaPhysAddress.9.10.7.219.56 " Value = " STRING: 0:19:d1:46:d3:16" />
   </ArgusEventData>
   <ArgusEventData Type = "SNMP Stats: /usr/bin/snmpget -Os -c xxxxxx -v 2c 192.168.0.1" >
      < Label = "ifInUcastPkts.2 " Value = " Counter32: 27505427" />
      < Label = "ifOutUcastPkts.2 " Value = " Counter32: 17997683" />
      < Label = "ifInOctets.2 " Value = " Counter32: 820077576" />
      < Label = "ifOutOctets.2 " Value = " Counter32: 3444316041" />
      < Label = "ifOutDiscards.2 " Value = " Counter32: 0" />
      < Label = "ifInUcastPkts.3 " Value = " Counter32: 20118859" />
      < Label = "ifOutUcastPkts.3 " Value = " Counter32: 23807470" />
      < Label = "ifInOctets.3 " Value = " Counter32: 3305162556" />
      < Label = "ifOutOctets.3 " Value = " Counter32: 3620106826" />
      < Label = "ifOutDiscards.3 " Value = " Counter32: 0" />
      < Label = "ifInUcastPkts.9 " Value = " Counter32: 848091" />
      < Label = "ifOutUcastPkts.9 " Value = " Counter32: 2618476" />
      < Label = "ifInOctets.9 " Value = " Counter32: 242767620" />
      < Label = "ifOutOctets.9 " Value = " Counter32: 2048906796" />
      < Label = "ifOutDiscards.9 " Value = " Counter32: 0" />
   </ArgusEventData>
</ArgusEvent>

Argus can now generate the data, it is up to us to write clients to parse the data and do good things
with it, such as graph it, correlate it, push it into mysql or rrd's, or just store it. 

There are a lot of candidate ArgusEvents generators, such as syslog() message collectors, temperature
sensors, multicore resource availability metri generators, etc.....   I hope that we create a lot of them.

Hope all is most excellent, and that lots of sites find this new feature useful!!!!!
Comments always more than welcome.

Carter



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100209/ae8f66ad/attachment.bin>