updates for argus-2.x compatbility and database support

[Carter Bullard]

Hey Ken,
One of the features of the "new" argus that will come out in a month or
so is the "ARGUS_EVENT" record, which I use to generate temperature,
processor load, SNMP data collection and lsof data collection.  Argus,
has an event queue that is configured from the argus.conf file.  Here is
one of mine below.

# Argus supports the generation of host originated processes
# to gather additional data and statistics.  These include
# periodic processes to poll for SNMP data, as an example, or
# to collect host statistics through reading procfs().  Or
# single run programs that run at a specified time.
#
# These argus events, are generated from the complete list of
# ARGUS_EVENT_DATA directives that are specified here.
#
# The syntax is:
#      Syntax is: "method:path|prog:interval[:postproc]"
#          Where:  method = [ "file" | "prog" ]
#                pathname | program = "%s"
#                interval = %d[smhd] [ zero means run once ]
#                postproc = [ "compress" | "compress2" ]
#
ARGUS_EVENT_DATA="prog:/usr/local/bin/ravms:20s:compress"
ARGUS_EVENT_DATA="prog:/usr/local/bin/ratp:5m:compress"
ARGUS_EVENT_DATA="prog:/usr/local/bin/rasnmp:1m:compress"
ARGUS_EVENT_DATA="file:/proc/vmstat:30s:compress"
ARGUS_EVENT_DATA="prog:/usr/bin/uptime:30s"
ARGUS_EVENT_DATA="prog:/usr/local/bin/ralsof:10s:compress"

Here is the contents of my ralsof script, as a very simple example:

#!/bin/bash
#
#  Gargoyle Software
#  Copyright (c) 2006-2009 QoSient, LLC
#  All rights reserved.
#
#  ralsof - Report open inet sockets and provide application names
#
# Carter Bullard
# QoSient, LLC
#
output=`lsof -i -n -P`
#
#
echo "<ArgusEventData>"
echo "$output"
echo "</ArgusEventData>"

The output of these programs is simply put in a buffer, optionally
compressed, and then shipped out in the argus data stream, with an
Argus transport header on it.   Radium will transport them, copy them,
even label them if needed, and most of the ra* programs will do
something with the records.  rasplit() and rastream() will put them
in the archive, so that the event data is commingled with the network
traffic data, and the source id of the event is the Argus source id, so
scope is maintained.

Most of the programs that I use generate XML formatted data, so that
the ra* programs just need to print the buffers, and some analytic can
do its thing later.  I have not written many analytics yet, so I'm not
graphing the data in ragraph() for instance.

That's as much support as I have now.  If you would like to help me
expand on this functionality on the list,  I'd love to hear your  
opinion.
Such as how to minimize the lsof() data set, and how to read the
data on the other end ;o)


Carter

On Feb 26, 2009, at 9:20 AM, Ken A wrote:

> Carter Bullard wrote:
>> Gentle people,
>> I am working on a major release of the clients this week and I should
>> have a package hopefully by Thurs/Fri (if nothing gets in the way).
>> The primary function is to get general bug fixes into the main  
>> release.
>> And backward compatibility was the bug of the week, last week, so I'm
>> working on that.
>> Many "standard" programs will have a number of tweaks to fix bugs  
>> that
>> have come up, that have not hit the mailing list.  While it will be  
>> a lot of
>> changes, , these programs have been stable for quite some time, so  
>> I'm
>> hoping that we won't have a lot of little problems.  Testing will  
>> need to
>> be done, however.
>> rabins(), rasplit() and rastream() have all had a lot of work done  
>> to support
>> aggregations units smaller than 1 second.  So that you can specify  
>> bin
>> sizes down to a uSec.   This is important in our high performance  
>> stream
>> analysis work.  Maybe not for everyone, but the code is doing much  
>> better
>> with these changes.
>> And we will have support for flow labeling in radium(), where you can
>> slip ascii metadata into the records to "pump up" the semantics.   
>> This
>> is really cool, and will take some discussions on the list to use  
>> it to the
>> fullest.
>> This major version release of the clients will have a lot of new  
>> undocumented
>> programs, but I will try to start describing them on the mailing  
>> list this week.
>> They cover two primary areas, user data analysis and database  
>> support.
>> It maybe possible that I only have one of these ready, but I'm  
>> working on both.
>> The database support causes one major change.  We will need to print
>> "sport" and "dport" values for ICMP flows.  This is guarantee that  
>> all flow
>> records will have a unique flow key, so we won't have trouble  
>> stuffing
>> ICMP flows into an indexed database table of argus records.
>> I seem to be in my office this week, which is a real surprise, so  
>> hopefully
>> I can make some progress.
>> A new release of argus will follow a month later, with support for  
>> packet
>> size and interpacket arrival histogram reporting, as well as a new
>> ArgusEvent feature, where we can collect SNMP, /proc, and lsof() data
>> and send them in the argus data stream.
>
> That sounds like a lot of data, and useful too. Will this enable me,  
> with the proper query, to access lsof data, like 'open files' of a  
> pid that  also had an open network connection that is of interest?  
> That would be quite helpful in a hosting environment. And I can  
> stuff it all into mysql too? very nice! Or am I dreaming?
> Thanks,
> Ken
>
>
>> This is primarily to tag flows with the applications that generated  
>> them.
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York  10022
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>
>
> -- 
> Ken Anderson
> Pacific Internet - http://www.pacific.net
>