updates for argus-2.x compatbility and database support

[Ken A]

Carter Bullard wrote:
> Hey Ken,
> One of the features of the "new" argus that will come out in a month or
> so is the "ARGUS_EVENT" record, which I use to generate temperature,
> processor load, SNMP data collection and lsof data collection.  Argus,
> has an event queue that is configured from the argus.conf file.  Here is
> one of mine below.
> 
> # Argus supports the generation of host originated processes
> # to gather additional data and statistics.  These include
> # periodic processes to poll for SNMP data, as an example, or
> # to collect host statistics through reading procfs().  Or
> # single run programs that run at a specified time.
> #
> # These argus events, are generated from the complete list of
> # ARGUS_EVENT_DATA directives that are specified here.
> #
> # The syntax is:
> #      Syntax is: "method:path|prog:interval[:postproc]"
> #          Where:  method = [ "file" | "prog" ]
> #                pathname | program = "%s"
> #                interval = %d[smhd] [ zero means run once ]
> #                postproc = [ "compress" | "compress2" ]
> #
> ARGUS_EVENT_DATA="prog:/usr/local/bin/ravms:20s:compress"
> ARGUS_EVENT_DATA="prog:/usr/local/bin/ratp:5m:compress"
> ARGUS_EVENT_DATA="prog:/usr/local/bin/rasnmp:1m:compress"
> ARGUS_EVENT_DATA="file:/proc/vmstat:30s:compress"
> ARGUS_EVENT_DATA="prog:/usr/bin/uptime:30s"
> ARGUS_EVENT_DATA="prog:/usr/local/bin/ralsof:10s:compress"
> 
> Here is the contents of my ralsof script, as a very simple example:
> 
> #!/bin/bash
> #
> #  Gargoyle Software
> #  Copyright (c) 2006-2009 QoSient, LLC
> #  All rights reserved.
> #
> #  ralsof - Report open inet sockets and provide application names
> #
> # Carter Bullard
> # QoSient, LLC
> #
> output=`lsof -i -n -P`
> #
> #
> echo "<ArgusEventData>"
> echo "$output"
> echo "</ArgusEventData>"
> 
> The output of these programs is simply put in a buffer, optionally
> compressed, and then shipped out in the argus data stream, with an
> Argus transport header on it.   Radium will transport them, copy them,
> even label them if needed, and most of the ra* programs will do
> something with the records.  rasplit() and rastream() will put them
> in the archive, so that the event data is commingled with the network
> traffic data, and the source id of the event is the Argus source id, so
> scope is maintained.
> 
> Most of the programs that I use generate XML formatted data, so that
> the ra* programs just need to print the buffers, and some analytic can
> do its thing later.  I have not written many analytics yet, so I'm not
> graphing the data in ragraph() for instance.
> 
> That's as much support as I have now.  If you would like to help me
> expand on this functionality on the list,  I'd love to hear your opinion.
> Such as how to minimize the lsof() data set, and how to read the
> data on the other end ;o)
> 

Hi Carter,

That's perfect. Getting the right script to extract the data I want is 
going to be a little tricky, but interesting! lsof with -F0 can be 
parsed to match pid,exe,owner (lsof -F0pLc) and then again to get any 
interesting pid or owner activity perhaps? - more thought needed here, 
since there might be a better ways to correlate & extract this info.

The on-the-fly database creation is going to be helpful in testing a lot 
of possibilities! If I had more time, I'd definitely put it in this 
direction, but I wear all of the technical hats (except support) at a 
small ISP, so I'm tied up with other things on many days.

Thanks,
Ken

> 
> Carter
> 
> On Feb 26, 2009, at 9:20 AM, Ken A wrote:
> 
>> Carter Bullard wrote:
>>> Gentle people,
>>> I am working on a major release of the clients this week and I should
>>> have a package hopefully by Thurs/Fri (if nothing gets in the way).
>>> The primary function is to get general bug fixes into the main release.
>>> And backward compatibility was the bug of the week, last week, so I'm
>>> working on that.
>>> Many "standard" programs will have a number of tweaks to fix bugs that
>>> have come up, that have not hit the mailing list.  While it will be a 
>>> lot of
>>> changes, , these programs have been stable for quite some time, so I'm
>>> hoping that we won't have a lot of little problems.  Testing will 
>>> need to
>>> be done, however.
>>> rabins(), rasplit() and rastream() have all had a lot of work done to 
>>> support
>>> aggregations units smaller than 1 second.  So that you can specify bin
>>> sizes down to a uSec.   This is important in our high performance stream
>>> analysis work.  Maybe not for everyone, but the code is doing much 
>>> better
>>> with these changes.
>>> And we will have support for flow labeling in radium(), where you can
>>> slip ascii metadata into the records to "pump up" the semantics.  This
>>> is really cool, and will take some discussions on the list to use it 
>>> to the
>>> fullest.
>>> This major version release of the clients will have a lot of new 
>>> undocumented
>>> programs, but I will try to start describing them on the mailing list 
>>> this week.
>>> They cover two primary areas, user data analysis and database support.
>>> It maybe possible that I only have one of these ready, but I'm 
>>> working on both.
>>> The database support causes one major change.  We will need to print
>>> "sport" and "dport" values for ICMP flows.  This is guarantee that 
>>> all flow
>>> records will have a unique flow key, so we won't have trouble stuffing
>>> ICMP flows into an indexed database table of argus records.
>>> I seem to be in my office this week, which is a real surprise, so 
>>> hopefully
>>> I can make some progress.
>>> A new release of argus will follow a month later, with support for 
>>> packet
>>> size and interpacket arrival histogram reporting, as well as a new
>>> ArgusEvent feature, where we can collect SNMP, /proc, and lsof() data
>>> and send them in the argus data stream.
>>
>> That sounds like a lot of data, and useful too. Will this enable me, 
>> with the proper query, to access lsof data, like 'open files' of a pid 
>> that  also had an open network connection that is of interest? That 
>> would be quite helpful in a hosting environment. And I can stuff it 
>> all into mysql too? very nice! Or am I dreaming?
>> Thanks,
>> Ken
>>
>>
>>> This is primarily to tag flows with the applications that generated 
>>> them.
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York  10022
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>
>>
>> -- 
>> Ken Anderson
>> Pacific Internet - http://www.pacific.net
>>
> 
> 
> 
> 


-- 
Ken Anderson
Pacific Internet - http://www.pacific.net