graph of bytes against protocols for network loopdetection?

Thu Feb 28 12:09:51 EST 2008

Hey Mark,
This is cool!!  When you're ready to scale up, there are newer  
mechanisms
for creating the data repository and importing the data, using programs
like rastream() and if you wanted to get a little closer to near real- 
time posting
of data, you could use rabins() with a few variations, out of the  
argus-3.0 code set.

As an example, if you wanted to insert data every 5 minutes, it can be  
as easy as:
    rastream -S live.argus.stream -f yourMysqlImport.sh -M time 5m -B  
15s \
       -w /opt/ARGUS/OUTBOUND/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S

This would generate an argus archive broken out by year/month/day  
containing
files every 5 minutes, and 15 seconds after then end of each 5 minute  
clock
boundary, your script would be run against the file, indexing the data  
and then
compressing the file.  It could remove the file if you're not  
interested in keeping
the archive etc......

yourMysqlImport.sh would be simply (using ./support/Config/rastream.sh  
to start):

------ begin yourMysqlImport.sh -----

#!/bin/sh
#
#  Argus Client Software.  Tools to read, analyze and manage Argus data.
#  Copyright (C) 2000-2008 QoSient, LLC.
#  All Rights Reserved
#
# Script called by rastream, to process files.
#
# Since this is being called from rastream(), it will have only a single
# parameter, filename,
#
# Carter Bullard <carter at qosient.com>
#

PATH="/usr/local/bin:$PATH"; export PATH
package="argus-clients"
version="3.0.0"

OPTIONS="$*"
FILES=
while  test $# != 0
do
     case "$1" in
     -r) shift; FILES="$1"; break;;
     esac
     shift
done

mysqlimport  -# -L --host=192.168.150.204 --user='argus' -- 
password='&&&&' --fields-terminated-by=',' --lines-terminated-by='\n' \
  --low-priority  --verbose --columns='srcid, sdate, stime, ldate,  
ltime, dur, saddr, daddr, proto, sport, dport, bytes, sbytes, dbytes,  
pkts, spkts, dpkts, dir' \
argus '$FILES'

gzip $FILES
exit 0

----- end yourMysqlImport.sh -----

Send mail to the list if/when you start, and we can all help.

Carter

On Feb 27, 2008, at 12:50 PM, Bartlett, Mark wrote:

> Hi Marten,
>
> I am working on an installation guide which will explain  
> installation of the "Probe", "Loader", DB Server, and Web Server and  
> installation scripts which will create users, install cron jobs,  
> install the DB schema, etc.
>
> The  Probe Machine 'moves' the /var/log/argus/argus.out file to the / 
> opt/ARGUS/OUTBOUND directory.
>
> The Loader Machine 'pulls' the argus files from remote Probe  
> Machines.  Then the argus.out files are sent through the 'ra' tool  
> to produce an ASCII comma delimited file.
>
> The ASCII file is imported into the DB using mysqlimport:
> mysqlimport  -# -L --host=192.168.150.204 --user='argus' -- 
> password='&&&&' --fields-
> terminated-by=',' --lines-terminated-by='\n' --low-priority  -- 
> verbose     --columns='sr
> cid, sdate, stime, ldate, ltime, dur, saddr, daddr, proto, sport,  
> dport, bytes, sbytes,
> dbytes, pkts, spkts, dpkts, dir'     argus '/opt/ARGUS/OUTBOUND/ 
> argus.out'
>
> That's about it for file transfer and data insertion...
>
> We are currently running ARGUS in a limited capacity (within our  
> Research Network, monitoring mostly Dark IP's and Test Networks).   
> But are working on doing a larger deployment (BETA).  So we should  
> get a better idea of performance 'issues' in the near future.
>
> The DB schema I created contains a probe, location, and company  
> tables so we can run queries against 'specific' probe locations  
> (DMZ, INTERNAL, EXTERNAL, Dark Nets, etc)  Part of the Web GUI  
> contains a probe insert page, to add new probes, etc..  It's working  
> pretty well for us at this point.
>
> If I get some 'spare time' I would like to try and create a  
> partitioned DB Schema for the ARGUS data based on date to speed up  
> query times and alleviate Database Storage issues (archiving,  
> dropping data, etc)....
>
> I'll let you know when the installation guides are finished and the  
> package has been posted for download.
>
> mark
>
> -----Original Message-----
> From: argus-info-bounces at lists.andrew.cmu.edu [mailto:argus-info-bounces at lists.andrew.cmu.edu 
> ] On Behalf Of David Nolan
> Sent: Wednesday, February 27, 2008 12:32 PM
> To: argus-info at lists.andrew.cmu.edu
> Subject: Re: [ARGUS] graph of bytes against protocols for network  
> loopdetection?
>
> Mark,
>
> I would certainly be interested in seeing what you've got.  In fact I
> suspect that there will be enough interest in your setup that you  
> might
> just want to put it up somewhere for download.
>
> How are you feeding your data into the database?  And at what scale  
> are you
> doing this?
>
> -David
>
> --On Wednesday, February 27, 2008 09:50:38 -0500 "Bartlett, Mark"
> <Mark.Bartlett at getronics.com> wrote:
>
>> Hi Marten,
>>
>> Our Argus data is being fed into a DB, and I have created an "argus
>> website" using basic html/php pages with JPGraph
>> (http://www.aditus.nu/jpgraph/)
>>
>> Attached is a sample of the Hourly Dashboard which shows 4 graphs -  
>> 1.
>> Events by Hour, 2.  Daily Breakdown of Protocols, 3.  Source Bytes  
>> vs.
>> Dest Bytes, 4.  Source Pkts vs. Dest Pkts.
>>
>> If you are 'feeding' your data into a db and think this might be
>> something you would like to use I can send you our install package.
>>
>> Thanks.
>>
>> Bartola
>>
>>
>> -----Original Message-----
>> From: argus-info-bounces at lists.andrew.cmu.edu on behalf of Carter  
>> Bullard
>> Sent: Wed 2/27/2008 9:13 AM
>> To: Marten Bauer
>> Cc: argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] graph of bytes against protocols for network
>> loopdetection?
>> Hey Marten,
>> ragraph() is primarily a time series graphing program.
>> The easiest is to use racluster() to get the data totals:
>>
>> racluster -m proto -r file -s proto bytes - ip or arp
>>  Proto   TotBytes
>>    pim        624
>>   ospf      36188
>>    esp   18070651
>>    gre       6637
>>   ipv6        702
>>    udp    5119990
>>    tcp  744143899
>>   igmp        662
>>   icmp     203201
>>    arp       4177
>>
>> And then use something easy, like excel() or gnuplot().
>> What kind of platform are you using?
>>
>> Carter
>>
>>
>> On Feb 27, 2008, at 1:52 AM, Marten Bauer wrote:
>>
>>> Hello,
>>>
>>> for detecting network loops I need a graph which
>>> prints the protocol on the x axes and the amount of
>>> bytes on the y axes.
>>>
>>> I tried to archive this with ragraph, but I never got
>>> what I want.
>>>
>>> Is it possible with ragraph or another ra* tool to
>>> generate such plot?
>>>
>>> Thx for helping
>>> Marten
>>>
>>
>>
>>
>
>
>
>
>
>