graph of bytes against protocols for network loopdetection?
Mark Bartlett
mabartle at gmail.com
Thu Feb 28 14:29:45 EST 2008
Hey Carter,
Thanks for the info... I'm trying to get the rastream command to work, I
can get it to 'print' to stout.. But not to the file getting some errors:
[root at argus-client-test ~]# rastream -F ~/excel.rarc -S ?.?.150.134:561 -f
/opt/ARGUS/SCRIPTS/argus_mysqlimport2.sh -M time 5m -B 15s -w
/opt/ARGUS/OUTBOUND/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
rastream[4809]: 2008-02-27,08:16:33 output string /opt/ARGUS/OUTBOUND/%Y/No
such file or directory/-1074160872/argus.%Y.No such file or
directory.0.%H.%M.(null) requires $srcid
Segmentation fault
I'm using the -F option for format requirements. Here is my excel.rarc file
options:
#RA_PRINT_LABELS=0
RA_FIELD_DELIMITER=','
RA_PRINT_NAMES=none
RA_FIELD_SPECIFIER="srcid stime ltime dur saddr daddr proto sport dport
bytes sbytes dbytes pkts spkts dpkts dir"
RA_TIME_FORMAT="%Y-%m-%d,%H:%M:%S"
RA_USEC_PRECISION=6
RA_FILTER="not man"
/opt/ARGUS/SCRIPTS/argus_mysqlimport2.sh is the "yourMysqlImport.sh" script
you sent me...
Any idea on the error code??? The directory is 'owned' by root...
thanks
Mark
From: Carter Bullard <carter <at> qosient.com>
Subject: Re: graph of bytes against protocols for network
loopdetection?<http://news.gmane.org/find-root.php?message_id=%3c29CB19E2%2dAAA0%2d4002%2d8CC7%2d2D32FE10E1FA%40qosient.com%3e>
Newsgroups: gmane.network.argus <http://news.gmane.org/gmane.network.argus>
Date: 2008-02-28 17:09:51 GMT (2 hours and 13 minutes ago)
Hey Mark,
This is cool!! When you're ready to scale up, there are newer
mechanisms
for creating the data repository and importing the data, using programs
like rastream() and if you wanted to get a little closer to near real-
time posting
of data, you could use rabins() with a few variations, out of the
argus-3.0 code set.
As an example, if you wanted to insert data every 5 minutes, it can be
as easy as:
rastream -S live.argus.stream -f yourMysqlImport.sh -M time 5m -B
15s \
-w /opt/ARGUS*/OUTBOUND/*%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
This would generate an argus archive broken out by year/month/day
containing
files every 5 minutes, and 15 seconds after then end of each 5 minute
clock
boundary, your script would be run against the file, indexing the data
and then
compressing the file. It could remove the file if you're not
interested in keeping
the archive etc......
yourMysqlImport.sh would be simply (using ./support/Config/rastream.sh
to start):
------ begin yourMysqlImport.sh -----
#!/bin/sh
#
# Argus Client Software. Tools to read, analyze and manage Argus data.
# Copyright (C) 2000-2008 QoSient, LLC.
# All Rights Reserved
#
# Script called by rastream, to process files.
#
# Since this is being called from rastream(), it will have only a single
# parameter, filename,
#
# Carter Bullard <carter <at> qosient.com>
#
PATH="/usr/local/bin:$PATH"; export PATH
package="argus-clients"
version="3.0.0"
OPTIONS="$*"
FILES=
while test $# != 0
do
case "$1" in
-r) shift; FILES="$1"; break;;
esac
shift
done
mysqlimport -# -L --host=192.168.150.204 --user='argus' --
password='&&&&' --fields-terminated-by=',' --lines-terminated-by='\n' \
--low-priority --verbose --columns='srcid, sdate, stime, ldate,
ltime, dur, saddr, daddr, proto, sport, dport, bytes, sbytes, dbytes,
pkts, spkts, dpkts, dir' \
argus '$FILES'
gzip $FILES
exit 0
----- end yourMysqlImport.sh -----
Send mail to the list if/when you start, and we can all help.
Carter
On Feb 27, 2008, at 12:50 PM, Bartlett, Mark wrote:
> Hi Marten,
>
> I am working on an installation guide which will explain
> installation of the "Probe", "Loader", DB Server, and Web Server and
> installation scripts which will create users, install cron jobs,
> install the DB schema, etc.
>
> The Probe Machine 'moves' the /var/log/argus/argus.out file to the /
> opt/ARGUS/OUTBOUND directory.
>
> The Loader Machine 'pulls' the argus files from remote Probe
> Machines. Then the argus.out files are sent through the 'ra' tool
> to produce an ASCII comma delimited file.
>
> The ASCII file is imported into the DB using mysqlimport:
> mysqlimport -# -L --host=192.168.150.204 --user='argus' --
> password='&&&&' --fields-
> terminated-by=',' --lines-terminated-by='\n' --low-priority --
> verbose --columns='sr
> cid, sdate, stime, ldate, ltime, dur, saddr, daddr, proto, sport,
> dport, bytes, sbytes,
> dbytes, pkts, spkts, dpkts, dir' argus '/opt/ARGUS*/OUTBOUND/*
> argus.out'
>
> That's about it for file transfer and data insertion...
>
> We are currently running ARGUS in a limited capacity (within our
> Research Network, monitoring mostly Dark IP's and Test Networks).
> But are working on doing a larger deployment (BETA). So we should
> get a better idea of performance 'issues' in the near future.
>
> The DB schema I created contains a probe, location, and company
> tables so we can run queries against 'specific' probe locations
> (DMZ, INTERNAL, EXTERNAL, Dark Nets, etc) Part of the Web GUI
> contains a probe insert page, to add new probes, etc.. It's working
> pretty well for us at this point.
>
> If I get some 'spare time' I would like to try and create a
> partitioned DB Schema for the ARGUS data based on date to speed up
> query times and alleviate Database Storage issues (archiving,
> dropping data, etc)....
>
> I'll let you know when the installation guides are finished and the
> package has been posted for download.
>
> mark
>
> -----Original Message-----
> From: argus-info-bounces <at> lists.andrew.cmu.edu
[mailto:argus-info-bounces <at> lists.andrew.cmu.edu
> ] On Behalf Of David Nolan
> Sent: Wednesday, February 27, 2008 12:32 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: Re: [ARGUS] graph of bytes against protocols for network
> loopdetection?
>
> Mark,
>
> I would certainly be interested in seeing what you've got. In fact I
> suspect that there will be enough interest in your setup that you
> might
> just want to put it up somewhere for download.
>
> How are you feeding your data into the database? And at what scale
> are you
> doing this?
>
> -David
>
> --On Wednesday, February 27, 2008 09:50:38 -0500 "Bartlett, Mark"
> <Mark.Bartlett <at> getronics.com> wrote:
>
>> Hi Marten,
>>
>> Our Argus data is being fed into a DB, and I have created an "argus
>> website" using basic html/php pages with JPGraph
>> (http://www.aditus.nu/jpgraph/)
>>
>> Attached is a sample of the Hourly Dashboard which shows 4 graphs -
>> 1.
>> Events by Hour, 2. Daily Breakdown of Protocols, 3. Source Bytes
>> vs.
>> Dest Bytes, 4. Source Pkts vs. Dest Pkts.
>>
>> If you are 'feeding' your data into a db and think this might be
>> something you would like to use I can send you our install package.
>>
>> Thanks.
>>
>> Bartola
>>
>>
>> -----Original Message-----
>> From: argus-info-bounces <at> lists.andrew.cmu.edu on behalf of Carter
>> Bullard
>> Sent: Wed 2/27/2008 9:13 AM
>> To: Marten Bauer
>> Cc: argus-info <at> lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] graph of bytes against protocols for network
>> loopdetection?
>> Hey Marten,
>> ragraph() is primarily a time series graphing program.
>> The easiest is to use racluster() to get the data totals:
>>
>> racluster -m proto -r file -s proto bytes - ip or arp
>> Proto TotBytes
>> pim 624
>> ospf 36188
>> esp 18070651
>> gre 6637
>> ipv6 702
>> udp 5119990
>> tcp 744143899
>> igmp 662
>> icmp 203201
>> arp 4177
>>
>> And then use something easy, like excel() or gnuplot().
>> What kind of platform are you using?
>>
>> Carter
>>
>>
>> On Feb 27, 2008, at 1:52 AM, Marten Bauer wrote:
>>
>>> Hello,
>>>
>>> for detecting network loops I need a graph which
>>> prints the protocol on the x axes and the amount of
>>> bytes on the y axes.
>>>
>>> I tried to archive this with ragraph, but I never got
>>> what I want.
>>>
>>> Is it possible with ragraph or another ra* tool to
>>> generate such plot?
>>>
>>> Thx for helping
>>> Marten
>>>
>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20080228/0a08901b/attachment.html>
More information about the argus
mailing list