graph of bytes against protocols for network loopdetection?
Carter Bullard
carter at qosient.com
Fri Feb 29 12:32:44 EST 2008
Hey Mark,
So rastream() builds the binary argus data repository, and provides
unattended file processing, and that is a good thing. If you script
wants ascii input, you can have a ra() command do that. So try this
type
of strategy in the yourMysqlImport.sh file:
---- Being yourMysqlImport.sh -----
#!/bin/sh
PATH="/usr/local/bin:$PATH"; export PATH
FILES=
while test $# != 0
do
case "$1" in
-r) shift; FILES="$1"; break;;
esac
shift
done
ra -F /full/path/to/excel.rarc -r $FILES > /tmp/ra.txt
if ! test -z /tmp/ra.txt ; then
mysqlimport -# -L --host=192.168.150.204 --user='argus' --
password='&&&&' --fields-terminated-by=',' --lines-terminated-by='\n' \
--low-priority --verbose --columns='srcid, sdate, stime, ldate,
ltime, dur, saddr, daddr, proto, sport, dport, bytes, sbytes, dbytes,
pkts, spkts, dpkts, dir' \
argus /tmp/ra.txt
fi
rm -f /tmp/ra.txt
gzip $FILES
exit 0
---- End yourMysqlImport.sh-----
And be mindful of where you put your excel.rarc file so the script can
find it.
You'll eventually want to mature the script, so that ra() output
doesn't clobber
another program that may be running at the same time, but this should
get you
started?
Carter
On Feb 29, 2008, at 10:16 AM, Mark Bartlett wrote:
> So... The command "below" works:
> rastream -F ~/excel.rarc -S X.X.1.131:561 -f /opt/ARGUS/SCRIPTS/
> argus_mysqlimport2.sh -M time 1m -B 15s -w /opt/ARGUS/OUTBOUND/\
> $srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>
> But now my problem is it is a binary file that is created.. Anyway
> to have it kick out an ascii file instead?
>
> here are my excel.rarc options:
>
> RA_PRINT_LABELS=0
> RA_FIELD_DELIMITER=','
> RA_PRINT_NAMES=none
> RA_FIELD_SPECIFIER="srcid stime ltime dur saddr daddr proto sport
> dport bytes sbytes dbytes pkts spkts dpkts dir"
> RA_TIME_FORMAT="%Y-%m-%d,%H:%M:%S"
> RA_USEC_PRECISION=6
> RA_USERDATA_ENCODE=Ascii
> RA_FILTER="not man"
>
>
> Was hoping if I added the RA_USERDATA_ENCODE=Ascii that it would
> output it in ASCII.. Or am I off on that one?? (just read the rarc
> file in the /support/Config/ directory looks like this option only
> pertains to raxml()..
>
> The command above 'barfs' when it gets to the sql side of the
> house... But that's because it's a binary file...
>
> Thanks.
>
> Mark
>
> On Thu, Feb 28, 2008 at 5:13 PM, Carter Bullard <carter at qosient.com>
> wrote:
> Hey Mark,
> Interesting. The problem is that its trying to print out an error
> message, but
> because the string has all these escape sequences in them, it barfs.
> The solution is to correct what its complaining about.
>
> Try this string for the "-w " option:
> /opt/ARGUS/OUTBOUND/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>
> Carter
>
>
>
>
> On Feb 28, 2008, at 2:29 PM, Mark Bartlett wrote:
>
>> Hey Carter,
>>
>> Thanks for the info... I'm trying to get the rastream command to
>> work, I can get it to 'print' to stout.. But not to the file
>> getting some errors:
>>
>> [root at argus-client-test ~]# rastream -F ~/excel.rarc -S ?.?.
>> 150.134:561 -f /opt/ARGUS/SCRIPTS/argus_mysqlimport2.sh -M time 5m -
>> B 15s -w /opt/ARGUS/OUTBOUND/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>> rastream[4809]: 2008-02-27,08:16:33 output string /opt/ARGUS/
>> OUTBOUND/%Y/No such file or directory/-1074160872/argus.%Y.No such
>> file or directory.0.%H.%M.(null) requires $srcid
>> Segmentation fault
>>
>>
>>
>>
>> I'm using the -F option for format requirements. Here is my
>> excel.rarc file options:
>> #RA_PRINT_LABELS=0
>> RA_FIELD_DELIMITER=','
>> RA_PRINT_NAMES=none
>> RA_FIELD_SPECIFIER="srcid stime ltime dur saddr daddr proto sport
>> dport bytes sbytes dbytes pkts spkts dpkts dir"
>> RA_TIME_FORMAT="%Y-%m-%d,%H:%M:%S"
>> RA_USEC_PRECISION=6
>> RA_FILTER="not man"
>>
>>
>> /opt/ARGUS/SCRIPTS/argus_mysqlimport2.sh is the
>> "yourMysqlImport.sh" script you sent me...
>>
>> Any idea on the error code??? The directory is 'owned' by root...
>>
>> thanks
>>
>> Mark
>>
>>
>> From: Carter Bullard <carter <at> qosient.com>
>> Subject: Re: graph of bytes against protocols for network
>> loopdetection?
>> Newsgroups: gmane.network.argus
>> Date: 2008-02-28 17:09:51 GMT (2 hours and 13 minutes ago)
>> Hey Mark,
>> This is cool!! When you're ready to scale up, there are newer
>>
>> mechanisms
>> for creating the data repository and importing the data, using
>> programs
>> like rastream() and if you wanted to get a little closer to near
>> real-
>>
>>
>> time posting
>> of data, you could use rabins() with a few variations, out of the
>> argus-3.0 code set.
>>
>> As an example, if you wanted to insert data every 5 minutes, it can
>> be
>> as easy as:
>> rastream -S live.argus.stream -f yourMysqlImport.sh -M time 5m -B
>>
>>
>>
>> 15s \
>> -w /opt/ARGUS/OUTBOUND/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>
>> This would generate an argus archive broken out by year/month/day
>> containing
>> files every 5 minutes, and 15 seconds after then end of each 5 minute
>>
>>
>>
>> clock
>> boundary, your script would be run against the file, indexing the
>> data
>> and then
>> compressing the file. It could remove the file if you're not
>> interested in keeping
>> the archive etc......
>>
>>
>>
>>
>> yourMysqlImport.sh would be simply (using ./support/Config/
>> rastream.sh
>> to start):
>>
>> ------ begin yourMysqlImport.sh -----
>>
>> #!/bin/sh
>> #
>> # Argus Client Software. Tools to read, analyze and manage Argus
>> data.
>>
>>
>>
>> # Copyright (C) 2000-2008 QoSient, LLC.
>> # All Rights Reserved
>> #
>> # Script called by rastream, to process files.
>> #
>> # Since this is being called from rastream(), it will have only a
>> single
>> # parameter, filename,
>>
>>
>>
>> #
>> # Carter Bullard <carter <at> qosient.com>
>> #
>>
>> PATH="/usr/local/bin:$PATH"; export PATH
>> package="argus-clients"
>> version="3.0.0"
>>
>>
>>
>>
>> OPTIONS="$*"
>> FILES=
>> while test $# != 0
>> do
>> case "$1" in
>> -r) shift; FILES="$1"; break;;
>> esac
>> shift
>> done
>>
>> mysqlimport -# -L --host=192.168.150.204 --user='argus' --
>>
>>
>>
>> password='&&&&' --fields-terminated-by=',' --lines-terminated-
>> by='\n' \
>> --low-priority --verbose --columns='srcid, sdate, stime, ldate,
>> ltime, dur, saddr, daddr, proto, sport, dport, bytes, sbytes, dbytes,
>>
>>
>>
>> pkts, spkts, dpkts, dir' \
>> argus '$FILES'
>>
>> gzip $FILES
>> exit 0
>>
>> ----- end yourMysqlImport.sh -----
>>
>> Send mail to the list if/when you start, and we can all help.
>>
>> Carter
>>
>> On Feb 27, 2008, at 12:50 PM, Bartlett, Mark wrote:
>>
>>
>>
>>
>> > Hi Marten,
>> >
>> > I am working on an installation guide which will explain
>> > installation of the "Probe", "Loader", DB Server, and Web Server
>> and
>> > installation scripts which will create users, install cron jobs,
>>
>>
>>
>> > install the DB schema, etc.
>> >
>> > The Probe Machine 'moves' the /var/log/argus/argus.out file to
>> the /
>> > opt/ARGUS/OUTBOUND directory.
>> >
>> > The Loader Machine 'pulls' the argus files from remote Probe
>>
>>
>>
>> > Machines. Then the argus.out files are sent through the 'ra' tool
>> > to produce an ASCII comma delimited file.
>> >
>> > The ASCII file is imported into the DB using mysqlimport:
>> > mysqlimport -# -L --host=192.168.150.204 --user='argus' --
>>
>>
>>
>> > password='&&&&' --fields-
>> > terminated-by=',' --lines-terminated-by='\n' --low-priority --
>> > verbose --columns='sr
>> > cid, sdate, stime, ldate, ltime, dur, saddr, daddr, proto, sport,
>>
>>
>>
>> > dport, bytes, sbytes,
>> > dbytes, pkts, spkts, dpkts, dir' argus '/opt/ARGUS/OUTBOUND/
>> > argus.out'
>> >
>> > That's about it for file transfer and data insertion...
>> >
>>
>>
>>
>> > We are currently running ARGUS in a limited capacity (within our
>> > Research Network, monitoring mostly Dark IP's and Test Networks).
>> > But are working on doing a larger deployment (BETA). So we should
>>
>>
>>
>> > get a better idea of performance 'issues' in the near future.
>> >
>> > The DB schema I created contains a probe, location, and company
>> > tables so we can run queries against 'specific' probe locations
>>
>>
>>
>> > (DMZ, INTERNAL, EXTERNAL, Dark Nets, etc) Part of the Web GUI
>> > contains a probe insert page, to add new probes, etc.. It's
>> working
>> > pretty well for us at this point.
>> >
>> > If I get some 'spare time' I would like to try and create a
>>
>>
>>
>> > partitioned DB Schema for the ARGUS data based on date to speed up
>> > query times and alleviate Database Storage issues (archiving,
>> > dropping data, etc)....
>> >
>> > I'll let you know when the installation guides are finished and the
>>
>>
>>
>> > package has been posted for download.
>> >
>> > mark
>> >
>> > -----Original Message-----
>> > From: argus-info-bounces <at> lists.andrew.cmu.edu
>>
>>
>> [mailto:argus-info-bounces <at> lists.andrew.cmu.edu
>>
>> > ] On Behalf Of David Nolan
>> > Sent: Wednesday, February 27, 2008 12:32 PM
>> > To: argus-info <at> lists.andrew.cmu.edu
>> > Subject: Re: [ARGUS] graph of bytes against protocols for network
>>
>>
>>
>> > loopdetection?
>> >
>> > Mark,
>> >
>> > I would certainly be interested in seeing what you've got. In
>> fact I
>> > suspect that there will be enough interest in your setup that you
>> > might
>>
>>
>>
>> > just want to put it up somewhere for download.
>> >
>> > How are you feeding your data into the database? And at what scale
>> > are you
>> > doing this?
>> >
>> > -David
>> >
>> > --On Wednesday, February 27, 2008 09:50:38 -0500 "Bartlett, Mark"
>>
>>
>>
>> > <Mark.Bartlett <at> getronics.com> wrote:
>> >
>> >> Hi Marten,
>> >>
>> >> Our Argus data is being fed into a DB, and I have created an
>> "argus
>>
>>
>>
>> >> website" using basic html/php pages with JPGraph
>> >> (http://www.aditus.nu/jpgraph/)
>> >>
>> >> Attached is a sample of the Hourly Dashboard which shows 4
>> graphs -
>>
>>
>>
>> >> 1.
>> >> Events by Hour, 2. Daily Breakdown of Protocols, 3. Source Bytes
>> >> vs.
>> >> Dest Bytes, 4. Source Pkts vs. Dest Pkts.
>> >>
>> >> If you are 'feeding' your data into a db and think this might be
>>
>>
>>
>> >> something you would like to use I can send you our install
>> package.
>> >>
>> >> Thanks.
>> >>
>> >> Bartola
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: argus-info-bounces <at> lists.andrew.cmu.edu on behalf of
>> Carter
>>
>>
>>
>> >> Bullard
>> >> Sent: Wed 2/27/2008 9:13 AM
>> >> To: Marten Bauer
>> >> Cc: argus-info <at> lists.andrew.cmu.edu
>> >> Subject: Re: [ARGUS] graph of bytes against protocols for network
>>
>>
>>
>> >> loopdetection?
>> >> Hey Marten,
>> >> ragraph() is primarily a time series graphing program.
>> >> The easiest is to use racluster() to get the data totals:
>> >>
>> >> racluster -m proto -r file -s proto bytes - ip or arp
>>
>>
>>
>> >> Proto TotBytes
>> >> pim 624
>> >> ospf 36188
>> >> esp 18070651
>> >> gre 6637
>> >> ipv6 702
>> >> udp 5119990
>> >> tcp 744143899
>>
>>
>>
>> >> igmp 662
>> >> icmp 203201
>> >> arp 4177
>> >>
>> >> And then use something easy, like excel() or gnuplot().
>> >> What kind of platform are you using?
>> >>
>>
>>
>>
>> >> Carter
>> >>
>> >>
>> >> On Feb 27, 2008, at 1:52 AM, Marten Bauer wrote:
>> >>
>> >>> Hello,
>> >>>
>> >>> for detecting network loops I need a graph which
>>
>>
>> >>> prints the protocol on the x axes and the amount of
>>
>> >>> bytes on the y axes.
>> >>>
>> >>> I tried to archive this with ragraph, but I never got
>> >>> what I want.
>> >>>
>> >>> Is it possible with ragraph or another ra* tool to
>>
>>
>>
>> >>> generate such plot?
>> >>>
>> >>> Thx for helping
>> >>> Marten
>> >>>
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20080229/c0a04bb8/attachment.html>
More information about the argus
mailing list