Need some help writing "simple" query

[Carter Bullard]

Hey  Slacker T,
Your set of queries are not simple, and so understand that you're starting
with an advanced set of questions to solve.   It will take a bit to get the
right final answer but the approach should become logical after you go
through a few steps toward building your result.

First you should upgrade to argus-3.0.6.  That will fix a number of bugs
and will be a good starting point.  Your argus-3.0.4 data is fine, but the
upgrade will solve some problems.

Rasplit() is generating your basic chunks, so we'll work with these files to get going.
When you decide that there's a reason to look at a specific chunk, we'll process that
whole 5 minute chunk.  Doing other granularities is easy, but will add complexity so lets
start with your 5 minute files.

You're ultimately looking for individual top talkers, not top flows or top matrix, so
we'll need to process the data looking for individual addresses.  This is a complicated
idea, because you have data that has pairs of addresses in it, but you're interested in
single addresses.  Not hard, but just complicated at first.  The secret is the " -M rmon "
option.

Uploaded and downloaded generally refer to the data that is transported, minus the
transport headers, we can do that no problem, so to look at real data loaded, we'll
look at the " appbytes " metric.

With all of this, lets try to get a list of single address oriented data, and try to rank them
by the data they send, the data they receive, both sent and received, and then total
bytes, regardless of protocol, and regardless of who initiated the connection.  

Hopefully this will be helpful.  

With any of your 5 minute files, try this:

   racluster -r 5.min.file -w addrs.file -M rmon -m saddr - ip

This will generate a set of argus data, that is oriented around the
saddr field as the key object.  By using the " -M rmon " option,
racluster() will move all the addresses into the saddr field, so that
simple aggregation will generate a good set of data.

Then with this addrs.file, read the file to generate your top talkers.  Lets do
N=20.

   rasort -r addrs.file -m sappbytes -No 20
   rasort -r addrs.file -m dappbytes -No 20
   rasort -r addrs.file -m sappbytes dappbytes -No 20
   rasort -r addrs.file -m appbytes -No 20

The -No option, specifies the number of records to output.

The source metrics report on the data that the object sourced (sent),
and the dst metrics are the data where the obj was the destination,
so its the data that it received.  If this corresponds to your upload/download,
then success ?

The object field that is useful is the "saddr" field, because that was what we
aggregated on (" -m saddr "), as as result, it has the actual address your
interested in.   In the above commands, to specify the fields you want to
print, use the -s option.
   
   -s stime dur saddr spkts dpkts sbytes dbytes sappbytes dappbytes

You can be a bit more clever, and aggregate on the CIDR saddr, so that you
get aggregate data for subnets, rather than actual addresses:

   racluster -r 5.min.file -w cidrs.file -M rmon -m saddr/24 - ip

Now, the address data is pretty straight foward.  However, to couple that with
L4 data is a bit more interesting.  This is complicated enough to wait for another
email.

Give this a try, and if it seems like its working, great, we can move onto how
to take a list of IP addresses and try to do your port reporting.  Perl programs
like raports() will really help here.


Carter

On Sep 25, 2012, at 6:27 PM, Slacker T <slackert at gmail.com> wrote:

> Hello,
> 
> I'm trying to tie argus into our alerting system. The goal is to take
> a snapshot of current usage when certain firewall/switch interfaces
> exceed a threshold. I'm having difficulty writing a proper command to
> get the information I'm looking for. I'd like to be able to show the
> top talkers for a given period of time based first on total amount
> downloaded, uploaded, both, or a total of both. Then from that list
> I'd like to take the IP and show similar information broken up across
> L4 ports (total amount downloaded, uploaded, both, or a total of
> both).
> 
> I've gone through the man pages and nsmwiki and I'm a bit overwhelmed.
> I'm excited by the granularity of detail argus can provide, and how
> easy it is to keep a historical record of traffic, but the learning
> curve seems quite steep. I'm thinking that racluster and rasort is
> where I need to focus on, but I'm having trouble getting the info I'm
> looking for.
> 
> Somewhat relevant info:
> I'm collecting argus data from a SPAN port with the following command:
> argus -mAJZRU 1024 -i eth3 -P561 -B 127.0.0.1 -w /argus/argus.log -d
> 
> I use rasplit to give me 5 minute chunks to work with (I've got a
> script that does some log rotation, but this is the core of it):
> /usr/bin/rasplit -r $DATADIR/argus.log-to_be_archived -M time 5m -w
> $DATADIR'/archive/%Y/%m/%d/argus.%H.%M.%S.arg3'
> 
> # argus --version
> Argus Version 3.0.4
> 
> This is about as far as I have got, if I understand correctly this
> uses racluster to total the bytes used by the source addresses in the
> argus.14.00.00.arg3 source, then filter out only the IP records and
> pipe them to rasort. Rasort sorts on bytes and prints the source
> address and byte count..then head shows the top 5.
> 
> racluster -m saddr -r argus.14.00.00.arg3 -w - - ip | rasort -m bytes
> -s saddr bytes | head -n 5
>     10.210.20.214   50509905
>      10.210.21.33   49000815
>     10.210.20.110   48677815
>       10.210.21.4   37675239
>   192.168.100.159   33097697
> ...
> 
> If I'm on the right path, great, if not, what am I doing wrong? How do
> I get the byte count to show per port and per destination IP?
> 
> Thanks!
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120925/cf9db456/attachment.bin>