Graphing traffic of the top 10 ports

Thu Dec 29 23:22:01 EST 2011

Hey Jeffrey,
The last email I sent presented a way to prepare data in a general way,
so you can graph it with any tools.  Its a somewhat complicated process,
but not too terrible.  First aggregate, using the intended flow key, to the
time granularity you want, then pick out the objects that you want to graph,
using a filter.  Hopefully I didn't scare you off with that earlier description.

For what you want to graph, however, ragraph() on its own, should be able
to do the trick.  ragraph() is a perl script that, in its own right, is quite
complex.  It uses perl's associative arrays extensively to do a lot of magic
on tracking multiple objects and their time series data.  Using the "-N x"
option to ragraph() , you can generate the top x talkers of whatever metric
you are printing, but it will be the "top whatever" from the entire file.

Because you are dealing with one object and a simple scalar metric, 
you can generate your graph with a single call to ragraph, using:

   ragraph pkts dport -m dport -M 30s -r file -w ragraph.png -N 10 -nn -t -24h \
       -title "Top 10 destination ports" - \(tcp and syn and synack\) or udp  

So, graph the pkts for the top 10 dst ports during the last 24 hours of
data from file "file".  While you are specifying 30s bins, it maybe more
useful to do 2 or 5 minute bins, as the resolution is pretty high (24 * 60 * 2).

The real difference from your earlier attempt is removal of the "pkts" from your
flow key definition, the use of -N x in ragraph() to graph the top x objects, and
the filter to make sure that the dport is a real service port for TCP traffic.

Using argus-clients-3.0.5.25 and above, when specifying the dport or
sport in the flow key definition, the ra* tools will add the proto automatically
so that is no longer needed.

I have made some changes in argus-clients-3.0.5.28 that will allow you
to do the other "top talker" strategy, where the top whatever is determined
for each time period, rather than the entire file, using a variation of the
"-N x" option.  I'll elaborate on that if you would find it useful.

Please give this a try, and if you have problems, don't hesitate to send
email to the list.

Carter

On Dec 29, 2011, at 3:22 PM, Carter Bullard wrote:

> Hey Jeffrey,
> You just need to make a few modifications, and currently you'll have to
> run multiple sets of commands to accomplish your graph.  This may
> seem a bit complicated, but its not really.
> 
> You are trying to draw a time series graph for the last 24 hours of the
> top 10 ports in your file, with a granularity of 30 seconds.  This graph
> will have  (24 * 60 * 2) slots in the X axis, and up to 10 values in the Y axis.
> 
> The first question is, do you want to track the same "top 10 ports" from the
> entire file, or are you interested in the "top 10 ports"  for each 30 second
> graph interval?  The simpler graph is to track 10 ports over your entire
> 24 hour period.  That way you're looking at only 10 ports.
> 
> If you are interested in the top 10 ports for each 30 second period
> that is another story, and I can answer that in another email.
> 
> The first thing you'll need for your graph, is the list of  " top 10 " ports.
> This is not easy to do with a single pass through the data, so we'll have
> to generate the port list, and then use it to filter the data to generate the graph.
> I'll break it down such that we can do this efficiently, with limited passes
> against the original file.
> 
> Because you are looking to draw with 30 seconds granularity, we 
> should aggregate the data to dst ports, preserving the time resolution
> for the graph.  To do this, use rabins().  rabins() is basically racluster(), but
> confined to the bin strategy and size specified on the command line.
> 
>    rabins -r file -t -24h -m proto dport -M hard time 30s -w data.file - \(tcp and syn and synack\)  or udp
> 
> OK, aggregate data for the last 24 hours from file, using the "proto dport"
> as the aggregation key, into 30 second hard time bins, and write the data to
> the the file " data.file ", this will generate a file that has all the ports in them
> aggregated in 30 second bins.  The filter is there to ensure that the "dst port" is
> actually the service port.  When dealing with TCP, the dst port is the service
> port only when we've seen the connection setup (syn and synack).
> 
> OK now we have an aggregated intermediate file, " data.file ", from which we can
> determine the top 10 ports.
> 
>    racluster -r data.file -w - - not frag | rasort -m pkts -w - | ra -N 10 -s dport -n -L -1 > /tmp/ra.top.10.ports
> 
> OK, just aggregate up the data using racluster().  Pipe the output into rasort(), like you did
> before, and let ra() trim the output to the top 10.  If you are interested in the next 10, you
> will use another -N option, such as "-N 10-20".
> 
> Write the ascii port numbers (-n) to an intermediate file, without the column label (-L -1).
> 
> This will generate the list of top 10 ports.  To turn this list into a port filter, we'll have to use
> some sed.1 magic, such as.
> 
>    % sed -e "2,\$s/^/or /" /tmp/ra.top.10.ports
> 
> This just prepends a "or " in front of all the lines except the first one, so putting it all together,
> using the intermediate data file, and the port list, you can run ragraph() like so:
> 
>    ragraph pkts dport -M time 30s -r /tmp/data.file -w graph.png ……  - port \(`sed -e "2,\\$s/^/or /" /tmp/ra.top.10.ports` \)
> 
> Then when you're done, delete the temporary files.
> 
> OK, I know that was a lot, but hopefully that helps you out.  If there are problems,
> don't hesitate to send more email !!!!!!
> 
> 
> Carter
> 
> 
> On Dec 29, 2011, at 7:01 AM, Jeffrey Everling wrote:
> 
>> Hello,
>> 
>> Can anyone point me in the right direction?
>> 
>> So far I have:
>> 
>> racluster -r /var/log/argus/argus-id5-log -t -24h -m proto pkts dport -w
>> - | rasort -m pkts -r - -w - | ra -N 9 -r - -w - | ragraph pkts dport -M
>> 30s -r - -w /usr/local/www/data/img/top10ports-24h.png -nn -title "Top
>> 10 destination ports
>> 
>> But I think I need a replacement for racluster, because he graphs the
>> clustered data.
>> 
>> What I want:
>> 
>> A graph that shows the traffic of the top 10 busiest ports, But not
>> summarized like the above command does.
>> 
>> -- 
>> Kindly regards,
>> 
>> 
>> Jeffrey Everling
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20111229/b31a5f1c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20111229/b31a5f1c/attachment.bin>