Using Argus to generate daily stats in OpenWrt
Carter Bullard
carter at qosient.com
Mon Apr 15 16:29:46 EDT 2013
Hey Graeme,
Scripts that do NAT tracking? Not currently in the free distribution, but
I have programs that do multi-probe correlation on the fly, that deal with
the address mappings. What are you doing this for? Yourself?
How you collect and archive the data will drive which programs you will
need for metrics. Be sure and read how rasplit() and rastream() work
to build data files to make your metric generation straight forward.
We recommend using rasplit() to generate data files that cover 5 minute
ranges, organized by year/month/day. Makes generating daily, hourly
metrics pretty easy.
You will want to use data aggregation, like that type provided by racluster()
to generate the metrics you want. Sounds like you want to realize
something like daily matrix data:
racluster -r daily.file -m matrix -s stime dur saddr daddr pkts bytes rate load ........
matrix data will report on IP address <-> IP address metrics, removing the
protocol or port values.
read the racluster.1 man page. Depending on whether you want daily metrics
from daily data, or hourly stats from weekly data, or whatever, there are different
programs to generate the time bins, and report on aggregate metrics per period.
Programs like rabins() are going to be important if you want to do something
clever.
Argus will capture URL's if you set the ARGUS_CAPTURE_DATA_LEN
to something like 256 bytes.
Carter
On Apr 14, 2013, at 12:13 AM, Graeme Russ <graeme.russ at gmail.com> wrote:
> Hi Carter,
>
> On Apr 14, 2013 3:43 AM, "Carter Bullard" <carter at qosient.com> wrote:
> >
> > Hey Graeme,
> > Glad to see you got things going... Anything I need to do to the code
> > base to make it better ?
>
> I don't think so. OpenWrt has moved on a bit since the comments in the INSTALL file.
>
> Here is my quick and dirty build script:
>
> #! /bin/bash
>
> export BASE_DIR=/home/graeme/devel/openwrt
> export STAGING_DIR=$BASE_DIR/staging_dir/
> export BUILD_DIR=$BASE_DIR/build_dir/
>
> export TOOLCHAIN_SUB_VERSION=uClibc-0.9.33.2
> export TOOLCHAIN_VERSION=toolchain-mips_r2_gcc-4.6-linaro_$TOOLCHAIN_SUB_VERSION
>
> export TARGET_DIR=$BUILD_DIR/target-mips_r2_$TOOLCHAIN_SUB_VERSION
>
> export TOOLCHAIN_ARCH=mips-openwrt-linux-uclibc
>
> export PATH=$PATH:$STAGING_DIR$TOOLCHAIN_VERSION/bin/
> export PATH=$PATH:$STAGING_DIR$TOOLCHAIN_VERSION/libexec/gcc/mips-openwrt-linux-uclibc/4.6.3/
>
> export AR=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-ar
> export AS=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-as
> export LD=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-ld
> export NM=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-nm
> export CC=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-gcc
> export CPP=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-cpp
> export GCC=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-gcc
> export CXX=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-g++
> export RANLIB=$STAGING_DIR$TOOLCHAIN_VERSION/bin/$TOOLCHAIN_ARCH-ranlib
>
> # Prevent malloc/realloc from being redefined as rpl_malloc/rpl_realloc
> export ac_cv_func_malloc_0_nonnull=yes
> export ac_cv_func_realloc_0_nonnull=yes
>
> # Set the kernel version used in OpenWrt (not the host system)
> export ac_cv_linux_vers=3.3.8
>
> export LDFALGS="-static"
> export CFLAGS="-Os -s"
>
> cp $BUILD_DIR$TOOLCHAIN_VERSION/$TOOLCHAIN_SUB_VERSION/include/rpc/types.h $STAGING_DIR$TOOLCHAIN_VERSION/include/rpc/
>
> ./configure --host=mips-openwrt-linux --with-libpcap=$TARGET_DIR/libpcap-1.1.1
>
> make
>
> >
> > So your box is doing NAT, and that is why all the local traffic is addressed
> > to/from the gateway address ? Argus captures enough flow info so
> > you can track flows through NAT'ing.
>
> Do you have any scripts which do this (it's almost like an SQL join)
>
> >
> > You can have argus monitor both the interfaces on the box at once,
> > assigning different srcid's to each monitored interface. Your clients
> > can then pick which observation domain they want to use to calculate
> > the metrics you're interested in.
>
> Speaking of metrics, are you aware of any scripts that would gererate the metrics I mentioned earlier?
>
> >
> > So for aggregate interface stats, use the ethernet addresses of your
> > lan0.2 interface to do the aggregates. Use the lan0.1 stats to do
> > IP address and port based stats, and compare them, on occasion,
> > to make sure that both interface numbers seem reasonable.
>
> I need to generate specific local/remote device metrics (which local device is causing what traffic to which remote sites) so I need to resolve the NAT on the fly almost)
>
> Can Argus log http URLs btw?
>
> Regards,
>
> Graeme
> >
> > Carter
> >
> >
> > On Apr 12, 2013, at 9:13 PM, Graeme Russ <graeme.russ at gmail.com> wrote:
> >
> >> Hi Carter,
> >>
> >> Thanks for the encouragement - I managed to get Argus compiled for OpenWrt Attitude Adjustment after a few false starts (the old rpl_malloc/rpl_realloc autoconf issue and the toolchain missing rpc/types.h)
> >>
> >> So now I can monitor lan0.2 which connects to the ADSL modem with:
> >>
> >> root at OpenWrt:/tmp# ./argus -i lan0.1 -B 10.1.1.1 -P 561
> >>
> >> and collect the data remotely (on a server connected to lan0.1) with:
> >>
> >> [graeme at fs1 argus-3.0.6]$ ra -S 10.1.1.1:561 - ip
> >>
> >> One issue that I have is that all the local traffic is addressed to/from the gateway address:
> >>
> >> 10:13:44.337312 * tcp 10.2.1.2.51206 -> 10.220.60.110.newacc 1 66 REQ
> >> 10:13:45.545508 * tcp 130.239.18.172.ircu-3 <?> 10.2.1.2.58990 6 538 CON
> >> 10:13:45.651617 * tcp 10.3.1.224.49474 <?> 10.1.1.21.ssh 10 1396 CON
> >> 10:13:45.943295 * tcp 10.1.1.21.54524 -> 10.1.1.1.monito 14 1476 CON
> >> 10:13:48.832722 * udp 10.2.1.2.37307 <-> 10.2.1.1.domain 4 726 CON
> >> 10:13:50.333756 * s tcp 10.2.1.2.51206 -> 10.220.60.110.newacc 1 66 REQ
> >> 10:13:50.576155 * tcp 10.2.1.2.51208 -> 125.56.205.35.http 26 21019 CON
> >> 10:13:50.577190 * udp 10.2.1.2.62523 <-> 10.2.1.1.domain 5 844 CON
> >> 10:13:50.577597 * udp 10.2.1.2.61546 <-> 10.2.1.1.domain 2 379 CON
> >> 10:13:50.578009 * udp 10.2.1.2.53972 <-> 10.2.1.1.domain 5 789 CON
> >> 10:13:50.578436 * udp 10.2.1.2.nacnl <-> 10.2.1.1.domain 2 555 CON
> >> 10:13:50.578860 * udp 10.2.1.2.33314 <-> 10.2.1.1.domain 2 291 CON
> >> 10:13:50.579301 * udp 10.2.1.2.63472 <-> 10.2.1.1.domain 5 835 CON
> >> 10:13:50.618052 * tcp 10.2.1.2.51209 -> 199.27.75.193.http 3 190 CON
> >> 10:13:50.618224 * tcp 10.2.1.2.51210 -> 199.27.75.193.http 3 190 CON
> >> 10:13:50.618487 * tcp 10.2.1.2.51211 -> 199.27.75.193.http 3 190 CON
> >> 10:13:50.619067 * udp 10.2.1.2.14520 <-> 10.2.1.1.domain 2 605 CON
> >>
> >>
> >> I can, of course, monitor the local side of the router (in this case the wireless interface) with:
> >> root at OpenWrt:/tmp# ./argus -i lan0.1 -B 10.1.1.1 -P 561
> >>
> >> 11:07:38.070886 e tcp 10.3.1.224.52688 -> 125.56.205.25.http 3 162 FIN
> >> 11:07:38.070979 e tcp 10.3.1.224.52689 -> 125.56.205.25.http 3 162 FIN
> >> 11:07:38.071069 e tcp 10.3.1.224.52693 -> 125.56.205.25.http 3 162 FIN
> >> 11:07:38.071199 e tcp 10.3.1.224.52690 -> 125.56.205.25.http 3 162 FIN
> >> 11:07:38.071362 e tcp 10.3.1.224.52738 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071434 e tcp 10.3.1.224.52739 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071490 e tcp 10.3.1.224.52740 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071546 e tcp 10.3.1.224.52737 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071602 e tcp 10.3.1.224.52742 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071657 e tcp 10.3.1.224.52741 -> 125.56.205.233.http 3 162 FIN
> >> 11:07:38.071740 e tcp 10.3.1.224.52734 -> 118.214.198.126.http 3 162 FIN
> >> 11:07:38.071831 e tcp 10.3.1.224.52736 -> 118.214.198.126.http 3 162 FIN
> >> 11:07:38.071931 e tcp 10.3.1.224.52779 -> 125.56.204.128.http 3 162 FIN
> >> 11:07:38.072024 e tcp 10.3.1.224.52780 -> 125.56.204.128.http 3 162 FIN
> >> 11:07:38.072114 e tcp 10.3.1.224.52781 -> 125.56.204.128.http 3 162 FIN
> >> 11:07:38.072207 e tcp 10.3.1.224.52762 -> 125.56.204.128.http 3 162 FIN
> >> 11:07:38.072298 e tcp 10.3.1.224.52701 -> 125.56.205.48.http 3 162 FIN
> >> 11:07:38.072391 e tcp 10.3.1.224.52669 -> 125.56.204.88.http 3 162 FIN
> >>
> >> But what I want is the total aggregated stats on the ADSL connection. So now I need to figure out a way to match each entry with the device on the local LAN - any ideas?
> >>
> >>
> >> Regards,
> >>
> >> Graeme
> >>
> >>
> >> On Fri, Apr 12, 2013 at 11:26 PM, Carter Bullard <carter at qosient.com> wrote:
> >>>
> >>> Hey Grame,
> >>> Argus compiled and ran great on OpenWRT years ago, but I haven't done anything OpenWRT related in a while, so not sure if it will be easy still. Should be able to monitor wan0, no problem, if memory serves, but it is hardware specific, ...., which interfaces OpenWRT can monitor.
> >>>
> >>> Collecting agrus data from an argus on OpenWRT is easy and generating the type of metrics you're interested is straight forward.
> >>> If you want to store these values in rrd's, we have perl scripts ( ragraph ) that can be used to maintain rrd's for your values. No problem.
> >>>
> >>> So I say go for it. We'll help you get there !!!
> >>> Carter
> >>>
> >>>
> >>> On Apr 11, 2013, at 7:09 PM, Graeme Russ <graeme.russ at gmail.com> wrote:
> >>>
> >>> > Hi All,
> >>> >
> >>> > I've installed OpenWrt on my WiFi router and now I'm looking for a way to generate network statistics for the ADSL connection. The ADSL connection is through a separate router, so generating the statistics should be a simple matter of processing the packets passing through the WAN0 interface. Daily statistics I would like to generate include:
> >>> >
> >>> > - Total inbound and outbound data
> >>> > - Inbound and outbound data per local IP address/port/protocol (TCP/UDP)
> >>> > - Inbound and outbound data per remote IP address/port/protocol (TCP/UDP)
> >>> > - Inbound and outbound data per unique local IP/Remote IP/port/protocol (TCP/UDP)
> >>> > - Average inbound and outbound throughput per 5 minute interval (total bytes/second)
> >>> >
> >>> > Two questions
> >>> > - Is Argus the right solution?
> >>> > - How hard will it be to get Argus running in OpenWrt Attitude Adjustment?
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Graeme
> >>> >
> >>> >
> >>
> >>
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130415/bb99eacd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130415/bb99eacd/attachment.bin>
More information about the argus
mailing list