Using Argus to generate daily stats in OpenWrt

[Graeme Russ]

Hi Carter,

Thanks for all the info - I think I have enough now to tinker with.

I'm off on holidays for the next week and a bit - I'll get back to it when
I'm back

Hopefully I can contribute some useful code back to you and the OpenWrt
guys to make OpenWrt + Argus more novice friendly :)

Regards,

Graeme


On Tue, Apr 16, 2013 at 9:55 PM, Carter Bullard <carter at qosient.com> wrote:

> Hey Graeme,
> So assuming you use rasplit() to generate 5 minute data files......
>
> To get a sense of how to do NAT tracking, print argus fields that are not
> modified by NAT'ing, the source user data field is a good example.  For
> specific protocols, like RTP, TCP, DNS, there are sequence numbers
> that can't change.  For TCP, look at the " stcpb ", which is the source tcp
> base sequence number.  Within a short time frame, these are very unique,
> like world-wide unique, and you can match them to find records
> that are NAT'd.  You can sort on " stcpb " and that will get the records
> that should be mapped adjacent in the output list.
>
>    racluster -r 5.minute.file -w - - tcp | \
>    rasort -m stcpb -s stime dur saddr sport dir daddr dport spkts dpkts
> sbytes dbytes stcpb dtcpb
>
> If you wanted to process an entire day's worth of data:
>
>    racluster -r *.minute.files -w - - tcp | \
>    rasort -m stcpb -s stime dur saddr sport dir daddr dport spkts dpkts
> sbytes dbytes stcpb dtcpb
>
> You should get similar results.
>
> With this output, you should see that the NAT correction should be very
> straightforward.  You can have a program modify the ascii output to
> generate
> the data you want, then use raconvert() to convert the ascii records to
> binary
> records, so you can then do the metrics processing you want to do.
>
> You can capture 32-64 bytes of user data, and use the " suser " field to
> provide
> matching patterns to make the conversion.  We don't currently sort on user
> data, but
> that is easy to add.
>
> Carter
>
>
> On Apr 15, 2013, at 8:14 PM, Graeme Russ <graeme.russ at gmail.com> wrote:
>
> Hi Carter,
>
> On Apr 16, 2013 6:29 AM, "Carter Bullard" <carter at qosient.com> wrote:
> >
> > Hey Graeme,
> > Scripts that do NAT tracking?   Not currently in the free distribution,
> but
> > I have programs that do multi-probe correlation on the fly, that deal
> with
> > the address mappings.  What are you doing this for?  Yourself?
>
> Yes, this is for a home network so no big loads and no commercial
> interests. I just want to make sure I know what is happening on my own
> network so I can make some educated decisions later on.
>
> > How you collect and archive the data will drive which programs you will
> > need for metrics.   Be sure and read how rasplit() and rastream() work
> > to build data files to make your metric generation straight forward.
> > We recommend using rasplit() to generate data files that cover 5 minute
> > ranges, organized by year/month/day.  Makes generating daily, hourly
> > metrics pretty easy.
>
> Thanks for the pointers - I'll look at them in more detail.
>
> > You will want to use data aggregation, like that type provided by
> racluster()
> >  to generate the metrics you want.  Sounds like you want to realize
> > something like daily matrix data:
> >
> >    racluster -r daily.file -m matrix -s stime dur saddr daddr pkts bytes
> rate load ........
> >
> > matrix data will report on IP address <-> IP address metrics, removing
> the
> > protocol or port values.
>
> That will be a great start - Of course the NAT adds a layer of complexity
> as the best I'll have is a set of three logs
>
>  - Remote IP <-> Gateway IP
>  - Gateway IP <-> Local LAN IP
>  - Gateway IP <-> Local WLAN IP
>
> So I will need to do NAT resolution. I imagine that the timestamps for the
> packets on either side of the NAT will be pretty close and if I had a
> complete set of three files (WAN, LAN, WLAN) for any given interval I could
> successfully reconstruct the Remote IP <-> Local IP map.
>
> Right now, I don't care about traffic between the local LAN and WLAN -
> Only traffic going through the WAN (ADSL Modem) needs to be measured.
>
> For WAN performance metrics (5 minute bandwidth measurement) I only care
> about the WAN packets. I'm thinking:
>
>  - 1 hour WAN/LAN/WLAN logs, processed every 24 hours and archived
>  - 5 minute WAN logs, processed every 5 minutes and discarded (only
> calculated stats saved)
>
> For the WAN stats, I'm thinking, for each 5 minute interval:
>  - Total Tx and Rx bytes
>  - Per Remote-IP Tx and Rx bytes
>
> > read the racluster.1 man page.  Depending on whether you want daily
> metrics
> > from daily data, or hourly stats from weekly data, or whatever, there
> are different
> > programs to generate the time bins, and report on aggregate metrics per
> period.
> > Programs like rabins() are going to be important if you want to do
> something
> > clever.
> >
> > Argus will capture URL's if you set the ARGUS_CAPTURE_DATA_LEN
> > to something like 256 bytes.
>
> Good to know - I think initially I'll keep this up my sleeve as I'm not
> sure the router can handle it :)
>
> Regards,
>
> Graeme
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130416/8157f678/attachment.html>