Using Argus to generate daily stats in OpenWrt

[Carter Bullard]

Hey Graeme,
So assuming you use rasplit() to generate 5 minute data files......

To get a sense of how to do NAT tracking, print argus fields that are not
modified by NAT'ing, the source user data field is a good example.  For
specific protocols, like RTP, TCP, DNS, there are sequence numbers
that can't change.  For TCP, look at the " stcpb ", which is the source tcp
base sequence number.  Within a short time frame, these are very unique,
like world-wide unique, and you can match them to find records
that are NAT'd.  You can sort on " stcpb " and that will get the records
that should be mapped adjacent in the output list.

   racluster -r 5.minute.file -w - - tcp | \
   rasort -m stcpb -s stime dur saddr sport dir daddr dport spkts dpkts sbytes dbytes stcpb dtcpb

If you wanted to process an entire day's worth of data:

   racluster -r *.minute.files -w - - tcp | \
   rasort -m stcpb -s stime dur saddr sport dir daddr dport spkts dpkts sbytes dbytes stcpb dtcpb

You should get similar results.

With this output, you should see that the NAT correction should be very
straightforward.  You can have a program modify the ascii output to generate
the data you want, then use raconvert() to convert the ascii records to binary
records, so you can then do the metrics processing you want to do.

You can capture 32-64 bytes of user data, and use the " suser " field to provide
matching patterns to make the conversion.  We don't currently sort on user data, but
that is easy to add.

Carter


On Apr 15, 2013, at 8:14 PM, Graeme Russ <graeme.russ at gmail.com> wrote:

> Hi Carter,
> 
> On Apr 16, 2013 6:29 AM, "Carter Bullard" <carter at qosient.com> wrote:
> >
> > Hey Graeme,
> > Scripts that do NAT tracking?   Not currently in the free distribution, but
> > I have programs that do multi-probe correlation on the fly, that deal with
> > the address mappings.  What are you doing this for?  Yourself?
> 
> Yes, this is for a home network so no big loads and no commercial interests. I just want to make sure I know what is happening on my own network so I can make some educated decisions later on.
> 
> > How you collect and archive the data will drive which programs you will
> > need for metrics.   Be sure and read how rasplit() and rastream() work
> > to build data files to make your metric generation straight forward.
> > We recommend using rasplit() to generate data files that cover 5 minute
> > ranges, organized by year/month/day.  Makes generating daily, hourly
> > metrics pretty easy.
> 
> Thanks for the pointers - I'll look at them in more detail.
> 
> > You will want to use data aggregation, like that type provided by racluster()
> >  to generate the metrics you want.  Sounds like you want to realize
> > something like daily matrix data:
> >
> >    racluster -r daily.file -m matrix -s stime dur saddr daddr pkts bytes rate load ........
> >
> > matrix data will report on IP address <-> IP address metrics, removing the
> > protocol or port values.
> 
> That will be a great start - Of course the NAT adds a layer of complexity as the best I'll have is a set of three logs
> 
>  - Remote IP <-> Gateway IP
>  - Gateway IP <-> Local LAN IP
>  - Gateway IP <-> Local WLAN IP
> 
> So I will need to do NAT resolution. I imagine that the timestamps for the packets on either side of the NAT will be pretty close and if I had a complete set of three files (WAN, LAN, WLAN) for any given interval I could successfully reconstruct the Remote IP <-> Local IP map.
> 
> Right now, I don't care about traffic between the local LAN and WLAN - Only traffic going through the WAN (ADSL Modem) needs to be measured.
> 
> For WAN performance metrics (5 minute bandwidth measurement) I only care about the WAN packets. I'm thinking:
> 
>  - 1 hour WAN/LAN/WLAN logs, processed every 24 hours and archived
>  - 5 minute WAN logs, processed every 5 minutes and discarded (only calculated stats saved)
> 
> For the WAN stats, I'm thinking, for each 5 minute interval:
>  - Total Tx and Rx bytes
>  - Per Remote-IP Tx and Rx bytes
> 
> > read the racluster.1 man page.  Depending on whether you want daily metrics
> > from daily data, or hourly stats from weekly data, or whatever, there are different
> > programs to generate the time bins, and report on aggregate metrics per period.
> > Programs like rabins() are going to be important if you want to do something
> > clever.
> >
> > Argus will capture URL's if you set the ARGUS_CAPTURE_DATA_LEN
> > to something like 256 bytes.
> 
> Good to know - I think initially I'll keep this up my sleeve as I'm not sure the router can handle it :)
> 
> Regards,
> 
> Graeme

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130416/55f7ca59/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130416/55f7ca59/attachment.bin>