normalized appbyte ratio for producer/consumer relationship

[John Gerth]

Nice example. I'm looking forward to using this.

As your example shows, this metric is available for any existing argus
files that were created containing appbyte values. I'm assuming that if
the sensor wasn't configured to capture those, 'abr' is not available.


--
John Gerth      gerth at graphics.stanford.edu  Gates 378   (650) 725-3273 fax 725-6949

On 5/6/2013 11:19 AM, Carter Bullard wrote:
> Hey John,
> OK, so I've implemented " abr " as a new metric, using our normalized equation:
> 
>    abr = (sappbytes - dappbytes)/(sappbytes + dappbytes)
> 
> This generates values between +1.0 - -1.0.  +1.0 means that all the app bytes
> were from the source, indicating that the source is a pure PRODUCER, and the
> destination is a pure CONSUMER.  You see this in FTP PUT file transfers,
> as an example.  The sign bit reverses this relationship.
> 
> -0.0 denotes the special case, when there are no appbytes seen.
> 
> In the new argus-clients that I'll put up later today, you can print this out using:
> 
>    ra -r argus.data -s +abr
> 
> You can also do operations using this metric, such as filter and generate histograms.
> Here is a run that I did to show how this maybe used in an anomaly detection
> application.  Here is the simple frequency distribution for all the internal DNS
> requests made to my local DNS server from a specific client, for all of 2013:
> 
> thoth:06 carter$ pwd
> /Volumes/Data/Archive/QoSient/192.168.0.68/2013
> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -R . -s mean stddev - udp port domain and src pkts 1 and dst pkts 1
> N = 1027764  mean = -0.726195  stddev =  0.140532  max = 0.000000  min = -0.909605
>            median = -0.749129     95% = -0.292517
>              mode = -0.782609
>  Class      Interval         Freq    Rel.Freq     Cum.Freq          Mean     StdDev 
>      1   -1.000000e+00     225379    21.9291%     21.9291%     -0.815238   0.000650
>      2   -8.000000e-01     738148    71.8208%     93.7498%     -0.740887   0.043837
>      3   -6.000000e-01      10553     1.0268%     94.7766%     -0.534672   0.048717
>      4   -4.000000e-01      35511     3.4552%     98.2318%     -0.283067   0.021374
>      5   -2.000000e-01        250     0.0243%     98.2561%     -0.162791   0.000051
>      6    0.000000e+00      17923     1.7439%    100.0000%      0.000000   0.000000
>      7    2.000000e-01          0     0.0000%    100.0000%    
>      8    4.000000e-01          0     0.0000%    100.0000%    
>      9    6.000000e-01          0     0.0000%    100.0000%    
>     10    8.000000e-01          0     0.0000%    100.0000%    
> 
> 
> OK, should be very clear, that my host is a net CONSUMER of DNS data, not a net PRODUCER 
> because the " abr <= 0 ".  The corollary holds true, the local DNS service is a net PRODUCER of
> data, and not a net CONSUMER of data, from the prospective of this particular end system.
> So testing filters like this:
>    ra -r daily.file - abr gt 0 and port domain and src pkts 1 and dst pkts 1
> 
> Should reveal flows that deserve a closer look.
> 
> OK, there were a lot of flows where the ( abr == 0 ), which was surprising.
> When DNS experiences a ServFail, the response is the same as the request, just with an error bit
> set in the DNS header.  QoSient had a big issue in Jan, 2013, when 17923 DNS ServFail failures
> occurred, so that is where the ( abr == 0 ) flows occured.  Important to know this when evaluating
> DNS as a channel for CONSUMER to PRODUCER conversion.
> 
> But for DNS health and operability, looking for flows where the ( sappbytes == dappbytes ) is
> also a pretty interesting thing to look for.
> 
> Hope this is helpful,
> 
> Carter
> 
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
> 
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>