normalized appbyte ratio for producer/consumer relationship

[Carter Bullard]

Hey John,
If no appbytes, currently we return -0.0, but the library knows if there are
appbytes or not, so we can return nada, when printing out the values.
Right now, when using xml format, you won't get a value.

Having problems getting my compiler to tell the difference between 0.0 and -0.0,
but should hopefully have this working by this afternoon.

Carter


On May 6, 2013, at 2:37 PM, John Gerth <gerth at graphics.stanford.edu> wrote:

> Nice example. I'm looking forward to using this.
> 
> As your example shows, this metric is available for any existing argus
> files that were created containing appbyte values. I'm assuming that if
> the sensor wasn't configured to capture those, 'abr' is not available.
> 
> 
> --
> John Gerth      gerth at graphics.stanford.edu  Gates 378   (650) 725-3273 fax 725-6949
> 
> On 5/6/2013 11:19 AM, Carter Bullard wrote:
>> Hey John,
>> OK, so I've implemented " abr " as a new metric, using our normalized equation:
>> 
>>   abr = (sappbytes - dappbytes)/(sappbytes + dappbytes)
>> 
>> This generates values between +1.0 - -1.0.  +1.0 means that all the app bytes
>> were from the source, indicating that the source is a pure PRODUCER, and the
>> destination is a pure CONSUMER.  You see this in FTP PUT file transfers,
>> as an example.  The sign bit reverses this relationship.
>> 
>> -0.0 denotes the special case, when there are no appbytes seen.
>> 
>> In the new argus-clients that I'll put up later today, you can print this out using:
>> 
>>   ra -r argus.data -s +abr
>> 
>> You can also do operations using this metric, such as filter and generate histograms.
>> Here is a run that I did to show how this maybe used in an anomaly detection
>> application.  Here is the simple frequency distribution for all the internal DNS
>> requests made to my local DNS server from a specific client, for all of 2013:
>> 
>> thoth:06 carter$ pwd
>> /Volumes/Data/Archive/QoSient/192.168.0.68/2013
>> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -R . -s mean stddev - udp port domain and src pkts 1 and dst pkts 1
>> N = 1027764  mean = -0.726195  stddev =  0.140532  max = 0.000000  min = -0.909605
>>           median = -0.749129     95% = -0.292517
>>             mode = -0.782609
>> Class      Interval         Freq    Rel.Freq     Cum.Freq          Mean     StdDev 
>>     1   -1.000000e+00     225379    21.9291%     21.9291%     -0.815238   0.000650
>>     2   -8.000000e-01     738148    71.8208%     93.7498%     -0.740887   0.043837
>>     3   -6.000000e-01      10553     1.0268%     94.7766%     -0.534672   0.048717
>>     4   -4.000000e-01      35511     3.4552%     98.2318%     -0.283067   0.021374
>>     5   -2.000000e-01        250     0.0243%     98.2561%     -0.162791   0.000051
>>     6    0.000000e+00      17923     1.7439%    100.0000%      0.000000   0.000000
>>     7    2.000000e-01          0     0.0000%    100.0000%    
>>     8    4.000000e-01          0     0.0000%    100.0000%    
>>     9    6.000000e-01          0     0.0000%    100.0000%    
>>    10    8.000000e-01          0     0.0000%    100.0000%    
>> 
>> 
>> OK, should be very clear, that my host is a net CONSUMER of DNS data, not a net PRODUCER 
>> because the " abr <= 0 ".  The corollary holds true, the local DNS service is a net PRODUCER of
>> data, and not a net CONSUMER of data, from the prospective of this particular end system.
>> So testing filters like this:
>>   ra -r daily.file - abr gt 0 and port domain and src pkts 1 and dst pkts 1
>> 
>> Should reveal flows that deserve a closer look.
>> 
>> OK, there were a lot of flows where the ( abr == 0 ), which was surprising.
>> When DNS experiences a ServFail, the response is the same as the request, just with an error bit
>> set in the DNS header.  QoSient had a big issue in Jan, 2013, when 17923 DNS ServFail failures
>> occurred, so that is where the ( abr == 0 ) flows occured.  Important to know this when evaluating
>> DNS as a channel for CONSUMER to PRODUCER conversion.
>> 
>> But for DNS health and operability, looking for flows where the ( sappbytes == dappbytes ) is
>> also a pretty interesting thing to look for.
>> 
>> Hope this is helpful,
>> 
>> Carter
>> 
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E. 57th Street Suite 12D
>> New York, New York 10022
>> 
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130506/5b87fc42/attachment.bin>