normalized appbyte ratio for producer/consumer relationship
Carter Bullard
carter at qosient.com
Tue May 7 23:31:59 EDT 2013
Hey John,
Here is a run to try out the abr metric for basic producer/consumer anomaly detection.
Picked a bunch of hosts that are involved in a service that runs on a specific port.
There is a data transfer function, and there is a command and control function.
The abr metric picks out the producer and consumers for the data transfer, and it points to those that are involved in command and control, here you go.
thoth:backups root# racluster -M rmon -m saddr -r monthly.data -w - | \
rasort -m abr -s stime dur:16 proto saddr spkts:12 dpkts:12 abr
StartTime Dur Proto SrcAddr SrcPkts DstPkts ABRatio
2013/02/05.14:03:48.304265 2022912.500000 tcp 192.168.1.31 118813516 85735688 0.999937
2013/02/06.16:19:37.099642 1895160.500000 tcp 192.168.2.75 3621 1899 0.997145
2013/02/07.12:06:09.973606 27554.181641 tcp 192.168.2.34 732 650 0.915472
2013/02/27.11:40:47.093087 4.957941 tcp 192.168.4.35 13 12 0.551477
2013/02/18.14:17:01.287529 758586.750000 tcp 192.168.7.166 23 18 0.422487
2013/02/06.10:10:26.473381 30.864573 tcp 192.168.12.149 7 5 0.295455
2013/02/13.16:45:55.793151 11.732343 tcp 192.168.7.155 15 13 0.250169
2013/02/06.09:31:55.244962 1228359.500000 tcp 192.168.2.54 32 35 0.241102
2013/02/08.10:09:14.794703 1727145.250000 tcp 192.168.0.125 21 18 0.213155
2013/02/06.13:17:45.550931 1819270.875000 tcp 192.168.1.138 1227 1231 0.135222
2013/02/15.11:28:36.104691 50.191151 tcp 192.168.2.71 75 59 0.100396
2013/02/07.14:00:35.770555 83.157898 tcp 192.168.0.70 894 890 0.057422
2013/02/12.14:19:09.720183 1.038289 tcp 192.168.1.125 7 8 0.029588
2013/02/21.03:07:04.628043 32.868526 tcp 192.168.0.72 7 5 0.023810
2013/02/11.08:39:44.024865 1478973.500000 tcp 192.168.2.45 268 157 -0.015058
2013/02/19.14:25:03.376258 28.092548 tcp 192.168.7.153 7 5 -0.043860
2013/02/06.14:05:03.101059 65.313805 tcp 192.168.2.58 6 7 -0.055469
2013/02/06.13:17:45.550931 1772498.375000 tcp 192.168.2.57 13 16 -0.164201
2013/02/06.12:20:05.543201 1913705.250000 tcp 192.168.2.138 256 317 -0.173220
2013/02/20.12:43:35.083127 424459.562500 tcp 192.168.2.102 1147 1160 -0.343431
2013/02/20.13:46:25.772704 940.071899 tcp 192.168.12.2 16 17 -0.392946
2013/02/07.12:55:49.369160 1822594.375000 tcp 192.168.1.110 296 400 -0.456629
2013/02/06.12:51:40.056876 1901223.250000 tcp 192.168.1.123 165 222 -0.601214
2013/02/06.09:37:40.607721 1900794.750000 tcp 192.168.1.111 681 995 -0.612489
2013/02/05.14:44:23.517669 1818851.000000 tcp 192.168.1.117 79 102 -0.627868
2013/02/05.14:05:30.070586 1889467.750000 tcp 192.168.1.127 143 175 -0.669750
2013/02/06.09:31:55.244962 697157.125000 tcp 192.168.1.115 51 56 -0.683094
2013/02/13.16:07:23.915251 1291093.750000 tcp 192.168.1.130 61 75 -0.694824
2013/02/07.13:33:41.019318 1557935.000000 tcp 192.168.1.113 100 133 -0.706807
2013/02/06.12:06:43.001669 1721456.875000 tcp 192.168.1.112 618 1010 -0.740505
2013/02/23.17:46:07.768913 0.588042 tcp 192.168.9.84 12 15 -0.758408
2013/02/06.11:31:22.894660 1906436.625000 tcp 192.168.1.122 76 101 -0.762083
2013/02/05.15:07:18.072928 1977283.125000 tcp 192.168.1.114 90 120 -0.781786
2013/02/08.09:07:55.007936 1140006.125000 tcp 192.168.1.118 112 136 -0.787711
2013/02/05.14:04:02.548134 2022898.250000 tcp 192.168.2.47 327011 243367 -0.795821
2013/02/06.14:30:10.891759 1905899.875000 tcp 192.168.1.116 343 588 -0.934336
2013/02/07.12:06:09.973606 1807785.625000 tcp 192.168.1.121 2579 4398 -0.990442
2013/02/05.14:03:48.304265 1465134.125000 tcp 192.168.2.29 85407678 118569168 -0.999999
So we take a some records, in this case a complete month's worth of traffic involved in a specific application,
involving a specific subnet. We want to know what hosts are producers and consumers for this app.
We need to get the bi-directional flow data into a single object statistic, so we'll aggregate the data for RMON
data processing (one object, in and out stats), and merge for the " saddr ", then just rasort() on the abr field.
We get a list from Producers to Consumers, and the guys in the middle where the abr approaches 0, and we have
balanced communications, we see the complete spectrum of data push agents (producers) where ( ABRation > 0.75 )
on top, and we have the pure data sinks, where the ( ABRatio < -0.75 ), and we've got maybe command
and control in the ( -0.5 < ABRatio < 0.5 ) range ? Probably need to add a threshold for the amount of
data sent and received, to weed out the announcers in the command and control network...
I'd go for that set of rules for this specific application, in this observation domain…..
Carter
On May 6, 2013, at 6:34 PM, John Gerth <gerth at graphics.stanford.edu> wrote:
> Yes, well an abr of 0.0, even without using -0.0 can have multiple meanings
> since you will get 0 as long as s=d even if they are large. The use of -0.0
> is perhaps a bit cute, and, as you point out, since IEEE 754 requires 0.0 == -0.0
> in all relational tests, one has to use signbit() to disambiguate in C.
>
> Even so, I think your example shows that abr has good potential as a metric.
>
> John Gerth gerth at graphics.stanford.edu Gates 378
>
> On 5/6/2013 2:09 PM, Carter Bullard wrote:
>> Hey John,
>> OK, so there is a problem, in that IEEE floating point has ( -0.0 == 0.0 )
>> by definition. So, I fixed it in my compiler, but others are going to have
>> issues when needing to discriminate between 0.0 and -0.0.
>>
>> So, with the new argus-clients, you can filter for any value for abr, using
>> any of the ra* programs. Still have to work on graphing abr, but I'll get
>> to that later tonight.
>>
>> Here is the abr behavior for every DNS request here at QoSient World HQ
>> for 2013, that weren't ServFail errors:
>>
>> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -r argus*domain* -s mean stddev - src pkts 1 and dst pkts 1 and not abr 0.0
>> N = 1009841 mean = -0.739084 stddev = 0.102829 max = -0.162791 min = -0.909605
>> median = -0.749129 95% = -0.653846
>> mode = -0.782609
>> Class Interval Freq Rel.Freq Cum.Freq Mean StdDev
>> 1 -1.000000e+00 225379 22.3183% 22.3183% -0.815238 0.000650
>> 2 -8.000000e-01 738148 73.0955% 95.4137% -0.740887 0.043837
>> 3 -6.000000e-01 10553 1.0450% 96.4588% -0.534672 0.048717
>> 4 -4.000000e-01 35511 3.5165% 99.9752% -0.283067 0.021374
>> 5 -2.000000e-01 250 0.0248% 100.0000% -0.162791 0.000051
>> 6 0.000000e+00 0 0.0000% 100.0000%
>> 7 2.000000e-01 0 0.0000% 100.0000%
>> 8 4.000000e-01 0 0.0000% 100.0000%
>> 9 6.000000e-01 0 0.0000% 100.0000%
>> 10 8.000000e-01 0 0.0000% 100.0000%
>>
>>
>> So, anything positive would be a behavioral anomaly, from the perspective of this host.
>>
>> ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr gt 0.0
>>
>> and this would pick out candidates for DNS server availability errors:
>>
>> ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr eq 0.0
>>
>> Of course, you can do an analysis of every service, and get a rather interesting set of
>> " what is normal " behaviors, using this simple type of metric. For those services where
>> the abr is always positive or always negative, seeing a shift to the other side, can
>> indicate events that should be of interest.
>>
>> Hope this is helpful,
>>
>> Carter
>>
>>
>> On May 6, 2013, at 2:41 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>
>>> Hey John,
>>> If no appbytes, currently we return -0.0, but the library knows if there are
>>> appbytes or not, so we can return nada, when printing out the values.
>>> Right now, when using xml format, you won't get a value.
>>>
>>> Having problems getting my compiler to tell the difference between 0.0 and -0.0,
>>> but should hopefully have this working by this afternoon.
>>>
>>> Carter
>>>
>>>
>>> On May 6, 2013, at 2:37 PM, John Gerth <gerth at graphics.stanford.edu <mailto:gerth at graphics.stanford.edu>> wrote:
>>>
>>>> Nice example. I'm looking forward to using this.
>>>>
>>>> As your example shows, this metric is available for any existing argus
>>>> files that were created containing appbyte values. I'm assuming that if
>>>> the sensor wasn't configured to capture those, 'abr' is not available.
>>>>
>>>>
>>>> --
>>>> John Gerth gerth at graphics.stanford.edu <mailto:gerth at graphics.stanford.edu> Gates 378 (650) 725-3273 fax 725-6949
>>>>
>>>> On 5/6/2013 11:19 AM, Carter Bullard wrote:
>>>>> Hey John,
>>>>> OK, so I've implemented " abr " as a new metric, using our normalized equation:
>>>>>
>>>>> abr = (sappbytes - dappbytes)/(sappbytes + dappbytes)
>>>>>
>>>>> This generates values between +1.0 - -1.0. +1.0 means that all the app bytes
>>>>> were from the source, indicating that the source is a pure PRODUCER, and the
>>>>> destination is a pure CONSUMER. You see this in FTP PUT file transfers,
>>>>> as an example. The sign bit reverses this relationship.
>>>>>
>>>>> -0.0 denotes the special case, when there are no appbytes seen.
>>>>>
>>>>> In the new argus-clients that I'll put up later today, you can print this out using:
>>>>>
>>>>> ra -r argus.data -s +abr
>>>>>
>>>>> You can also do operations using this metric, such as filter and generate histograms.
>>>>> Here is a run that I did to show how this maybe used in an anomaly detection
>>>>> application. Here is the simple frequency distribution for all the internal DNS
>>>>> requests made to my local DNS server from a specific client, for all of 2013:
>>>>>
>>>>> thoth:06 carter$ pwd
>>>>> /Volumes/Data/Archive/QoSient/192.168.0.68/2013
>>>>> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -R . -s mean stddev - udp port domain and src pkts 1 and dst pkts 1
>>>>> N = 1027764 mean = -0.726195 stddev = 0.140532 max = 0.000000 min = -0.909605
>>>>> median = -0.749129 95% = -0.292517
>>>>> mode = -0.782609
>>>>> Class Interval Freq Rel.Freq Cum.Freq Mean StdDev
>>>>> 1 -1.000000e+00 225379 21.9291% 21.9291% -0.815238 0.000650
>>>>> 2 -8.000000e-01 738148 71.8208% 93.7498% -0.740887 0.043837
>>>>> 3 -6.000000e-01 10553 1.0268% 94.7766% -0.534672 0.048717
>>>>> 4 -4.000000e-01 35511 3.4552% 98.2318% -0.283067 0.021374
>>>>> 5 -2.000000e-01 250 0.0243% 98.2561% -0.162791 0.000051
>>>>> 6 0.000000e+00 17923 1.7439% 100.0000% 0.000000 0.000000
>>>>> 7 2.000000e-01 0 0.0000% 100.0000%
>>>>> 8 4.000000e-01 0 0.0000% 100.0000%
>>>>> 9 6.000000e-01 0 0.0000% 100.0000%
>>>>> 10 8.000000e-01 0 0.0000% 100.0000%
>>>>>
>>>>>
>>>>> OK, should be very clear, that my host is a net CONSUMER of DNS data, not a net PRODUCER
>>>>> because the " abr <= 0 ". The corollary holds true, the local DNS service is a net PRODUCER of
>>>>> data, and not a net CONSUMER of data, from the prospective of this particular end system.
>>>>> So testing filters like this:
>>>>> ra -r daily.file - abr gt 0 and port domain and src pkts 1 and dst pkts 1
>>>>>
>>>>> Should reveal flows that deserve a closer look.
>>>>>
>>>>> OK, there were a lot of flows where the ( abr == 0 ), which was surprising.
>>>>> When DNS experiences a ServFail, the response is the same as the request, just with an error bit
>>>>> set in the DNS header. QoSient had a big issue in Jan, 2013, when 17923 DNS ServFail failures
>>>>> occurred, so that is where the ( abr == 0 ) flows occured. Important to know this when evaluating
>>>>> DNS as a channel for CONSUMER to PRODUCER conversion.
>>>>>
>>>>> But for DNS health and operability, looking for flows where the ( sappbytes == dappbytes ) is
>>>>> also a pretty interesting thing to look for.
>>>>>
>>>>> Hope this is helpful,
>>>>>
>>>>> Carter
>>>>>
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E. 57th Street Suite 12D
>>>>> New York, New York 10022
>>>>>
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130507/d4ddb6f6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130507/d4ddb6f6/attachment.bin>
More information about the argus
mailing list