ABRatio question
Carter Bullard
carter at qosient.com
Fri Jun 7 12:00:51 EDT 2013
Hey Craig,
Sorry, but I didn't really address all the questions that you had in your email.
And don't interpret my earlier response and conclude that aggregate ABR's
are not good. No, no, no, no, no….. You would like to be able to take an
entire enterprise's ABR, possibly calculated on a 0.5-1.0 minute basis,
and use that metric to tell you if something is up, and or changing.
But what I was trying to say earlier, is that given the fact that a specific node,
workgroup or specific enterprise, all share the property that they are
both producer and consumer at the same time, the sensitivity of the metric
will come from focusing on specific services, nodes, etc….
As an exercise, you should be able to take a day's worth of argus data,
and for an entire enterprise, calculate the per 5 second ABR for each host.
The best way to calculate this is to generate our IP address table, but
in the time range your interested in. This will give you the starting
data, and then aggregate that data to formulate an enterprise, subnet,
remote network wide value.
Generate per host data, in the time domain we're interested in (5s),
to deal effectively with the source / dest labels:
rabins -M rmon -m saddr -M time 5s -r days.worth.of.data -w /tmp/ip.addrs.out - ip
If you notice, this is my goto database for nodal behavioral classification.
But this is just one of the views that I use for flow data mining.
So, from this, I get all the IP addresses. I can now assemble
any type of network aggregation, to measure for producer / consumer
relationships.
So this is easy:
racluster -r /tmp/ip.addrs.out -m saddr/24 -w - | rasort -m abr -s +abr
This will give you all the /24 CIDR networks, and if they were producers
or consumers. For remote networks, they are generally, single service
sites, i.e. your hosts will generally only talk to them on a single service.
That could be HTTP, SSH, or through VLAN tunnels, but generally they
are single services. Which means you get your service oriented selection
bias, for free.
So here is mine for the day:
StartTime Dur SrcAddr SrcPkts DstPkts ABRatio
2013/05/11.20:50:14.43 0.00 224.0.1.0/24 0 40 -1.00
2013/05/11.22:13:12.08 0.00 255.255.255.0/24 0 769 -1.00
2013/05/11.00:06:58.29 85721.91 239.255.255.0/24 0 1823 -1.00
2013/05/11.00:00:02.60 86292.16 224.0.0.0/24 0 2824 -1.00
2013/05/10.23:59:47.11 86403.78 192.168.0.0/24 1213346 1234569 -0.36
2013/05/11.22:23:29.79 0.04 184.28.152.0/24 4 6 -0.26
2013/05/11.09:14:24.03 129.31 17.154.66.0/24 187 211 -0.01
2013/05/11.00:28:02.95 84578.13 17.151.16.0/24 21 21 0.00
2013/05/11.12:07:58.21 39601.69 96.30.62.0/24 24 48 -0.00
2013/05/11.12:07:58.33 39601.71 128.171.104.0/24 24 48 -0.00
2013/05/11.12:07:58.26 39601.66 69.12.162.0/24 25 49 -0.00
2013/05/11.12:07:58.21 39601.74 208.68.208.0/24 25 49 -0.00
2013/05/11.12:07:58.17 39601.66 131.247.254.0/24 26 50 -0.00
2013/05/11.12:07:59.04 39601.02 69.162.87.0/24 26 50 -0.00
2013/05/11.12:07:58.20 39601.74 204.45.83.0/24 25 51 -0.00
2013/05/11.12:07:58.88 39600.92 103.18.207.0/24 27 51 -0.00
2013/05/11.12:07:58.31 39601.67 140.211.166.0/24 27 55 -0.00
2013/05/11.12:07:58.22 39601.66 67.20.126.0/24 50 98 -0.00
2013/05/11.01:40:07.70 19.82 17.254.2.0/24 40 47 0.01
2013/05/11.00:03:09.82 85953.34 17.172.232.0/24 393 457 0.02
2013/05/11.00:03:11.65 86106.31 17.172.34.0/24 5195 5371 0.22
2013/05/11.09:14:11.57 1009.40 17.173.66.0/24 103 114 0.41
2013/05/11.00:02:58.91 86117.97 17.172.208.0/24 10473 11830 0.54
2013/05/11.09:23:16.48 52.19 207.241.224.0/24 20 24 0.55
2013/05/11.09:14:11.39 103.45 184.85.18.0/24 216 221 0.57
2013/05/11.12:23:24.60 1.65 23.66.178.0/24 11 14 0.66
2013/05/11.12:07:57.99 0.11 199.187.126.0/24 15 18 0.66
2013/05/11.16:30:24.68 2.93 17.167.136.0/24 16 19 0.68
2013/05/11.16:30:27.56 21199.33 17.167.137.0/24 32 36 0.74
2013/05/11.00:01:16.02 86280.72 66.39.3.0/24 13891 18844 0.77
2013/05/11.18:23:27.87 0.03 184.85.29.0/24 5 6 0.78
2013/05/11.09:12:44.11 29526.04 208.16.159.0/24 24 24 0.79
2013/05/11.12:07:58.19 39603.14 128.175.60.0/24 66 102 0.79
2013/05/11.12:07:58.43 39601.36 69.39.81.0/24 43 73 0.80
2013/05/11.09:23:15.68 841.53 74.207.242.0/24 20 26 0.80
2013/05/11.12:07:58.22 39601.63 209.118.59.0/24 34 60 0.80
2013/05/11.00:07:55.83 39603.29 128.122.215.0/24 60 72 0.81
2013/05/11.09:14:11.39 1.01 199.7.48.0/24 48 42 0.81
2013/05/11.01:40:09.05 55850.70 23.6.133.0/24 35 35 0.81
2013/05/11.18:23:11.32 16.04 199.7.59.0/24 23 20 0.81
2013/05/11.02:00:05.04 26106.43 184.85.19.0/24 197 226 0.82
2013/05/11.00:11:11.82 84046.64 173.194.43.0/24 1641 1768 0.84
2013/05/11.18:23:11.90 28.75 8.27.149.0/24 1311 1156 0.85
2013/05/11.09:15:06.37 546.56 17.152.16.0/24 35777 29429 0.85
2013/05/11.12:07:58.17 39601.66 146.137.96.0/24 58 127 0.86
2013/05/11.12:07:59.01 39601.24 198.82.184.0/24 74 97 0.87
2013/05/11.12:07:59.06 39600.78 208.100.4.0/24 51 75 0.87
2013/05/11.00:07:56.40 39601.64 129.98.1.0/24 72 72 0.87
2013/05/11.12:07:58.16 39602.00 69.9.32.0/24 84 108 0.87
2013/05/11.00:07:56.16 39603.04 134.173.34.0/24 60 72 0.88
2013/05/11.09:14:25.41 37269.79 17.154.65.0/24 1738 1557 0.93
2013/05/11.01:47:54.40 73517.68 0.0.0.0/24 4 2 0.95
2013/05/11.18:23:10.83 59.88 17.164.1.0/24 91 72 0.96
2013/05/11.18:23:27.66 12.98 8.26.202.0/24 81 57 0.99
2013/05/11.09:12:53.16 0.61 69.58.181.0/24 207 164 1.00
2013/05/11.09:14:11.10 613.48 208.59.201.0/24 97413 70773 1.00
2013/05/11.09:23:17.28 51.40 207.241.227.0/24 1958 1395 1.00
Now, I want the remote networks I talk to, to be producers, and I want
my network to be a consumer, but I do have to learn the relationships.
For the DNS traffic between two networks, your going to get around 0.0 for
the ABR, as I mentioned, control plane traffic tends to be balanced. So
I'm going to be interested really in remote networks that have their
ABR < 0.0. So for Sat, May 11, 2013, the only real host of interest is
this guy:
StartTime Dur SrcAddr SrcPkts DstPkts ABRatio
2013/05/11.22:23:29.79 0.04 184.28.152.0/24 4 6 -0.26
So, I go and grab the primitive data for that remote network, to see what
was going on:
thoth:tmp carter$ ra -r /tmp/alarm.out -s stime dur suser:32 duser:32
StartTime Dur srcUdata dstUdata
2013/05/11.22:23:29.797440 0.041634 s[32]=GET /configurations/macosx/xprot d[32]=HTTP/1.1 304 Not Modified..Conte
OK, not a problem, but just in case, I go and look at every daily occurrence of
this address for the last year, from my border sensor, to see what is up:
thoth:tmp carter$ rasql -t -365d+365d -M time 1d -r mysql://root@localhost/ratop/etherHost_%Y_%m_%d \
-M sql="saddr='184.28.152.224'" -s stime dur saddr pkts bytes abr - srcid 192.168.0.1
StartTime Dur SrcAddr TotPkts TotBytes ABRatio
2012/10/11.18:43:20.240178 0.242364 184.28.152.224 54 44454 0.977685
2012/10/12.00:14:47.370741 66517.203* 184.28.152.224 452 246560 0.868444
2012/10/13.13:02:04.107234 6325.0288* 184.28.152.224 81 51262 0.936933
2012/10/14.00:04:10.044048 79592.210* 184.28.152.224 164 105840 0.939111
2012/10/15.07:42:37.328684 270.697784 184.28.152.224 25 5939 0.484657
2012/10/16.13:02:09.922434 0.401774 184.28.152.224 55 45254 0.981011
2012/10/17.00:03:28.057789 67217.703* 184.28.152.224 163 102044 0.934272
2012/10/18.00:17:09.578972 66400.179* 184.28.152.224 164 102110 0.934272
2012/10/19.00:25:45.184674 65888.812* 184.28.152.224 178 123825 0.913400
2012/10/20.00:25:49.868630 77807.718* 184.28.152.224 192 110938 0.917838
2012/10/21.00:17:20.390333 66402.062* 184.28.152.224 140 96190 0.955099
2012/10/22.05:08:33.115062 50827.363* 184.28.152.224 178 109232 0.912240
2012/10/23.13:02:23.544898 33950.242* 184.28.152.224 135 95713 0.955017
2012/10/24.00:15:18.276884 53905.097* 184.28.152.224 180 63579 0.903839
2012/10/25.00:26:09.864482 82408.500* 184.28.152.224 187 104598 0.929145
2012/10/26.00:15:05.384850 78453.585* 184.28.152.224 287 182745 0.906283
2012/10/27.00:10:09.021225 66858.906* 184.28.152.224 164 102109 0.934271
2012/10/28.00:26:35.640783 65876.476* 184.28.152.224 139 96124 0.955099
2012/10/29.00:14:34.397775 46081.527* 184.28.152.224 111 57721 0.899033
2013/02/05.22:17:47.228398 3847.0629* 184.28.152.224 94 52962 0.907157
2013/02/06.22:17:51.519308 62712.769* 184.28.152.224 112 92433 0.979977
2013/02/07.22:17:55.552219 62710.781* 184.28.152.224 114 92565 0.979977
2013/02/08.22:17:59.827679 62707.917* 184.28.152.224 20 2732 -0.240525
2013/02/09.22:18:04.078163 0.037729 184.28.152.224 10 1401 -0.264910
2013/02/10.22:00:09.946186 41502.117* 184.28.152.224 187 62862 0.667249
2013/02/11.22:18:12.340802 62745.562* 184.28.152.224 125 30805 0.419671
2013/02/12.22:18:16.655185 62699.433* 184.28.152.224 141 99955 0.955160
2013/02/13.22:18:20.784155 63222.882* 184.28.152.224 103 23930 0.430099
2013/02/14.22:18:25.004362 70561.492* 184.28.152.224 374 44991 0.460870
2013/02/16.00:00:02.751673 18429.470* 184.28.152.224 71 20437 0.547439
2013/02/16.22:18:33.603872 62690.175* 184.28.152.224 95 54956 0.914274
2013/02/17.19:02:30.534623 81958.093* 184.28.152.224 312 150981 0.794266
2013/02/18.21:58:55.035988 63872.742* 184.28.152.224 154 99963 0.965994
2013/02/20.01:17:31.527337 51958.082* 184.28.152.224 72 50804 0.962836
2013/02/20.22:18:50.418635 62681.203* 184.28.152.224 119 94209 0.980260
2013/02/21.22:18:54.561408 62679.027* 184.28.152.224 149 101350 0.961201
2013/02/22.20:24:58.236513 69517.335* 184.28.152.224 321 188075 0.857267
2013/02/23.19:50:51.316463 71566.117* 184.28.152.224 173 74984 0.804917
2013/02/24.22:19:07.183847 62673.371* 184.28.152.224 216 151249 0.936410
2013/02/25.22:19:11.588784 65683.906* 184.28.152.224 223 119990 0.856182
2013/02/26.22:19:15.757411 62667.339* 184.28.152.224 149 68149 0.820918
2013/02/27.22:19:20.027904 40147.496* 184.28.152.224 82 19134 0.337832
2013/02/28.21:17:21.923772 70119.890* 184.28.152.224 234 118834 0.886054
2013/03/01.21:26:13.950830 65855.054* 184.28.152.224 123 60140 0.792862
2013/03/02.19:54:10.677039 76746.335* 184.28.152.224 203 96144 0.816420
2013/03/03.19:30:10.975939 72821.921* 184.28.152.224 185 52113 0.528408
2013/03/04.19:34:27.899775 79636.015* 184.28.152.224 250 134511 0.883100
2013/03/05.19:39:24.781146 3599.7658* 184.28.152.224 133 79397 0.895016
2013/04/15.17:08:15.113251 2445.6494* 184.28.152.224 84 57344 0.862126
2013/04/15.23:10:32.323638 69079.882* 184.28.152.224 295 200875 0.889556
2013/04/17.09:04:42.604515 37430.957* 184.28.152.224 300 166025 0.880096
2013/04/18.12:29:50.089838 24143.085* 184.28.152.224 225 138917 0.938160
2013/04/18.21:45:35.726030 72212.773* 184.28.152.224 159 74735 0.762863
2013/04/20.01:46:32.764015 57757.781* 184.28.152.224 82 53025 0.944862
2013/04/21.04:34:32.026012 47680.445* 184.28.152.224 83 53089 0.944859
2013/04/21.23:22:58.145742 66376.351* 184.28.152.224 256 107669 0.794811
2013/04/22.20:15:23.480801 77632.976* 184.28.152.224 230 115691 0.822634
2013/04/24.01:10:45.663979 59912.757* 184.28.152.224 142 90444 0.921405
2013/04/25.12:30:19.899730 24862.531* 184.28.152.224 91 56464 0.917516
2013/04/25.21:56:55.604930 75122.460* 184.28.152.224 374 195931 0.897320
2013/04/27.01:43:26.839082 57957.796* 184.28.152.224 138 101447 0.971495
2013/04/27.20:15:23.050780 77643.679* 184.28.152.224 165 110760 0.949440
2013/04/28.23:53:21.006963 52321.429* 184.28.152.224 134 71072 0.812474
2013/05/01.01:31:20.539316 58692.230* 184.28.152.224 183 135440 0.955751
2013/05/01.20:05:41.333809 59108.316* 184.28.152.224 121 65726 0.883075
2013/05/09.22:23:21.478210 0.108438 184.28.152.224 62 48721 0.979440
2013/05/10.15:25:08.644817 29468.128* 184.28.152.224 216 150011 0.868032
2013/05/11.17:49:51.991014 16418.003* 184.28.152.224 20 2741 -0.232440
2013/05/12.17:49:53.975577 0.071207 184.28.152.224 10 1340 -0.196970
2013/05/13.17:49:56.045838 16422.525* 184.28.152.224 82 33046 0.846338
2013/05/14.07:53:39.564801 52203.406* 184.28.152.224 94 43212 0.705936
2013/05/15.20:36:14.703907 4502.7358* 184.28.152.224 117 42100 0.729727
2013/05/16.12:24:45.843919 8.403816 184.28.152.224 21 2601 0.038298
2013/05/17.00:40:39.466776 61764.933* 184.28.152.224 371 241153 0.911680
2013/05/18.00:57:48.589020 41218.335* 184.28.152.224 81 52969 0.961294
2013/05/19.04:33:45.141900 28266.085* 184.28.152.224 73 52188 0.961293
2013/05/20.01:13:40.877454 79498.625* 184.28.152.224 137 73048 0.816588
2013/05/21.01:16:40.899938 59611.750* 184.28.152.224 918 194120 0.853857
2013/05/22.01:04:02.659273 60372.046* 184.28.152.224 226 141120 0.949995
2013/05/23.00:57:38.056896 60758.589* 184.28.152.224 73 52921 0.964422
2013/05/24.01:19:48.266635 59430.265* 184.28.152.224 131 101625 0.971764
2013/05/25.00:22:39.300505 83129.835* 184.28.152.224 186 118423 0.929274
2013/05/26.03:58:40.499295 49902.015* 184.28.152.224 160 109467 0.949271
2013/05/27.04:32:15.570498 47889.023* 184.28.152.224 133 101698 0.971764
2013/05/28.12:25:29.473000 36589.074* 184.28.152.224 435 284734 0.914443
2013/05/29.00:47:52.608999 61984.292* 184.28.152.224 221 146360 0.939423
2013/05/30.01:07:02.554303 77914.609* 184.28.152.224 251 168966 0.895431
2013/05/31.01:21:11.778745 77470.250* 184.28.152.224 123 65423 0.921447
2013/06/01.00:48:04.384857 61350.234* 184.28.152.224 130 101500 0.971764
2013/06/02.04:32:40.316898 47876.382* 184.28.152.224 135 101829 0.971764
2013/06/03.01:14:25.206337 59776.417* 184.28.152.224 196 108680 0.961000
2013/06/04.00:29:00.783472 77764.062* 184.28.152.224 8637 1035195 0.621063
2013/06/05.00:23:25.419316 84839.421* 184.28.152.224 2325 297403 0.754082
2013/06/06.00:07:34.665736 82343.617* 184.28.152.224 2055 327703 0.742971
2013/06/07.01:33:11.982544 30247.156* 184.28.152.224 71 52091 0.961292
As you can see, this node, which has been the target for this type of query
since Oct, 2012, is predominately a producer, but for two back to back days
(Fri/Sat) in Feb, 2013, and two days in May (Sat/Sun) WE were producing data
rather than consuming, from this AKAMAI host.
In the final analysis, we find that these periodic occurrences are the result
of the remote web server changing its response, to the same question. The
response got shorter, shifting the ABR to < 0.0. I say, in this case, no
harm no foul. I would reject this based on size and duration of the transaction,
but if you filter for small short connections, then there is an avenue for
undetectable leakage, so I wouldn't do that.
So that is how I use the ABR, as a general rule. If one of my hosts ends up in
a peer-to-peer network, where data is getting sucked out of my infrastructure,
I will end up with at least 1 remote network that is a consumer of data. If
the data is stripped across a lot of remote nodes, then I'll get multiple
remote networks that are consumers, more than likely they will be new.
If its really cleaver, I'll see a familiar remote network shift toward negative.
OK, with regard to your question about averaging out the ABR…...
The formula for the metric has the volume of traffic in the equation
(sappbytes - dappbytes) / (sappbytes + dappbytes)
when we aggregate records, and calculate the abr, we don't average
the abr's, we sum the appbyte metrics, and recalculate:
(sum(sappbytes) - sum(dappbytes)) / (sum(sappbytes) + sum(dappbytes))
so in your example the little consumers contribution to the total
abr is weighted by the amount of traffic they send and received.
Hope this is helpful, keep those questions coming !!!!!
Carter
On Jun 6, 2013, at 5:11 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> Carter,
>
> I’ve been playing around a bit with the ABRatio field. In trying to figure out the best way to identify if a host is a consumer or producer across all flows. If a host has a single high-volume flow where it is a producer and a bunch of small flows where it is a consumer, averaging the ABRatio would seem to skew the results towards it being a consumer.
>
> Would it be more accurate to multiply the ABRatio by either the volume of bytes in the flow or the duration of the flow prior to averaging a host’s ABRatio? And if so, do you think it’s better to look at that ratio over the volume of data or over time?
>
> Thx.
>
> C
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130607/ce956ff8/attachment.bin>
More information about the argus
mailing list