rabins and racluster.conf
Carter Bullard
carter at qosient.com
Tue Apr 24 03:13:39 EDT 2012
Hey Mark,
So there is sample data available on the web site to test these types of issues.
http://qosient.com/argus/data
If you grab the argus.simple.data.out and do a few tests, you'll see, like I did,
that your example demonstrates a bug in rabins(), I very much hate to say.
Using your c3.conf as racluster.conf in these examples, here you go:
thoth:data carter$ racount -r argus.simple.data.out
racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
sum 10 101 48 53 51942 6494 45448
thoth:data carter$ racluster -f racluster.conf -r argus.simple.data.out -w - | racount
racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
sum 4 101 48 53 51942 6494 45448
thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out -w - | racount
racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
sum 5 85 38 47 50823 5922 44901
So, rabins isn't generating correct data in this case. Not good. More detailed output is:
thoth:data carter$ ra -r argus.simple.data.out
StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
2012/02/13.18:03:53.527324 0.000000 192.168.0.68 man 0. 0 0. 0 0 0 0 0 STA
2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
thoth:data carter$ racluster -r argus.simple.data.out
StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
rabins -M time 1h -r argus.simple.data.out
StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
So, without a racluster.conf file, all the programs do the right thing.
With your racluster configuration, we get this using racluster.1, which is correct behavior:
racluster -f racluster.conf -M time 1h -r argus.simple.data.out
StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
2012/02/13.17:48:36.592155 27.203648 192.168.0.68 e g tcp 192.168.0.68.* -> 128.0.0.0/1.http 38 47 5922 44901 FIN
2012/02/13.17:48:36.589413 213.552917 192.168.0.68 M arp 0.0.0.0 who 0.0.0.0 8 4 424 256 CON
2012/02/13.17:48:36.589949 0.000608 192.168.0.68 e udp 0.0.0.0.* <-> 0.0.0.0.* 2 2 148 291 CON
However, rabins, when run with the "-M time 1h" option, should generate the exact same output as racluster (should have only one bin) but:
thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out
StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
So something happened to the arp and udp traffic. I'll have to fix this, hopefully by Thurs :O(
Carter
On Apr 23, 2012, at 5:03 PM, Mark E. Mallett wrote:
> On Mon, Apr 23, 2012 at 03:40:52PM -0400, Carter Bullard wrote:
>> Hey Mark,
>>
>> Yes, the "-f " is for a racluster.conf file. It should aggregate
>> within the bin, so its as if you ran racluster on each bin as the data
>> arrives.
>>
>> Sorry about the man page, I'll take a look later today. Any updates
>> will go on the web server. You're using 3.0.6 code?
>
> yes, 3.0.6, on various systems.
>
> And fantastic, that's what I hoped. Let me describe my simple tests
> and perhaps somebody can tell me what dumb thing I am doing.
>
> It looked like the racluster.conf aggregation rules could be pretty
> useful so I thought I'd try it out with some simple reports where the
> details aren't too important but the method is. I figured if the method
> works I can go onwards from there. But I got stuck early on.
>
> The simple test is to get some per-hour totals broken down into
> potentially high-interest categories in different ways. I chose http
> and https and ssh and "everything else" as categories for the test.
> Here's my file c3.conf
>
> =======
> # I've tried this on or off
> RACLUSTER_PRESERVE_FIELDS=no
> #
> filter="tcp and dst port eq 80" model="proto dport"
> filter="tcp and dst port eq 443" model="proto dport"
> filter="tcp and dst port eq 22" model="proto dport"
> # filter="tcp or udp" model="proto"
> filter="" model="proto"
> =========
>
> I processed this with rabins doing time binning (see command below),
> expecting this to give me one record of aggregated data for http, one
> for https, one for ssh per time bin, and then other records in each bin
> aggregated only by protocol.
>
> When I run rabins I either write the argus records to a file and look at
> that with some other ra* utility or just look at the output directly;
> the result seems to be the same. This is the sort of command I use to
> time bin and look at the output:
>
> rabins -n -f c3.conf -M zero -M time 1h -r input-file \
> -s stime proto dport pkts bytes dur:12 | less
>
> With that command, I do not seem to get any aggregation at all. I get a
> lot of individual flow records matching the tcp statements (http, https,
> ssh), and no records for any other protocol or ports. This last part
> surprises me most because I thought the c3.conf file will catch
> everything, even if the aggregation doesn't happen.
>
> If I add a "model" set on the command line, i.e. an '-m' switch like this:
>
> rabins -n -f c3.conf -M zero -M time 1h -m proto dport -r input-file \
> -s stime proto dport pkts bytes dur:12 | less
>
> then I get the per-bin aggregation I expected of the http, https, and ssh
> traffic, but I still get no other output (except what's generated by the
> "-M zero" for otherwise empty bins). Here I would still expect the last
> line in c3.conf to catch some things and produce things aggregated
> by "proto".
>
> If I uncomment the next-to-last filter line, then I do see some other
> records show up; but for the most part they also appear unaggregated
> except by flow.
>
> I could go on for a while but this is already long enough and probably
> quite enough to introduce my confusion. One of the issues seems to be
> that the "model" section of each line isn't being applied, and the only
> model that gets applied is the one in '-m' on the command line. But I
> don't think that's the only issue. There's probably some dots that I
> haven't connected..
>
> Yours,
> -mm-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120424/9470382b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120424/9470382b/attachment.bin>
More information about the argus
mailing list