rabins and racluster.conf
Carter Bullard
carter at qosient.com
Mon Apr 30 09:38:53 EDT 2012
Hey Mark,
Sorry for the delayed response !!!!! OK, here are a few patches that should fix the problem.
Looked to be a "cut, copy and no paste" style of omission in the rabins.c source.
The patch to ./common/argus_client.c is not related to your problem, but can be included,
especially if you've gotten any filter error messages when running rabins with the cluster.conf
approach.
Please give these a test, and if all is well, I'll put out the first round of argus-client-3.0.6 fixes !!!
I've included a new copy of rabins.c if you aren't comfortable with patch files.
Hope all is most excellent,
Carter
==== //depot/argus/clients/clients/rabins.c#70 - /Volumes/Users/carter/argus/clients/clients/rabins.c ====
453a454,458
> if (parser->ArgusAggregatorFile != NULL)
> free(parser->ArgusAggregatorFile);
>
> parser->ArgusAggregatorFile = strdup(parser->ArgusFlowModelFile);
>
==== //depot/argus/clients/common/argus_client.c#268 - /Volumes/Users/carter/argus/clients/common/argus_client.c ====
11620,11621c11620,11621
< if (ArgusFilterCompile (&agg->filter, agg->filterstr, ArgusParser->Oflag) < 0)
< ArgusLog (LOG_ERR, "ArgusNewAggregator ArgusFilterCompile returned error");
---
> // if (ArgusFilterCompile (&agg->filter, agg->filterstr, ArgusParser->Oflag) < 0)
> // ArgusLog (LOG_ERR, "ArgusNewAggregator ArgusFilterCompile returned error");
On Apr 24, 2012, at 3:13 AM, Carter Bullard wrote:
> Hey Mark,
> So there is sample data available on the web site to test these types of issues.
>
> http://qosient.com/argus/data
>
> If you grab the argus.simple.data.out and do a few tests, you'll see, like I did,
> that your example demonstrates a bug in rabins(), I very much hate to say.
> Using your c3.conf as racluster.conf in these examples, here you go:
>
> thoth:data carter$ racount -r argus.simple.data.out
> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
> sum 10 101 48 53 51942 6494 45448
>
> thoth:data carter$ racluster -f racluster.conf -r argus.simple.data.out -w - | racount
> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
> sum 4 101 48 53 51942 6494 45448
>
> thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out -w - | racount
> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
> sum 5 85 38 47 50823 5922 44901
>
>
> So, rabins isn't generating correct data in this case. Not good. More detailed output is:
>
> thoth:data carter$ ra -r argus.simple.data.out
> StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
> 2012/02/13.18:03:53.527324 0.000000 192.168.0.68 man 0. 0 0. 0 0 0 0 0 STA
> 2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
> 2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
> 2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
> 2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
> 2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
> 2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
> 2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
> 2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
> 2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
>
> thoth:data carter$ racluster -r argus.simple.data.out
> StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
> 2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
> 2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
> 2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
> 2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
> 2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
> 2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
> 2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
> 2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
> 2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
>
> rabins -M time 1h -r argus.simple.data.out
> StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
> 2012/02/13.17:48:36.589413 213.551453 192.168.0.68 M arp 192.168.0.68 who 192.168.0.66 3 3 126 192 CON
> 2012/02/13.17:48:36.589949 0.000000 192.168.0.68 e udp 192.168.0.68.50251 <-> 192.168.0.66.domain 1 1 77 155 CON
> 2012/02/13.17:48:36.590557 0.000000 192.168.0.68 e udp 192.168.0.68.53404 <-> 192.168.0.66.domain 1 1 71 136 CON
> 2012/02/13.17:48:36.590954 0.000000 192.168.0.68 M arp 192.168.0.68 who 192.168.0.1 1 1 42 64 CON
> 2012/02/13.17:48:36.591391 213.550949 192.168.0.68 e arp 192.168.0.66 who 192.168.0.1 4 0 256 0 INT
> 2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
> 2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
> 2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
> 2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
>
> So, without a racluster.conf file, all the programs do the right thing.
> With your racluster configuration, we get this using racluster.1, which is correct behavior:
>
> racluster -f racluster.conf -M time 1h -r argus.simple.data.out
> StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
> 2012/02/13.17:48:36.592155 27.203648 192.168.0.68 e g tcp 192.168.0.68.* -> 128.0.0.0/1.http 38 47 5922 44901 FIN
> 2012/02/13.17:48:36.589413 213.552917 192.168.0.68 M arp 0.0.0.0 who 0.0.0.0 8 4 424 256 CON
> 2012/02/13.17:48:36.589949 0.000608 192.168.0.68 e udp 0.0.0.0.* <-> 0.0.0.0.* 2 2 148 291 CON
>
> However, rabins, when run with the "-M time 1h" option, should generate the exact same output as racluster (should have only one bin) but:
>
> thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out
> StartTime Dur SrcId Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
> 2012/02/13.17:48:36.592155 27.203621 192.168.0.68 e tcp 192.168.0.68.60245 -> 128.2.129.188.http 12 15 2314 15894 FIN
> 2012/02/13.17:48:36.632662 27.163141 192.168.0.68 e tcp 192.168.0.68.60246 -> 216.92.14.146.http 10 14 1001 14167 FIN
> 2012/02/13.17:48:36.705481 27.090235 192.168.0.68 e tcp 192.168.0.68.60247 -> 128.2.129.188.http 10 13 1433 13292 FIN
> 2012/02/13.17:48:36.705669 27.090014 192.168.0.68 e i tcp 192.168.0.68.60248 -> 128.2.129.188.http 6 5 1174 1548 FIN
>
> So something happened to the arp and udp traffic. I'll have to fix this, hopefully by Thurs :O(
>
> Carter
>
> On Apr 23, 2012, at 5:03 PM, Mark E. Mallett wrote:
>
>> On Mon, Apr 23, 2012 at 03:40:52PM -0400, Carter Bullard wrote:
>>> Hey Mark,
>>>
>>> Yes, the "-f " is for a racluster.conf file. It should aggregate
>>> within the bin, so its as if you ran racluster on each bin as the data
>>> arrives.
>>>
>>> Sorry about the man page, I'll take a look later today. Any updates
>>> will go on the web server. You're using 3.0.6 code?
>>
>> yes, 3.0.6, on various systems.
>>
>> And fantastic, that's what I hoped. Let me describe my simple tests
>> and perhaps somebody can tell me what dumb thing I am doing.
>>
>> It looked like the racluster.conf aggregation rules could be pretty
>> useful so I thought I'd try it out with some simple reports where the
>> details aren't too important but the method is. I figured if the method
>> works I can go onwards from there. But I got stuck early on.
>>
>> The simple test is to get some per-hour totals broken down into
>> potentially high-interest categories in different ways. I chose http
>> and https and ssh and "everything else" as categories for the test.
>> Here's my file c3.conf
>>
>> =======
>> # I've tried this on or off
>> RACLUSTER_PRESERVE_FIELDS=no
>> #
>> filter="tcp and dst port eq 80" model="proto dport"
>> filter="tcp and dst port eq 443" model="proto dport"
>> filter="tcp and dst port eq 22" model="proto dport"
>> # filter="tcp or udp" model="proto"
>> filter="" model="proto"
>> =========
>>
>> I processed this with rabins doing time binning (see command below),
>> expecting this to give me one record of aggregated data for http, one
>> for https, one for ssh per time bin, and then other records in each bin
>> aggregated only by protocol.
>>
>> When I run rabins I either write the argus records to a file and look at
>> that with some other ra* utility or just look at the output directly;
>> the result seems to be the same. This is the sort of command I use to
>> time bin and look at the output:
>>
>> rabins -n -f c3.conf -M zero -M time 1h -r input-file \
>> -s stime proto dport pkts bytes dur:12 | less
>>
>> With that command, I do not seem to get any aggregation at all. I get a
>> lot of individual flow records matching the tcp statements (http, https,
>> ssh), and no records for any other protocol or ports. This last part
>> surprises me most because I thought the c3.conf file will catch
>> everything, even if the aggregation doesn't happen.
>>
>> If I add a "model" set on the command line, i.e. an '-m' switch like this:
>>
>> rabins -n -f c3.conf -M zero -M time 1h -m proto dport -r input-file \
>> -s stime proto dport pkts bytes dur:12 | less
>>
>> then I get the per-bin aggregation I expected of the http, https, and ssh
>> traffic, but I still get no other output (except what's generated by the
>> "-M zero" for otherwise empty bins). Here I would still expect the last
>> line in c3.conf to catch some things and produce things aggregated
>> by "proto".
>>
>> If I uncomment the next-to-last filter line, then I do see some other
>> records show up; but for the most part they also appear unaggregated
>> except by flow.
>>
>> I could go on for a while but this is already long enough and probably
>> quite enough to introduce my confusion. One of the issues seems to be
>> that the "model" section of each line isn't being applied, and the only
>> model that gets applied is the one in '-m' on the command line. But I
>> don't think that's the only issue. There's probably some dots that I
>> haven't connected..
>>
>> Yours,
>> -mm-
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120430/db45b355/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabins.c
Type: application/octet-stream
Size: 45550 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120430/db45b355/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120430/db45b355/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120430/db45b355/attachment.bin>
More information about the argus
mailing list