Problems with racluster
Carter Bullard
carter at qosient.com
Tue Sep 4 14:21:19 EDT 2012
Hey Rafael,
I commented out a few lines, for testing purposes. I'll put them back.
When you are doing aggregation, you should have this entry in your rarc file.
RA_CIDR_ADDRESS_FORMAT="yes"
The output you are getting should be:
06:59:53.978852 e g tcp 172.31.1.96/27.* <?> 172.31.1.96/27.* 70 62 13172 4336 CON
This is because we are not correcting the record for direction, so racluster
aggregates two records where the addresses and ports are swapped.
The actual aggregation is correct.
Carter
On Sep 4, 2012, at 5:29 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
> Hi Carter,
>
> The racluster version you sent to this list seems to cause another bug. These are the outputs of racluster v3.0.7.1 (correct) and the patched version you provided:
>
> $> ~/workspace/argus-clients-3.0.7.1/bin/racluster -r port0.argus -f racluster.conf | grep 13:35:04.950149
> 13:35:04.950149 e g tcp 172.31.1.102.40767 <?> 172.31.1.123.x11 43 6494 CON
> $> ~/workspace/argus-clients-3.0.7.1-patch/bin/racluster -r port0.argus -f racluster.conf | grep 13:35:04.950149
> 13:35:04.950149 e g tcp 172.31.1.96.* <?> 172.31.1.96.* 43 6494 CON
>
> I uploaded port0.argus to the ftp.
>
> Best regards,
> Rafael Barbosa
> http://www.ewi.utwente.nl/~barbosarr/
>
>
>
> On Mon, Sep 3, 2012 at 9:38 PM, Carter Bullard <carter at qosient.com> wrote:
> Hy Rafael,
> For the others on the list, your racluster.conf file has an idle timeout of 300s
> applied to any flow. The problem that you are seeing is a bug, and is caused
> by the timestamp in the management records. The man records are advancing the
> ArgusGlobalTime, (it should no do this), which is the master clock for the
> aggregator. When racluster() gets the close MAR record in the file created
> from your f1.pcap file, the timestamp, which is the time the file was created, not
> the timestamps of the packets, this causes racluster() to idle out your flow cache.
>
> Try this version of your command:
>
> racluster -r f1.argus f2.argus -f ~/config/racluster.conf - not man
>
>
> I will fix this bug later today / tomorrow.
>
> Sorry for the inconvenience,
>
> Carter
>
> On Sep 3, 2012, at 10:13 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>
>> Just uploaded the two pcap files and the racluster.conf to the ftp server with the name 'racluster.tar.gz'.
>>
>> Rafael Barbosa
>> http://www.ewi.utwente.nl/~barbosarr/
>>
>>
>>
>> On Mon, Sep 3, 2012 at 3:33 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Rafael,
>> The problem relates to your racluster.conf file. For some reason its timing out
>> the flow in f1.argus, before the f2.argus file shows up ?
>>
>> Send both files so I can debug this. Also send the racluster.conf file, if possible.
>>
>> Carter
>>
>> On Sep 3, 2012, at 4:46 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>
>>> Hi Carter,
>>>
>>> The issue with the direction seems to be solved using clients 3.0.7.1: the single duplicated ACK from the previous connection is aggregated in the second connection, and the direction is not reversed.
>>>
>>> $> ra -r f1.argus f2.argus
>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 22 6772 FIN
>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?> 172.31.1.102.61722 1 66 CON
>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 23 6838 FIN
>>>
>>> $> racluster -r f1.argus f2.argus -f ~/config/racluster.conf
>>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 22 6772 FIN
>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 24 6904 FIN
>>>
>>> However, if the duplicated packet is aggregated in the second record, why isn't the 'start time' also set to the time of the duplicated packet? In any case, would it be possible to aggregate the dup packet in the first record?
>>>
>>> Best regards,
>>> Rafael Barbosa
>>> http://www.ewi.utwente.nl/~barbosarr/
>>>
>>>
>>>
>>> On Sat, Sep 1, 2012 at 6:09 PM, Carter Bullard <carter at qosient.com> wrote:
>>> Hey Rafael,
>>> Give this copy a run to see if it doesn't do the right thing for you.
>>> If so, I'll push the changes to all the aggregators.
>>> Are you using 3.0.6.2 clients or 3.0.7.1 ?
>>> Thanks !!!!!!
>>>
>>> Carter
>>>
>>>
>>>
>>> On Aug 30, 2012, at 8:29 AM, Carter Bullard <carter at qosient.com> wrote:
>>>
>>>> Hey Rafael,
>>>> Your situation should work fine, if not for the racluster() bug.
>>>>
>>>> The only problem is that racluster() gives precedence to the first record, not the record with the definitive data, so the fix should be simple, I hope. I'll try to fix this today.
>>>>
>>>> Regarding the model, I was just providing an example racluster.conf. The status and idle timers are the important part.
>>>>
>>>> Carter
>>>>
>>>>
>>>> On Aug 30, 2012, at 3:55 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>>>
>>>>> Hi Carter,
>>>>>
>>>>> I do not care much about the first record, as long as the second record keeps its correct direction. As you said, the second record contains the 3-way handshake necessary to decide the flow direction.
>>>>>
>>>>> I am not sure if this is important, but my original example is a little more complicated: I have another pcap file with the rest of the TCP connection from which the first packet (from the first record in f2.argus) belongs. The packet that generates the first record f2.argus is a duplicate for the ACK that completes the teardown of this TCP connection. To make sure I am clear:
>>>>>
>>>>> $> ra -r f1.argus
>>>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 22 6772 FIN
>>>>> $> ra -r f2.argus
>>>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?> 172.31.1.102.61722 1 66 CON
>>>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 23 6838 FIN
>>>>> $> racluster -r f1.argus f2.argus -f ~/config/racluster.conf
>>>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 22 6772 FIN
>>>>> 21:09:30.971095 e * tcp 172.31.1.100.10500 -> 172.31.1.102.61722 24 6904 FIN
>>>>>
>>>>> I just assumed the bug is in the second file. But if you think is necessary I can provide you with the first argus dump as well.
>>>>>
>>>>> Regarding my racluster.conf, I do not really care about real-time, as I am using argus to post-process pcap files (I do not have access to the network, just the dumps). But I have a question, why to you suggest to use the 5-tuple as a "model" in racluster.conf? Isn't this the default behavior?
>>>>>
>>>>> And as usual, thanks for the fast reply.
>>>>>
>>>>> Best regards,
>>>>> Rafael Barbosa
>>>>> http://www.ewi.utwente.nl/~barbosarr/
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 29, 2012 at 8:19 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>> Hey Rafael,
>>>>> Yes, I'll have to fix this bug were the second record is erroneously reversed. The fix, however,
>>>>> will end up reversing the first record. The second record saw the connect setup, so it knows
>>>>> the correct direction. The first flow, is just the one packet, without any detail, so the direction
>>>>> is unknown. When you try to merge the two flows together, the bug is that it only looks at the first
>>>>> record for the direction, rather than evaluating both, and choosing the one that is a better choice.
>>>>>
>>>>> Is that cool with your line of thinking?
>>>>>
>>>>> If you're planning on running argus with 300s status intervals, that will eliminate any real-time
>>>>> possibilities. But if you're thinking about letting argus generate shorter lived status records, then
>>>>> run racluster() with a racluster.conf file, where you set a status timer for all flows at 300s:
>>>>>
>>>>> filter="" model="saddr daddr proto sport dport" status=300 idle=300
>>>>>
>>>>> Hopefully this helps !!!!
>>>>>
>>>>> Carter
>>>>>
>>>>> On Aug 29, 2012, at 9:22 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am having some problems with a weird aggregation decision by racluster, which inverts the server/client roles (i.e., the direction field). This what happens:
>>>>>>
>>>>>> $> argus -r f2 -w f2.argus
>>>>>> $> ra -r f2.argus
>>>>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?> 172.31.1.102.61722 1 66 CON
>>>>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 -> 172.31.1.100.10500 23 6838 FIN
>>>>>> $> racluster -r f2.argus -f racluster.conf
>>>>>> 21:09:30.971095 e * tcp 172.31.1.100.10500 -> 172.31.1.102.61722 24 6904 FIN
>>>>>> $> cat racluster.conf
>>>>>> filter="" status=0 idle=300
>>>>>>
>>>>>> The pcap from which I generated f2.argus is attached to this message. The 'f2' pcap file is rather peculiar with duplicated frames and is a fragment of a larger capture where TCP ports are re-used in consecutive connections, but it is not a "toy" file, it was captured in a real network environment.
>>>>>>
>>>>>> My goal is to generate flow records with a 300s timeouts, and label hosts as servers and clients. Any thoughts on why this inversion happens, and how I can work around it?
>>>>>>
>>>>>> Thanks for the help,
>>>>>> Rafael Barbosa
>>>>>> http://www.ewi.utwente.nl/~barbosarr/
>>>>>>
>>>>>> <f2>
>>>>>
>>>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120904/6706918b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120904/6706918b/attachment.bin>
More information about the argus
mailing list