Problems with racluster
Rafael Barbosa
rrbarbosa at gmail.com
Mon Sep 10 05:50:26 EDT 2012
Hi Carter,
The new patch seems to solve the problem with the port0 test case, and I
did not see any records with a reversed direction.
However, I still do not understand the output for the first test, with
files f1 and f2:
$> ra -r f1.argus
21:09:30.970830 e * tcp 172.31.1.102.61722 ->
172.31.1.100.10500 22 6772 FIN
$> ra -r f2.argus
21:09:30.971095 e tcp 172.31.1.100.10500 ?>
172.31.1.102.61722 1 66 CON
21:11:52.493919 e * tcp 172.31.1.102.61722 ->
172.31.1.100.10500 23 6838 FIN
$> ~/workspace/argus-clients-3.0.7.1-patch2/bin/racluster -r f1.argus
f2.argus -f racluster.conf
StartTime Flgs Proto SrcAddr Sport Dir
DstAddr Dport TotPkts TotBytes State
21:09:30.970830 e * tcp 172.31.1.102.61722 ->
172.31.1.100.10500 22 6772 FIN
21:11:52.493919 e * tcp 172.31.1.102.61722 ->
172.31.1.100.10500 24 6904 FIN
I see 3 possible outputs:
1 - The first record in f2.argus is aggregated back to the record in
f1.argus. The output is two records with the same TotPkts/TotBytes. In this
case, each TCP connection becomes a separated record, which is what I
wanted.
2 - The first record in f2.argus is aggregated to the second record in
f2.argus. The time stamp would be the one of first record in f2.argus. This
is seems to be what racluster is doing, however the timestamp is not
updated.
3 - The records are not aggregated at all. The first record f2.argus does
not aggregate back to the record in f1.argus because this record is marked
as "FIN", and does not aggregate to the second record in f2.argus because
the connection wasn't open yet.
Could you explain how the aggregation is being done?
Thanks,
Rafael Barbosa
http://www.ewi.utwente.nl/~barbosarr/
On Fri, Sep 7, 2012 at 9:21 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Rafael,
> Can you try out this version of racluster.c? Should do better. May still
> have
> an issue, but I'm thinking it should work for your cases. Shouldn't need
> the
> filter.
>
> Carter
>
>
>
> On Sep 4, 2012, at 5:29 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>
> Hi Carter,
>
> The racluster version you sent to this list seems to cause another bug.
> These are the outputs of racluster v3.0.7.1 (correct) and the patched
> version you provided:
>
> $> ~/workspace/argus-clients-3.0.7.1/bin/racluster -r port0.argus -f
> racluster.conf | grep 13:35:04.950149
> 13:35:04.950149 e g tcp 172.31.1.102.40767 <?>
> 172.31.1.123.x11 43 6494 CON
> $> ~/workspace/argus-clients-3.0.7.1-patch/bin/racluster -r port0.argus -f
> racluster.conf | grep 13:35:04.950149
> 13:35:04.950149 e g tcp 172.31.1.96.* <?>
> 172.31.1.96.* 43 6494 CON
>
> I uploaded port0.argus to the ftp.
>
> Best regards,
> Rafael Barbosa
> http://www.ewi.utwente.nl/~barbosarr/
>
>
>
> On Mon, Sep 3, 2012 at 9:38 PM, Carter Bullard <carter at qosient.com> wrote:
>
>> Hy Rafael,
>> For the others on the list, your racluster.conf file has an idle timeout
>> of 300s
>> applied to any flow. The problem that you are seeing is a bug, and is
>> caused
>> by the timestamp in the management records. The man records are
>> advancing the
>> ArgusGlobalTime, (it should no do this), which is the master clock for the
>> aggregator. When racluster() gets the close MAR record in the file
>> created
>> from your f1.pcap file, the timestamp, which is the time the file was
>> created, not
>> the timestamps of the packets, this causes racluster() to idle out your
>> flow cache.
>>
>> Try this version of your command:
>>
>> racluster -r f1.argus f2.argus -f ~/config/racluster.conf - not man
>>
>>
>> I will fix this bug later today / tomorrow.
>>
>> Sorry for the inconvenience,
>>
>> Carter
>>
>> On Sep 3, 2012, at 10:13 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>
>> Just uploaded the two pcap files and the racluster.conf to the ftp server
>> with the name 'racluster.tar.gz'.
>>
>> Rafael Barbosa
>> http://www.ewi.utwente.nl/~barbosarr/
>>
>>
>>
>> On Mon, Sep 3, 2012 at 3:33 PM, Carter Bullard <carter at qosient.com>wrote:
>>
>>> Hey Rafael,
>>> The problem relates to your racluster.conf file. For some reason its
>>> timing out
>>> the flow in f1.argus, before the f2.argus file shows up ?
>>>
>>> Send both files so I can debug this. Also send the racluster.conf file,
>>> if possible.
>>>
>>> Carter
>>>
>>> On Sep 3, 2012, at 4:46 AM, Rafael Barbosa <rrbarbosa at gmail.com> wrote:
>>>
>>> Hi Carter,
>>>
>>> The issue with the direction seems to be solved using clients 3.0.7.1:
>>> the single duplicated ACK from the previous connection is aggregated in the
>>> second connection, and the direction is not reversed.
>>>
>>> $> ra -r f1.argus f2.argus
>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 ->
>>> 172.31.1.100.10500 22 6772 FIN
>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?>
>>> 172.31.1.102.61722 1 66 CON
>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 ->
>>> 172.31.1.100.10500 23 6838 FIN
>>>
>>> $> racluster -r f1.argus f2.argus -f ~/config/racluster.conf
>>> StartTime Flgs Proto SrcAddr Sport Dir
>>> DstAddr Dport TotPkts TotBytes State
>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 ->
>>> 172.31.1.100.10500 22 6772 FIN
>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 ->
>>> 172.31.1.100.10500 24 6904 FIN
>>>
>>> However, if the duplicated packet is aggregated in the second record,
>>> why isn't the 'start time' also set to the time of the duplicated packet?
>>> In any case, would it be possible to aggregate the dup packet in the first
>>> record?
>>>
>>> Best regards,
>>> Rafael Barbosa
>>> http://www.ewi.utwente.nl/~barbosarr/
>>>
>>>
>>>
>>> On Sat, Sep 1, 2012 at 6:09 PM, Carter Bullard <carter at qosient.com>wrote:
>>>
>>>> Hey Rafael,
>>>> Give this copy a run to see if it doesn't do the right thing for you.
>>>> If so, I'll push the changes to all the aggregators.
>>>> Are you using 3.0.6.2 clients or 3.0.7.1 ?
>>>> Thanks !!!!!!
>>>>
>>>> Carter
>>>>
>>>>
>>>>
>>>> On Aug 30, 2012, at 8:29 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>> Hey Rafael,
>>>> Your situation should work fine, if not for the racluster() bug.
>>>>
>>>> The only problem is that racluster() gives precedence to the first
>>>> record, not the record with the definitive data, so the fix should be
>>>> simple, I hope. I'll try to fix this today.
>>>>
>>>> Regarding the model, I was just providing an example racluster.conf.
>>>> The status and idle timers are the important part.
>>>>
>>>> Carter
>>>>
>>>>
>>>> On Aug 30, 2012, at 3:55 AM, Rafael Barbosa <rrbarbosa at gmail.com>
>>>> wrote:
>>>>
>>>> Hi Carter,
>>>>
>>>> I do not care much about the first record, as long as the second record
>>>> keeps its correct direction. As you said, the second record contains the
>>>> 3-way handshake necessary to decide the flow direction.
>>>>
>>>> I am not sure if this is important, but my original example is a little
>>>> more complicated: I have another pcap file with the rest of the TCP
>>>> connection from which the first packet (from the first record in f2.argus)
>>>> belongs. The packet that generates the first record f2.argus is a duplicate
>>>> for the ACK that completes the teardown of this TCP connection. To make
>>>> sure I am clear:
>>>>
>>>> $> ra -r f1.argus
>>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 ->
>>>> 172.31.1.100.10500 22 6772 FIN
>>>> $> ra -r f2.argus
>>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?>
>>>> 172.31.1.102.61722 1 66 CON
>>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 ->
>>>> 172.31.1.100.10500 23 6838 FIN
>>>> $> racluster -r f1.argus f2.argus -f ~/config/racluster.conf
>>>> 21:09:30.970830 e * tcp 172.31.1.102.61722 ->
>>>> 172.31.1.100.10500 22 6772 FIN
>>>> 21:09:30.971095 e * tcp 172.31.1.100.10500 ->
>>>> 172.31.1.102.61722 24 6904 FIN
>>>>
>>>> I just assumed the bug is in the second file. But if you think is
>>>> necessary I can provide you with the first argus dump as well.
>>>>
>>>> Regarding my racluster.conf, I do not really care about real-time, as I
>>>> am using argus to post-process pcap files (I do not have access to the
>>>> network, just the dumps). But I have a question, why to you suggest to use
>>>> the 5-tuple as a "model" in racluster.conf? Isn't this the default behavior?
>>>>
>>>> And as usual, thanks for the fast reply.
>>>>
>>>> Best regards,
>>>> Rafael Barbosa
>>>> http://www.ewi.utwente.nl/~barbosarr/
>>>>
>>>>
>>>>
>>>> On Wed, Aug 29, 2012 at 8:19 PM, Carter Bullard <carter at qosient.com>wrote:
>>>>
>>>>> Hey Rafael,
>>>>> Yes, I'll have to fix this bug were the second record is erroneously
>>>>> reversed. The fix, however,
>>>>> will end up reversing the first record. The second record saw the
>>>>> connect setup, so it knows
>>>>> the correct direction. The first flow, is just the one packet,
>>>>> without any detail, so the direction
>>>>> is unknown. When you try to merge the two flows together, the bug is
>>>>> that it only looks at the first
>>>>> record for the direction, rather than evaluating both, and choosing
>>>>> the one that is a better choice.
>>>>>
>>>>> Is that cool with your line of thinking?
>>>>>
>>>>> If you're planning on running argus with 300s status intervals, that
>>>>> will eliminate any real-time
>>>>> possibilities. But if you're thinking about letting argus generate
>>>>> shorter lived status records, then
>>>>> run racluster() with a racluster.conf file, where you set a status
>>>>> timer for all flows at 300s:
>>>>>
>>>>> filter="" model="saddr daddr proto sport dport" status=300
>>>>> idle=300
>>>>>
>>>>> Hopefully this helps !!!!
>>>>>
>>>>> Carter
>>>>>
>>>>> On Aug 29, 2012, at 9:22 AM, Rafael Barbosa <rrbarbosa at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I am having some problems with a weird aggregation decision by
>>>>> racluster, which inverts the server/client roles (i.e., the direction
>>>>> field). This what happens:
>>>>>
>>>>> $> argus -r f2 -w f2.argus
>>>>> $> ra -r f2.argus
>>>>> 21:09:30.971095 e tcp 172.31.1.100.10500 ?>
>>>>> 172.31.1.102.61722 1 66 CON
>>>>> 21:11:52.493919 e * tcp 172.31.1.102.61722 ->
>>>>> 172.31.1.100.10500 23 6838 FIN
>>>>> $> racluster -r f2.argus -f racluster.conf
>>>>> 21:09:30.971095 e * tcp 172.31.1.100.10500 ->
>>>>> 172.31.1.102.61722 24 6904 FIN
>>>>> $> cat racluster.conf
>>>>> filter="" status=0 idle=300
>>>>>
>>>>> The pcap from which I generated f2.argus is attached to this message.
>>>>> The 'f2' pcap file is rather peculiar with duplicated frames and is a
>>>>> fragment of a larger capture where TCP ports are re-used in consecutive
>>>>> connections, but it is not a "toy" file, it was captured in a real network
>>>>> environment.
>>>>>
>>>>> My goal is to generate flow records with a 300s timeouts, and label
>>>>> hosts as servers and clients. Any thoughts on why this inversion happens,
>>>>> and how I can work around it?
>>>>>
>>>>> Thanks for the help,
>>>>> Rafael Barbosa
>>>>> http://www.ewi.utwente.nl/~barbosarr/
>>>>>
>>>>> <f2>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120910/62526a18/attachment.html>
More information about the argus
mailing list