Problem with byte-swapped IP addresses
Carter Bullard
carter at qosient.com
Fri Mar 26 09:38:00 EDT 2010
Hey Martijn,
Did the mods to argus improve your situation?
Carter
On Mar 8, 2010, at 11:45 PM, Carter Bullard wrote:
> Hey Martijn,
> So I have an argus that doesn't write into the packets. Give this a try to see
> if things don't get a little bit better.
>
> http://qosient.com/argus/dev/argus-3.0.3.3.tar.gz
>
> Carter
>
> On Mar 8, 2010, at 10:03 AM, Martijn van Oosterhout wrote:
>
>> On Mon, Mar 8, 2010 at 4:29 AM, Peter Van Epp <vanepp at sfu.ca> wrote:
>>> I suspect we will find this is high load related in which case turning
>>> off pf-ring likely causes enough packet loss that the load doesn't tickle
>>> the bug :-). Have you given pf-ring lots of memory for buffers? As I recall
>>> (its been a number of years since I last played with pf-ring) we had to boost
>>> the kernel VM space limit to be able to get a couple of megs of buffer space
>>> for pf-ring although that may have been a 2.4 series kernel rather than 2.6.
>>
>> That's an idea. Unfortunately I don't see a simple way to determine if
>> argus is dropping many packets or not. We've configured argus to have
>> a ring-buffer of 16384 packets. You can make it bigger, but if argus
>> isn't keeping up then it doesn't really matter how big you make it.
>> argus isn't using 100% CPU, but maybe that's a lie.
>>
>> In any case, I have a comparison between tcpdump and argus. Here is
>> the output that argus dumped:
>>
>> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
>> 2854318090:2854319550(1460) ack 1379121316 win 64938
>> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
>> 2087784273:2087784391(118) ack 2513133511 win 65391
>> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
>> 4294954156 win 17520
>> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
>> 1460:2920(1460) ack 1 win 64938
>> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
>> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
>> 170
>> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
>> 46.20.168.192: tcp
>> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 1721333935:1721335395(1460) ack 3142743201 win 64675
>> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 1460:2920(1460) ack 1 win 64675
>> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 2920:4380(1460) ack 1 win 64675
>>
>> Note the jump in timestamps at the same moment. This is what tcpdump saw:
>>
>> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
>> 2854318090:2854319550(1460) ack 1379121316 win 64938
>> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
>> 2087784273:2087784391(118) ack 2513133511 win 65391
>> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
>> 4294954156 win 17520
>> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
>> 1460:2920(1460) ack 1 win 64938
>>
>> ... 3846 other packets ...
>>
>> 14:22:43.702911 IP 192.168.20.120.1287 > 192.168.20.95.139: P
>> 2739085214:2739085344(130) ack 4097980446 win 63368
>> 14:22:43.702936 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 1721332475:1721333935(1460) ack 3142743201 win 64675
>> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 1460:2920(1460) ack 1 win 64675
>> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 2920:4380(1460) ack 1 win 64675
>> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
>> 4380:5840(1460) ack 1 win 64675
>>
>> So basically, at the moment the byte-swapping happens, there's also a
>> huge amount of data lost (92ms worth).
>>
>> The first corrupted packet:
>>
>> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
>> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
>> 170
>> 0x0000: 000f 1f66 11bc 0050 5693 2a15 0008 4500 ...f...PV.*...E.
>> 0x0010: aa00 57d0 0040 8006 7fce 7814 a8c0 5f14 ..W.. at ....x..._.
>> 0x0020: a8c0 0507 008b a343 1f9e f442 381e 5018 .......C...B8.P.
>> 0x0030: f788 6006 0000 0000 007e ..`......~
>>
>> Has byte-swappped: ethernet type, IP total length, fragment offset, src&dst IP,
>>
>> The second corrupted packet:
>>
>> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
>> 46.20.168.192: tcp
>> 0x0000: 0050 5693 2a15 001a a02c b60d 0800 4500 .PV.*....,....E.
>> 0x0010: dc05 372d 0040 8006 1df1 2b14 a8c0 2e14 ..7-. at ....+.....
>> 0x0020: a8c0 006e 1270 6699 72fb bb52 74a1 5010 ...n.pf.r..Rt.P.
>> 0x0030: fca3 9627 0000 7654 6c46 ...'..vTlF
>>
>> is the same except for the ethernet type, which is OK. The TCP headers
>> are not swapped in either. Frankly, I'm totally confused.
>>
>> I think there are options in pcap-ringbuffer to dump extra tracing
>> info, so I'm going to check that. Finally, is it easy to disabled the
>> inplace switching, just to see if that make a difference (sure,
>> everything will be backwards, but if it's all consistently backwards
>> we've found the culprit)?
>>
>> Thanks in advance,
>> --
>> Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100326/d76ae080/attachment.bin>
More information about the argus
mailing list