Problem with byte-swapped IP addresses
Carter Bullard
carter at qosient.com
Mon Mar 8 23:45:21 EST 2010
Hey Martijn,
So I have an argus that doesn't write into the packets. Give this a try to see
if things don't get a little bit better.
http://qosient.com/argus/dev/argus-3.0.3.3.tar.gz
Carter
On Mar 8, 2010, at 10:03 AM, Martijn van Oosterhout wrote:
> On Mon, Mar 8, 2010 at 4:29 AM, Peter Van Epp <vanepp at sfu.ca> wrote:
>> I suspect we will find this is high load related in which case turning
>> off pf-ring likely causes enough packet loss that the load doesn't tickle
>> the bug :-). Have you given pf-ring lots of memory for buffers? As I recall
>> (its been a number of years since I last played with pf-ring) we had to boost
>> the kernel VM space limit to be able to get a couple of megs of buffer space
>> for pf-ring although that may have been a 2.4 series kernel rather than 2.6.
>
> That's an idea. Unfortunately I don't see a simple way to determine if
> argus is dropping many packets or not. We've configured argus to have
> a ring-buffer of 16384 packets. You can make it bigger, but if argus
> isn't keeping up then it doesn't really matter how big you make it.
> argus isn't using 100% CPU, but maybe that's a lie.
>
> In any case, I have a comparison between tcpdump and argus. Here is
> the output that argus dumped:
>
> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 2854318090:2854319550(1460) ack 1379121316 win 64938
> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
> 2087784273:2087784391(118) ack 2513133511 win 65391
> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
> 4294954156 win 17520
> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 1460:2920(1460) ack 1 win 64938
> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
> 170
> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
> 46.20.168.192: tcp
> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1721333935:1721335395(1460) ack 3142743201 win 64675
> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1460:2920(1460) ack 1 win 64675
> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 2920:4380(1460) ack 1 win 64675
>
> Note the jump in timestamps at the same moment. This is what tcpdump saw:
>
> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 2854318090:2854319550(1460) ack 1379121316 win 64938
> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
> 2087784273:2087784391(118) ack 2513133511 win 65391
> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
> 4294954156 win 17520
> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 1460:2920(1460) ack 1 win 64938
>
> ... 3846 other packets ...
>
> 14:22:43.702911 IP 192.168.20.120.1287 > 192.168.20.95.139: P
> 2739085214:2739085344(130) ack 4097980446 win 63368
> 14:22:43.702936 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1721332475:1721333935(1460) ack 3142743201 win 64675
> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1460:2920(1460) ack 1 win 64675
> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 2920:4380(1460) ack 1 win 64675
> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 4380:5840(1460) ack 1 win 64675
>
> So basically, at the moment the byte-swapping happens, there's also a
> huge amount of data lost (92ms worth).
>
> The first corrupted packet:
>
> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
> 170
> 0x0000: 000f 1f66 11bc 0050 5693 2a15 0008 4500 ...f...PV.*...E.
> 0x0010: aa00 57d0 0040 8006 7fce 7814 a8c0 5f14 ..W.. at ....x..._.
> 0x0020: a8c0 0507 008b a343 1f9e f442 381e 5018 .......C...B8.P.
> 0x0030: f788 6006 0000 0000 007e ..`......~
>
> Has byte-swappped: ethernet type, IP total length, fragment offset, src&dst IP,
>
> The second corrupted packet:
>
> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
> 46.20.168.192: tcp
> 0x0000: 0050 5693 2a15 001a a02c b60d 0800 4500 .PV.*....,....E.
> 0x0010: dc05 372d 0040 8006 1df1 2b14 a8c0 2e14 ..7-. at ....+.....
> 0x0020: a8c0 006e 1270 6699 72fb bb52 74a1 5010 ...n.pf.r..Rt.P.
> 0x0030: fca3 9627 0000 7654 6c46 ...'..vTlF
>
> is the same except for the ethernet type, which is OK. The TCP headers
> are not swapped in either. Frankly, I'm totally confused.
>
> I think there are options in pcap-ringbuffer to dump extra tracing
> info, so I'm going to check that. Finally, is it easy to disabled the
> inplace switching, just to see if that make a difference (sure,
> everything will be backwards, but if it's all consistently backwards
> we've found the culprit)?
>
> Thanks in advance,
> --
> Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100308/4804ce2d/attachment.bin>
More information about the argus
mailing list