Problem with byte-swapped IP addresses

[Carter Bullard]

Hey Martijn,
So the second corrupted packet sez "54825 bytes missing".  
Could 54K account for the missing 3846 packets?

OK, one possibility is that argus() could be modifying what it thinks are
packet contents, but the buffer offset is beyond the snap-length, and so
we end up munging the next packets header, timestamp, whatever.

What is your snaplen?  Are you capturing user data?  Does  the problem
go away if you make the snaplen larger, like adding an additional 64-128
bytes?

Carter

On Mar 8, 2010, at 10:03 AM, Martijn van Oosterhout wrote:

> On Mon, Mar 8, 2010 at 4:29 AM, Peter Van Epp <vanepp at sfu.ca> wrote:
>>        I suspect we will find this is high load related in which case turning
>> off pf-ring likely causes enough packet loss that the load doesn't tickle
>> the bug :-). Have you given pf-ring lots of memory for buffers? As I recall
>> (its been a number of years since I last played with pf-ring) we had to boost
>> the kernel VM space limit to be able to get a couple of megs of buffer space
>> for pf-ring although that may have been a 2.4 series kernel rather than 2.6.
> 
> That's an idea. Unfortunately I don't see a simple way to determine if
> argus is dropping many packets or not. We've configured argus to have
> a ring-buffer of 16384 packets. You can make it bigger, but if argus
> isn't keeping up then it doesn't really matter how big you make it.
> argus isn't using 100% CPU, but maybe that's a lie.
> 
> In any case, I have a comparison between tcpdump and argus. Here is
> the output that argus dumped:
> 
> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 2854318090:2854319550(1460) ack 1379121316 win 64938
> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
> 2087784273:2087784391(118) ack 2513133511 win 65391
> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
> 4294954156 win 17520
> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 1460:2920(1460) ack 1 win 64938
> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
> 170
> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
> 46.20.168.192: tcp
> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1721333935:1721335395(1460) ack 3142743201 win 64675
> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1460:2920(1460) ack 1 win 64675
> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 2920:4380(1460) ack 1 win 64675
> 
> Note the jump in timestamps at the same moment. This is what tcpdump saw:
> 
> 14:22:43.610709 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 2854318090:2854319550(1460) ack 1379121316 win 64938
> 14:22:43.610744 IP 192.168.15.23.1126 > 192.168.20.121.139: P
> 2087784273:2087784391(118) ack 2513133511 win 65391
> 14:22:43.610762 IP 192.168.7.168.1614 > 192.168.20.2.139: . ack
> 4294954156 win 17520
> 14:22:43.610790 IP 192.168.20.2.139 > 192.168.7.168.1614: .
> 1460:2920(1460) ack 1 win 64938
> 
> ... 3846 other packets ...
> 
> 14:22:43.702911 IP 192.168.20.120.1287 > 192.168.20.95.139: P
> 2739085214:2739085344(130) ack 4097980446 win 63368
> 14:22:43.702936 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1721332475:1721333935(1460) ack 3142743201 win 64675
> 14:22:43.702959 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 1460:2920(1460) ack 1 win 64675
> 14:22:43.702980 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 2920:4380(1460) ack 1 win 64675
> 14:22:43.703000 IP 192.168.20.43.110 > 192.168.20.120.4720: .
> 4380:5840(1460) ack 1 win 64675
> 
> So basically, at the moment the byte-swapping happens, there's also a
> huge amount of data lost (92ms worth).
> 
> The first corrupted packet:
> 
> 14:22:43.702911 00:50:56:93:2a:15 Null > 00:0f:1f:66:11:bc Unknown
> DSAP 0x44 Information, send seq 85, rcv seq 0, Flags [Command], length
> 170
>        0x0000:  000f 1f66 11bc 0050 5693 2a15 0008 4500  ...f...PV.*...E.
>        0x0010:  aa00 57d0 0040 8006 7fce 7814 a8c0 5f14  ..W.. at ....x..._.
>        0x0020:  a8c0 0507 008b a343 1f9e f442 381e 5018  .......C...B8.P.
>        0x0030:  f788 6006 0000 0000 007e                 ..`......~
> 
> Has byte-swappped: ethernet type, IP total length, fragment offset, src&dst IP,
> 
> The second corrupted packet:
> 
> 14:22:43.702936 IP truncated-ip - 54825 bytes missing! 43.20.168.192 >
> 46.20.168.192: tcp
>        0x0000:  0050 5693 2a15 001a a02c b60d 0800 4500  .PV.*....,....E.
>        0x0010:  dc05 372d 0040 8006 1df1 2b14 a8c0 2e14  ..7-. at ....+.....
>        0x0020:  a8c0 006e 1270 6699 72fb bb52 74a1 5010  ...n.pf.r..Rt.P.
>        0x0030:  fca3 9627 0000 7654 6c46                 ...'..vTlF
> 
> is the same except for the ethernet type, which is OK. The TCP headers
> are not swapped in either. Frankly, I'm totally confused.
> 
> I think there are options in pcap-ringbuffer to dump extra tracing
> info, so I'm going to check that. Finally, is it easy to disabled the
> inplace switching, just to see if that make a difference (sure,
> everything will be backwards, but if it's all consistently backwards
> we've found the culprit)?
> 
> Thanks in advance,
> -- 
> Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/
> 



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100308/4c273a8e/attachment.bin>