Direction and IP/TCP timeout settings
Craig Merchant
cmerchant at responsys.com
Fri Jul 19 19:14:38 EDT 2013
Hey, Carter...
I've made the changes you recommended in your previous emails. It'll take a while longer to see if the direction issue gets sorted out. Argus now runs at like 100% CPU utilization almost all the time.
I still get a crash trying to convert a pcap file to argus format. The output is:
[root at ids01-dc1 tcpdump]# argus -r tcpdump.pcap -A -J -R -Z -w tcpdump.argus
*** glibc detected *** argus: double free or corruption (fasttop): 0x0000000001ce5260 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x760e6)[0x7fcb6592e0e6]
argus[0x437516]
argus[0x41e36c]
argus[0x406274]
argus[0x4056fd]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fcb658d6cdd]
argus[0x403d79]
======= Memory map: ========
00400000-00465000 r-xp 00000000 fd:00 71223416 /usr/local/sbin/argus
00665000-00669000 rw-p 00065000 fd:00 71223416 /usr/local/sbin/argus
00669000-0066f000 rw-p 00000000 00:00 0
01ce4000-01d05000 rw-p 00000000 00:00 0 [heap]
7fcb65068000-7fcb6507e000 r-xp 00000000 fd:00 78233953 /lib64/libgcc_s-4.4.7-20120601.so.1
7fcb6507e000-7fcb6527d000 ---p 00016000 fd:00 78233953 /lib64/libgcc_s-4.4.7-20120601.so.1
7fcb6527d000-7fcb6527e000 rw-p 00015000 fd:00 78233953 /lib64/libgcc_s-4.4.7-20120601.so.1
7fcb65285000-7fcb65690000 rw-p 00000000 00:00 0
7fcb65690000-7fcb656b3000 r-xp 00000000 fd:00 9388269 /opt/rb/lib/libpfring.so
7fcb656b3000-7fcb658b2000 ---p 00023000 fd:00 9388269 /opt/rb/lib/libpfring.so
7fcb658b2000-7fcb658b4000 rw-p 00022000 fd:00 9388269 /opt/rb/lib/libpfring.so
7fcb658b8000-7fcb65a42000 r-xp 00000000 fd:00 78233613 /lib64/libc-2.12.so
7fcb65a42000-7fcb65c41000 ---p 0018a000 fd:00 78233613 /lib64/libc-2.12.so
7fcb65c41000-7fcb65c45000 r--p 00189000 fd:00 78233613 /lib64/libc-2.12.so
7fcb65c45000-7fcb65c46000 rw-p 0018d000 fd:00 78233613 /lib64/libc-2.12.so
7fcb65c46000-7fcb65c4b000 rw-p 00000000 00:00 0
7fcb65c50000-7fcb65cd3000 r-xp 00000000 fd:00 78233621 /lib64/libm-2.12.so
7fcb65cd3000-7fcb65ed2000 ---p 00083000 fd:00 78233621 /lib64/libm-2.12.so
7fcb65ed2000-7fcb65ed3000 r--p 00082000 fd:00 78233621 /lib64/libm-2.12.so
7fcb65ed3000-7fcb65ed4000 rw-p 00083000 fd:00 78233621 /lib64/libm-2.12.so
7fcb65ed8000-7fcb65eef000 r-xp 00000000 fd:00 78233637 /lib64/libpthread-2.12.so
7fcb65eef000-7fcb660ef000 ---p 00017000 fd:00 78233637 /lib64/libpthread-2.12.so
7fcb660ef000-7fcb660f0000 r--p 00017000 fd:00 78233637 /lib64/libpthread-2.12.so
7fcb660f0000-7fcb660f1000 rw-p 00018000 fd:00 78233637 /lib64/libpthread-2.12.so
7fcb660f1000-7fcb660f5000 rw-p 00000000 00:00 0
7fcb660f8000-7fcb6612f000 r-xp 00000000 fd:00 9388267 /opt/rb/lib/libpcap.so.1.1.1
7fcb6612f000-7fcb6632f000 ---p 00037000 fd:00 9388267 /opt/rb/lib/libpcap.so.1.1.1
7fcb6632f000-7fcb66331000 rw-p 00037000 fd:00 9388267 /opt/rb/lib/libpcap.so.1.1.1
7fcb66331000-7fcb66332000 rw-p 00000000 00:00 0
7fcb66338000-7fcb66358000 r-xp 00000000 fd:00 78233603 /lib64/ld-2.12.so
7fcb6654b000-7fcb66550000 rw-p 00000000 00:00 0
7fcb66555000-7fcb66557000 rw-p 00000000 00:00 0
7fcb66557000-7fcb66558000 r--p 0001f000 fd:00 78233603 /lib64/ld-2.12.so
7fcb66558000-7fcb66559000 rw-p 00020000 fd:00 78233603 /lib64/ld-2.12.so
7fcb66559000-7fcb6655a000 rw-p 00000000 00:00 0
7fff087c2000-7fff087d7000 rw-p 00000000 00:00 0 [stack]
7fff08800000-7fff08801000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted
Thx.
Craig
-----Original Message-----
From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, July 19, 2013 12:25 PM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
Sorry, add this snippet of code.
Carter
==== //depot/argus/argus/argus/ArgusSource.c#110 - /Volumes/Users/carter/argus/argus/argus/ArgusSource.c ====
2177a2178,2186
> void
> ArgusIpNetPacket (u_char *user, const struct pcap_pkthdr *h, const u_char *p)
> {
>
> #ifdef ARGUSDEBUG
> ArgusDebug (8, "ArgusIpNetPacket (%p, %p, %p) returning\n", user, h, p);
> #endif
> }
>
On Jul 19, 2013, at 3:12 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> Hey, Carter...
>
> I downloaded the latest version, did a "touch .devel", and then edited the ArgusSource.c file and applied the patch. I ran ./configure and then make, but the argus binary doesn't appear in the bin directory.
>
> I see this error when I run make:
>
> In file included from ./ArgusModeler.h:330,
> from ./argus.h:40,
> from ArgusSource.c:67:
> ./ArgusSource.h:893: error: âArgusIpNetPacketâ undeclared here (not in a function)
>
> Thx.
>
> Craig
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Thursday, July 18, 2013 9:58 PM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
>
> Grab argus-3.0.7.3 from here
> http://qosient.com/argus/dev/argus-latest.tar.gz
>
> Still need to apply the patch.
> Carter
>
> On Jul 19, 2013, at 12:19 AM, Craig Merchant <cmerchant at responsys.com> wrote:
>
>> We're running 3.0.7.2.
>>
>> I'll give the patch a try tomorrow and let you know what change (if anything).
>>
>> Thanks!
>>
>> Craig
>>
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Thursday, July 18, 2013 7:44 PM
>> To: Craig Merchant
>> Cc: Argus (argus-info at lists.andrew.cmu.edu)
>> Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
>>
>> Hey Craig,
>> OK, so I went through this complete thread, and I apologize for
>> the rudimentary question, but...
>>
>> What version of argus are you running ???
>>
>> We identified that one of the nanosleep() calls in the packet ingest
>> engine was a little long in earlier emails, and we took the call
>> out. It is possible that the other nanosleep()s need adjustment,
>> or removal.
>>
>> Try this type of patch, so see if things get better. Your line numbers
>> may not match, as this is from a modified ArgusSource.c. The specific
>> line is in the routine ArgusGetPackets().
>>
>> ==== //depot/argus/argus/argus/ArgusSource.c#110 - /Volumes/Users/carter/argus/argus/argus/ArgusSource.c ====
>> 3816c3825
>> < struct timespec tsbuf = {0, 250000}, *ts = &tsbuf;
>> ---
>>> struct timespec tsbuf = {0, 2500}, *ts = &tsbuf;
>>
>> This nanosleep() is in the notselectable() branch of the basic packet engine,
>> so should be the one that your pf_ring() code is using.
>>
>> If there is benefit, and argus isn't eating an entire core, then even 250 maybe
>> a good number.
>>
>> Carter
>>
>> On Jul 18, 2013, at 8:43 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>>
>>> Just wanted to give you another data point...
>>>
>>> During a sample period, racluster found 448391 flows that contained 5,266,137 packets. It was unsure of the direction of about 60% of those flows. So if Argus missed both the SYN and SYNACK for those 60% because those packets were dropped, we should see around 538,069 dropped packets. Which would be a little over 10% of the total packet volume. Yet the interface is showing something like 0.1% packet drop.
>>>
>>> I recorded about 10m packets using tcpdump (tcpdump -i eth3 -w tcpdump.pcap). I tried to convert them to argus format by running: argus -r tcpdump.pcap -A -J -R -Z -w tcpdump.argus
>>>
>>> I got the following:
>>>
>>> *** glibc detected *** argus: double free or corruption (fasttop): 0x00000000025 bc610 ***
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6(+0x760e6)[0x7fa22635e0e6]
>>> argus[0x42b465]
>>> argus[0x41b5fc]
>>> argus[0x40458b]
>>> argus[0x4070f6]
>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fa226306cdd]
>>> argus[0x403bd9]
>>> ======= Memory map: ========
>>> 00400000-00461000 r-xp 00000000 fd:00 71223417 /usr/lo cal/sbin/argus
>>> 00660000-00664000 rw-p 00060000 fd:00 71223417 /usr/lo cal/sbin/argus
>>> 00664000-0066a000 rw-p 00000000 00:00 0
>>> 025bc000-025dd000 rw-p 00000000 00:00 0 [heap]
>>> 7fa225a98000-7fa225aae000 r-xp 00000000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
>>> 7fa225aae000-7fa225cad000 ---p 00016000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
>>> 7fa225cad000-7fa225cae000 rw-p 00015000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
>>> 7fa225cb5000-7fa2260c0000 rw-p 00000000 00:00 0
>>> 7fa2260c0000-7fa2260e3000 r-xp 00000000 fd:00 9388269 /opt/rb /lib/libpfring.so
>>> 7fa2260e3000-7fa2262e2000 ---p 00023000 fd:00 9388269 /opt/rb /lib/libpfring.so
>>> 7fa2262e2000-7fa2262e4000 rw-p 00022000 fd:00 9388269 /opt/rb /lib/libpfring.so
>>> 7fa2262e8000-7fa226472000 r-xp 00000000 fd:00 78233613 /lib64/ libc-2.12.so
>>> 7fa226472000-7fa226671000 ---p 0018a000 fd:00 78233613 /lib64/ libc-2.12.so
>>> 7fa226671000-7fa226675000 r--p 00189000 fd:00 78233613 /lib64/ libc-2.12.so
>>> 7fa226675000-7fa226676000 rw-p 0018d000 fd:00 78233613 /lib64/ libc-2.12.so
>>> 7fa226676000-7fa22667b000 rw-p 00000000 00:00 0
>>> 7fa226680000-7fa226703000 r-xp 00000000 fd:00 78233621 /lib64/ libm-2.12.so
>>> 7fa226703000-7fa226902000 ---p 00083000 fd:00 78233621 /lib64/ libm-2.12.so
>>> 7fa226902000-7fa226903000 r--p 00082000 fd:00 78233621 /lib64/ libm-2.12.so
>>> 7fa226903000-7fa226904000 rw-p 00083000 fd:00 78233621 /lib64/ libm-2.12.so
>>> 7fa226908000-7fa22691f000 r-xp 00000000 fd:00 78233637 /lib64/ libpthread-2.12.so
>>> 7fa22691f000-7fa226b1f000 ---p 00017000 fd:00 78233637 /lib64/ libpthread-2.12.so
>>> 7fa226b1f000-7fa226b20000 r--p 00017000 fd:00 78233637 /lib64/ libpthread-2.12.so
>>> 7fa226b20000-7fa226b21000 rw-p 00018000 fd:00 78233637 /lib64/ libpthread-2.12.so
>>> 7fa226b21000-7fa226b25000 rw-p 00000000 00:00 0
>>> 7fa226b28000-7fa226b5f000 r-xp 00000000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
>>> 7fa226b5f000-7fa226d5f000 ---p 00037000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
>>> 7fa226d5f000-7fa226d61000 rw-p 00037000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
>>> 7fa226d61000-7fa226d62000 rw-p 00000000 00:00 0
>>> 7fa226d68000-7fa226d88000 r-xp 00000000 fd:00 78233603 /lib64/ ld-2.12.so
>>> 7fa226efd000-7fa226f80000 rw-p 00000000 00:00 0
>>> 7fa226f85000-7fa226f87000 rw-p 00000000 00:00 0
>>> 7fa226f87000-7fa226f88000 r--p 0001f000 fd:00 78233603 /lib64/ ld-2.12.so
>>> 7fa226f88000-7fa226f89000 rw-p 00020000 fd:00 78233603 /lib64/ ld-2.12.so
>>> 7fa226f89000-7fa226f8b000 rw-p 00000000 00:00 0
>>> 7fa226f8b000-7fa226f8d000 rw-p 00000000 00:00 0
>>> 7fff11d0c000-7fff11d21000 rw-p 00000000 00:00 0 [stack]
>>> 7fff11d70000-7fff11d71000 r-xp 00000000 00:00 0 [vdso]
>>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsysca ll]
>>> Aborted
>>>
>>> I've got no idea what that means...
>>>
>>> Am I following the right steps to convert the output of tcpdump into something ra clients can read?
>>>
>>> Thanks.
>>>
>>> Craig
>>
>>
>
>
More information about the argus
mailing list