Direction and IP/TCP timeout settings
Carter Bullard
carter at qosient.com
Sat Jul 20 08:29:51 EDT 2013
why not take some small steps at solving the problem you have.
reset all the timeouts to what we think they should be, which is
around 60 for ip and 120 seconds for tcp.
why are you running two instances? run one.
some people have big problems with pf_ring. the word hate is
used on occasion. a few are using netmap.
argus is polling pf_ring, so it will use constant CPU just to get packets
to process. that is the nature of pf_ring. pf_ring does have problems
when there are lots of competition for processing a single packet.
run argus without the snorts going to see if things get better.
when you say that argus crashes, you should send some form of
bug report, or you're just trolling the list. compile with devel
support (touch .devel;./configure;make) and run under gdb, but
not in daemon mode. when argus fails, minimally type
(gdb) where
Carter
On Jul 19, 2013, at 8:04 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> So, I’ve got one argus instance running on the standard intel ixgbe driver. The other instance is running on the DNA-aware ixgbe driver from pf_ring. They have a two hour timeout for tcp connections and a one hour timeout for IP.
>
> Both of the instances are running between 50-100% CPU all the time. The good news is that the directionality problems have dropped from around 70% of traffic to about 25%. The bad news is that argus went from seeing around 100K flows per minute to about 9K, so I think it’s dropping flows for some reason. I also tried running it on the DNA/libzero interface (pfdnacluster_master), but it crashed.
>
> Thoughts…?
>
> Thx.
>
> Craig
>
> From: argus-info-bounces+cmerchant=responsys.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+cmerchant=responsys.com at lists.andrew.cmu.edu] On Behalf Of Craig Merchant
> Sent: Friday, July 19, 2013 3:32 PM
> To: Carter Bullard
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
>
> I added this patch at line 2177 of ArgusSource.c and tried make again (after doing a make clean), but still got the same error.
>
> Here is what that section of ArgusSource.c looks like now:
>
> #ifdef ARGUSDEBUG
> ArgusDebug (8, "ArgusJuniperPacket (%p, %p, %p) returning\n", user, h, p);
> #endif
> }
>
> void
> ArgusIpNetPacket (u_char *user, const struct pcap_pkthdr *h, const u_char *p)
> {
>
> #ifdef ARGUSDEBUG
> ArgusDebug (8, "ArgusIpNetPacket (%p, %p, %p) returning\n", user, h, p);
> #endif
> }
>
> int
> ip_heuristic_guess(register const u_char *p, u_int length)
>
> The error message from make:
>
> gcc -g -Wall -Wmissing-prototypes -I. -I/opt/rb/include -I./../include -DHAVE_CONFIG_H -c ArgusSource.c
> In file included from ./ArgusModeler.h:330,
> from ./argus.h:40,
> from ArgusSource.c:67:
> ./ArgusSource.h:893: error: âArgusIpNetPacketâ undeclared here (not in a function)
> ArgusSource.c:2179: warning: no previous prototype for âArgusIpNetPacketâ
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Friday, July 19, 2013 12:25 PM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
>
> Sorry, add this snippet of code.
> Carter
>
> ==== //depot/argus/argus/argus/ArgusSource.c#110 - /Volumes/Users/carter/argus/argus/argus/ArgusSource.c ====
> 2177a2178,2186
> > void
> > ArgusIpNetPacket (u_char *user, const struct pcap_pkthdr *h, const u_char *p)
> > {
> >
> > #ifdef ARGUSDEBUG
> > ArgusDebug (8, "ArgusIpNetPacket (%p, %p, %p) returning\n", user, h, p);
> > #endif
> > }
> >
>
>
>
> On Jul 19, 2013, at 3:12 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>
> > Hey, Carter...
> >
> > I downloaded the latest version, did a "touch .devel", and then edited the ArgusSource.c file and applied the patch. I ran ./configure and then make, but the argus binary doesn't appear in the bin directory.
> >
> > I see this error when I run make:
> >
> > In file included from ./ArgusModeler.h:330,
> > from ./argus.h:40,
> > from ArgusSource.c:67:
> > ./ArgusSource.h:893: error: âArgusIpNetPacketâ undeclared here (not in a function)
> >
> > Thx.
> >
> > Craig
> >
> > -----Original Message-----
> > From: Carter Bullard [mailto:carter at qosient.com]
> > Sent: Thursday, July 18, 2013 9:58 PM
> > To: Craig Merchant
> > Cc: Argus (argus-info at lists.andrew.cmu.edu)
> > Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
> >
> > Grab argus-3.0.7.3 from here
> > http://qosient.com/argus/dev/argus-latest.tar.gz
> >
> > Still need to apply the patch.
> > Carter
> >
> > On Jul 19, 2013, at 12:19 AM, Craig Merchant <cmerchant at responsys.com> wrote:
> >
> >> We're running 3.0.7.2.
> >>
> >> I'll give the patch a try tomorrow and let you know what change (if anything).
> >>
> >> Thanks!
> >>
> >> Craig
> >>
> >> -----Original Message-----
> >> From: Carter Bullard [mailto:carter at qosient.com]
> >> Sent: Thursday, July 18, 2013 7:44 PM
> >> To: Craig Merchant
> >> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> >> Subject: Re: [ARGUS] Direction and IP/TCP timeout settings
> >>
> >> Hey Craig,
> >> OK, so I went through this complete thread, and I apologize for
> >> the rudimentary question, but...
> >>
> >> What version of argus are you running ???
> >>
> >> We identified that one of the nanosleep() calls in the packet ingest
> >> engine was a little long in earlier emails, and we took the call
> >> out. It is possible that the other nanosleep()s need adjustment,
> >> or removal.
> >>
> >> Try this type of patch, so see if things get better. Your line numbers
> >> may not match, as this is from a modified ArgusSource.c. The specific
> >> line is in the routine ArgusGetPackets().
> >>
> >> ==== //depot/argus/argus/argus/ArgusSource.c#110 - /Volumes/Users/carter/argus/argus/argus/ArgusSource.c ====
> >> 3816c3825
> >> < struct timespec tsbuf = {0, 250000}, *ts = &tsbuf;
> >> ---
> >>> struct timespec tsbuf = {0, 2500}, *ts = &tsbuf;
> >>
> >> This nanosleep() is in the notselectable() branch of the basic packet engine,
> >> so should be the one that your pf_ring() code is using.
> >>
> >> If there is benefit, and argus isn't eating an entire core, then even 250 maybe
> >> a good number.
> >>
> >> Carter
> >>
> >> On Jul 18, 2013, at 8:43 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> >>
> >>> Just wanted to give you another data point...
> >>>
> >>> During a sample period, racluster found 448391 flows that contained 5,266,137 packets. It was unsure of the direction of about 60% of those flows. So if Argus missed both the SYN and SYNACK for those 60% because those packets were dropped, we should see around 538,069 dropped packets. Which would be a little over 10% of the total packet volume. Yet the interface is showing something like 0.1% packet drop.
> >>>
> >>> I recorded about 10m packets using tcpdump (tcpdump -i eth3 -w tcpdump.pcap). I tried to convert them to argus format by running: argus -r tcpdump.pcap -A -J -R -Z -w tcpdump.argus
> >>>
> >>> I got the following:
> >>>
> >>> *** glibc detected *** argus: double free or corruption (fasttop): 0x00000000025 bc610 ***
> >>> ======= Backtrace: =========
> >>> /lib64/libc.so.6(+0x760e6)[0x7fa22635e0e6]
> >>> argus[0x42b465]
> >>> argus[0x41b5fc]
> >>> argus[0x40458b]
> >>> argus[0x4070f6]
> >>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fa226306cdd]
> >>> argus[0x403bd9]
> >>> ======= Memory map: ========
> >>> 00400000-00461000 r-xp 00000000 fd:00 71223417 /usr/lo cal/sbin/argus
> >>> 00660000-00664000 rw-p 00060000 fd:00 71223417 /usr/lo cal/sbin/argus
> >>> 00664000-0066a000 rw-p 00000000 00:00 0
> >>> 025bc000-025dd000 rw-p 00000000 00:00 0 [heap]
> >>> 7fa225a98000-7fa225aae000 r-xp 00000000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
> >>> 7fa225aae000-7fa225cad000 ---p 00016000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
> >>> 7fa225cad000-7fa225cae000 rw-p 00015000 fd:00 78233953 /lib64/ libgcc_s-4.4.7-20120601.so.1
> >>> 7fa225cb5000-7fa2260c0000 rw-p 00000000 00:00 0
> >>> 7fa2260c0000-7fa2260e3000 r-xp 00000000 fd:00 9388269 /opt/rb /lib/libpfring.so
> >>> 7fa2260e3000-7fa2262e2000 ---p 00023000 fd:00 9388269 /opt/rb /lib/libpfring.so
> >>> 7fa2262e2000-7fa2262e4000 rw-p 00022000 fd:00 9388269 /opt/rb /lib/libpfring.so
> >>> 7fa2262e8000-7fa226472000 r-xp 00000000 fd:00 78233613 /lib64/ libc-2.12.so
> >>> 7fa226472000-7fa226671000 ---p 0018a000 fd:00 78233613 /lib64/ libc-2.12.so
> >>> 7fa226671000-7fa226675000 r--p 00189000 fd:00 78233613 /lib64/ libc-2.12.so
> >>> 7fa226675000-7fa226676000 rw-p 0018d000 fd:00 78233613 /lib64/ libc-2.12.so
> >>> 7fa226676000-7fa22667b000 rw-p 00000000 00:00 0
> >>> 7fa226680000-7fa226703000 r-xp 00000000 fd:00 78233621 /lib64/ libm-2.12.so
> >>> 7fa226703000-7fa226902000 ---p 00083000 fd:00 78233621 /lib64/ libm-2.12.so
> >>> 7fa226902000-7fa226903000 r--p 00082000 fd:00 78233621 /lib64/ libm-2.12.so
> >>> 7fa226903000-7fa226904000 rw-p 00083000 fd:00 78233621 /lib64/ libm-2.12.so
> >>> 7fa226908000-7fa22691f000 r-xp 00000000 fd:00 78233637 /lib64/ libpthread-2.12.so
> >>> 7fa22691f000-7fa226b1f000 ---p 00017000 fd:00 78233637 /lib64/ libpthread-2.12.so
> >>> 7fa226b1f000-7fa226b20000 r--p 00017000 fd:00 78233637 /lib64/ libpthread-2.12.so
> >>> 7fa226b20000-7fa226b21000 rw-p 00018000 fd:00 78233637 /lib64/ libpthread-2.12.so
> >>> 7fa226b21000-7fa226b25000 rw-p 00000000 00:00 0
> >>> 7fa226b28000-7fa226b5f000 r-xp 00000000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
> >>> 7fa226b5f000-7fa226d5f000 ---p 00037000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
> >>> 7fa226d5f000-7fa226d61000 rw-p 00037000 fd:00 9388267 /opt/rb /lib/libpcap.so.1.1.1
> >>> 7fa226d61000-7fa226d62000 rw-p 00000000 00:00 0
> >>> 7fa226d68000-7fa226d88000 r-xp 00000000 fd:00 78233603 /lib64/ ld-2.12.so
> >>> 7fa226efd000-7fa226f80000 rw-p 00000000 00:00 0
> >>> 7fa226f85000-7fa226f87000 rw-p 00000000 00:00 0
> >>> 7fa226f87000-7fa226f88000 r--p 0001f000 fd:00 78233603 /lib64/ ld-2.12.so
> >>> 7fa226f88000-7fa226f89000 rw-p 00020000 fd:00 78233603 /lib64/ ld-2.12.so
> >>> 7fa226f89000-7fa226f8b000 rw-p 00000000 00:00 0
> >>> 7fa226f8b000-7fa226f8d000 rw-p 00000000 00:00 0
> >>> 7fff11d0c000-7fff11d21000 rw-p 00000000 00:00 0 [stack]
> >>> 7fff11d70000-7fff11d71000 r-xp 00000000 00:00 0 [vdso]
> >>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsysca ll]
> >>> Aborted
> >>>
> >>> I've got no idea what that means...
> >>>
> >>> Am I following the right steps to convert the output of tcpdump into something ra clients can read?
> >>>
> >>> Thanks.
> >>>
> >>> Craig
> >>
> >>
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130720/fe31b729/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130720/fe31b729/attachment.bin>
More information about the argus
mailing list