[maf@eng.oar.net: Re: [flow-tools] performance question]
Mark Fullmer
maf@eng.oar.net
Thu, 23 Jan 2003 18:19:06 -0500
----- Forwarded message from Mark Fullmer <maf@eng.oar.net> -----
Date: Wed, 22 Jan 2003 13:10:17 -0500
From: Mark Fullmer <maf@eng.oar.net>
To: "Craig A. Finseth" <fin@finseth.com>
Cc: flow-tools@splintered.net, craig@finseth.com
Subject: Re: [flow-tools] performance question
X-Mailer: Mutt 1.0i
On Fri, Jan 17, 2003 at 02:09:45PM -0600, Craig A. Finseth wrote:
> We have installed a netflow collection system and I have developed a
> set of scripts using flow-tools to analyze the collected data. So
> far, so good.
>
> The problem is, it currently takes about 3 days to analyze each days'
> data (on a 900 MHz top-of-the-line Sparc-something-impressive).
Could you give a few more details on what your scripts look like to
analyze the data? For example are you running all the data
through flow-tag and flow-nfilter 500 times?
One way to make this fast is tag the data once and use flow-split -g
or -G to create smaller datasets for each of the 500 customers. Then
run the various reports on each of the customer datasets.
Something else you should look at is the compile options. Flow-tools
by default is built with only -g. Replace this with -O or -O2 and
things will really speed up.
When using *-prefix in flow-tag the source/destination IP is looked
up in a patricia trie, that's probably where most of the CPU time
is going. There are faster data structures that could be implemented
(do a google search for "IP routing lookup"), but I have not had a need
to do so.
Also, your flow-tag file is not organized well. All the customers should
be listed in one tag-action, each with a different tag. The way you're
doing things now there are 500 trie's instead of one, on average 250
trie lookups are done per flow. You only need 1!
Hope this helps...
mark
>
> To give you an idea of the amount of data being analyzed, we are
> collecting netflow data on 18 routers, the data amoonts to about 9
> GBytes (uncompressed), and we are producing reports for over 500
> customers.
>
> After some analysis, I have determined that:
>
> 1) about 25% of the total wall clock time is spend in flow-tag, which
> is tagging the flows. During this time, the CPU is over 90% busy.
> A typical tag file snippet is:
>
> tag-action Cust-Src-4
> type src-prefix
> match 1.2.3.4/28 set-src 4
>
> tag-action Cust-Dst-4
> type dst-prefix
> match 1.2.3.4/28 set-dst 16384
>
> [ repeat the above for each customer: i.e., over 500 times ]
>
>
> tag-action Inet-Src
> type src-prefix
> match 1.2.3.4/18 set-src 0
> match 1.2.3.4/16 set-src 0
> match 1.2.3.4/16 set-src 0
> match 1.2.3.4/16 set-src 0
> match 1.2.3.4/16 set-src 0
> match 1.2.3.4/16 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/17 set-src 0
> match 1.2.3.4/20 set-src 0
> match 1.2.3.4/20 set-src 0
> match 1.2.3.4/20 set-src 0
> match 1.2.3.4/20 set-src 0
> match 1.2.3.4/20 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/23 set-src 0
> match 1.2.3.4/19 set-src 0
> match 1.2.3.4/19 set-src 0
> match 1.2.3.4/18 set-src 0
> match 1.2.3.4/21 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/24 set-src 0
> match 1.2.3.4/21 set-src 0
> match 1.2.3.4/22 set-src 0
> match 1.2.3.4/19 set-src 0
> match 1.2.3.4/18 set-src 0
> match 0/0 set-src 0x01000000
>
> tag-action Inet-Dst
> type dst-prefix
> match 1.2.3.4/18 set-dst 0
> match 1.2.3.4/16 set-dst 0
> match 1.2.3.4/16 set-dst 0
> match 1.2.3.4/16 set-dst 0
> match 1.2.3.4/16 set-dst 0
> match 1.2.3.4/16 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/17 set-dst 0
> match 1.2.3.4/20 set-dst 0
> match 1.2.3.4/20 set-dst 0
> match 1.2.3.4/20 set-dst 0
> match 1.2.3.4/20 set-dst 0
> match 1.2.3.4/20 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/23 set-dst 0
> match 1.2.3.4/19 set-dst 0
> match 1.2.3.4/19 set-dst 0
> match 1.2.3.4/18 set-dst 0
> match 1.2.3.4/21 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/24 set-dst 0
> match 1.2.3.4/21 set-dst 0
> match 1.2.3.4/22 set-dst 0
> match 1.2.3.4/19 set-dst 0
> match 1.2.3.4/18 set-dst 0
> match 0/0 set-dst 0x02000000
>
> tag-definition main
> term
> action Inet-Src
> action Inet-Dst
> action Cust-Src-1
> action Cust-Dst-1
> ...
> [ again repeat the last two over 500 times ]
>
> 2) Over 50% of the total wall clock time is spent in flow-nfilter.
> This step involves producing a flow file for each customer that
> contains only flows to or from that customer. The source of data for
> this step is the set of flow files that contains data from all parts
> of the network merged together. The CPU is about 40% busy during this
> step. A typical filter file is:
>
> filter-primitive any-source-tag
> type tag-mask
> permit 0x00000000 0x00000ff8
> default deny
>
> filter-primitive any-dest-tag
> type tag-mask
> permit 0x00000000 0x00ff8000
> default deny
>
> filter-definition default
> match source-tag any-source-tag
> or
> match destination-tag any-dest-tag
>
> I use uncompressed flow files, because it was taking large amounts of
> time to perform the compression. For example, a flow-nfilter run might
> take 10 minutes when writing uncompressed but nearly 90 minutes if run
> with -z 9.
>
> Obviously, I need to get the wall clock time down substantially. I
> am considering the following:
>
> A) Are there better ways to organize the tags file so that flow-tag
> runs faster (say, 3-5x)?
>
> B) Given my specialized needs, would it make sense to simply rewrite
> (=adapt) flow-tag to optimize it for my situation?]
>
> If so, can someone point me to any documentation on the flow file format?
>
> C) Anyone have any other ideas?
>
> Thanks,
>
> Craig A. Finseth craig@firwood.net
> Firwood Consulting, Inc. craigafinseth@alum.mit.edu
> 1343 Lafond, St Paul MN 55104 +1 651 644 4027
> USA http://www.firwood.net +1 651 644 4027 fax (yes, same number)
> A ship is safe in a harbor, but that's not what a ship is for--Adm Grace Hopper
>
>
> _______________________________________________
> flow-tools@splintered.net
> http://www.splintered.net/sw/flow-tools
----- End forwarded message -----