[flow-tools] performance question

Fri, 17 Jan 2003 14:09:45 -0600 (CST)

We have installed a netflow collection system and I have developed a
set of scripts using flow-tools to analyze the collected data.  So
far, so good.

The problem is, it currently takes about 3 days to analyze each days'
data (on a 900 MHz top-of-the-line Sparc-something-impressive).

To give you an idea of the amount of data being analyzed, we are
collecting netflow data on 18 routers, the data amoonts to about 9
GBytes (uncompressed), and we are producing reports for over 500
customers.

After some analysis, I have determined that:

1) about 25% of the total wall clock time is spend in flow-tag, which
   is tagging the flows.  During this time, the CPU is over 90% busy.
   A typical tag file snippet is:

	tag-action Cust-Src-4
	  type src-prefix
	  match 1.2.3.4/28	set-src	4

	tag-action Cust-Dst-4
	  type dst-prefix
	  match 1.2.3.4/28	set-dst	16384

	[ repeat the above for each customer: i.e., over 500 times ]

	tag-action Inet-Src
	  type src-prefix
	  match 1.2.3.4/18	set-src	0
	  match 1.2.3.4/16	set-src	0
	  match 1.2.3.4/16	set-src	0
	  match 1.2.3.4/16	set-src	0
	  match 1.2.3.4/16	set-src	0
	  match 1.2.3.4/16	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/17	set-src	0
	  match 1.2.3.4/20	set-src	0
	  match 1.2.3.4/20	set-src	0
	  match 1.2.3.4/20	set-src	0
	  match 1.2.3.4/20	set-src	0
	  match 1.2.3.4/20	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/23	set-src	0
	  match 1.2.3.4/19	set-src	0
	  match 1.2.3.4/19	set-src	0
	  match 1.2.3.4/18	set-src	0
	  match 1.2.3.4/21	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/24	set-src	0
	  match 1.2.3.4/21	set-src	0
	  match 1.2.3.4/22	set-src	0
	  match 1.2.3.4/19	set-src	0
	  match 1.2.3.4/18	set-src	0
	  match 0/0	set-src	0x01000000

	tag-action Inet-Dst
	  type dst-prefix
	  match 1.2.3.4/18	set-dst	0
	  match 1.2.3.4/16	set-dst	0
	  match 1.2.3.4/16	set-dst	0
	  match 1.2.3.4/16	set-dst	0
	  match 1.2.3.4/16	set-dst	0
	  match 1.2.3.4/16	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/17	set-dst	0
	  match 1.2.3.4/20	set-dst	0
	  match 1.2.3.4/20	set-dst	0
	  match 1.2.3.4/20	set-dst	0
	  match 1.2.3.4/20	set-dst	0
	  match 1.2.3.4/20	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/23	set-dst	0
	  match 1.2.3.4/19	set-dst	0
	  match 1.2.3.4/19	set-dst	0
	  match 1.2.3.4/18	set-dst	0
	  match 1.2.3.4/21	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/24	set-dst	0
	  match 1.2.3.4/21	set-dst	0
	  match 1.2.3.4/22	set-dst	0
	  match 1.2.3.4/19	set-dst	0
	  match 1.2.3.4/18	set-dst	0
	  match 0/0	set-dst	0x02000000

	tag-definition main
	  term
	  action Inet-Src
	  action Inet-Dst
	  action Cust-Src-1
	  action Cust-Dst-1
		...
	[ again repeat the last two over 500 times ]

2) Over 50% of the total wall clock time is spent in flow-nfilter.
This step involves producing a flow file for each customer that
contains only flows to or from that customer.  The source of data for
this step is the set of flow files that contains data from all parts
of the network merged together.  The CPU is about 40% busy during this
step.  A typical filter file is:

	filter-primitive any-source-tag
	  type tag-mask
	  permit 0x00000000 0x00000ff8
	  default deny

	filter-primitive any-dest-tag
	  type tag-mask
	  permit 0x00000000 0x00ff8000
	  default deny

	filter-definition default
	  match source-tag any-source-tag
	  or
	  match destination-tag any-dest-tag

I use uncompressed flow files, because it was taking large amounts of
time to perform the compression.  For example, a flow-nfilter run might
take 10 minutes when writing uncompressed but nearly 90 minutes if run
with -z 9.

Obviously, I need to get the wall clock time down substantially.  I
am considering the following:

A) Are there better ways to organize the tags file so that flow-tag
runs faster (say, 3-5x)?

B) Given my specialized needs, would it make sense to simply rewrite
(=adapt) flow-tag to optimize it for my situation?]

If so, can someone point me to any documentation on the flow file format?

C) Anyone have any other ideas?

Thanks,

Craig A. Finseth                        craig@firwood.net
Firwood Consulting, Inc.                craigafinseth@alum.mit.edu
1343 Lafond, St Paul MN 55104           +1 651 644 4027
USA             http://www.firwood.net  +1 651 644 4027 fax (yes, same number)
A ship is safe in a harbor, but that's not what a ship is for--Adm Grace Hopper