[flow-tools] performance question [resend]

[Clayton Fiske]

On Fri, Jan 24, 2003 at 10:14:36AM -0600, Craig A. Finseth wrote:
> As I mentioned in an earlier message (which may have yet to wend its way
> through the queues...), this problem has been fixed by writing tailored
> code for flow-tag.
> 
> Basically, we are collecting netflow data on about 20 routers.  This
> data is in:
> 
> 	/netflow/<router name>/<flow files at 15 minute intervals>
> 
> (I'm simplifying and paraphrasing, but the gist will be correct.)
> 
> The data is copied to
> 
> 	/netflow/filtered/<router name>/<flow files at 15 minute intervals>
> 
> by passing it through a chain of three commands:
> 
> 	flow-nfilter	-- remove duplicate flows
> 	flow-tag[*]	-- add tags for customer and Internet data
> 	flow-nfilter	-- remove all non-tagged flows
> 
> The data is then merged into:
> 
> 	/netflow/merged/<flow files at 15 minute intervals>
> 
> so that there is one set of files for the whole system.
> 
> [*] This is the step that originally took about 25% of the overall
> time and that I have sped up by a factor of 20 or so.  Time is now not
> an issue.

Am I correct in assuming you use flow-capture to generate the flow
files from the router data? Have you tried using flow-capture to do
the tagging (-t and -T flags) to see if it makes a performance difference?

-c