Collecting multiple types of information at once

[Martijn van Oosterhout]

Hoi,

We currently have a situation where we'd like to collect multiple bits
of information from a file at once, for example:

- total data
- top ten hosts by bandwidth
- top ten ports by bandwidth

Additionally, we would want to do all of these with different BPF
filters (for example, restricted to only one LAN).

Currently you can do all this with racluster and rasort but it
requires going through the data files multiple times. We have a
sort-of solution which involves one ra, lots of tee processes and
running many raclusters in parallel. But this is not scalable. I was
wonder if there was a more efficient way.

I've thought of some possibilities, like being able to tag streams
based on BPF. Basically, a filter that checks if a record matches a
BPF, if so it replicates it with a special marker. Then you'd just
include that marker in your racluster key and you get all your answers
at once.

Perhaps this is already possible?

Another possibility is an ra tool where you could embed something like
a lua script so you could write your own aggregations easier.

Any other ideas?
-- 
Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/