[ARGUS] sorting larger logs
thx+argus at medic.chalmers.se
Tue Mar 23 05:14:32 EST 2004
I need to merge and sort somewhat large logs from argus, but rasort is
a bit to general and needs lots of resources.
Background: I'm working with four logs from argus (from eight sources)
that I need to merge into one log. They are 30 - 150MB each (one hour
worth of capturing). This is from a backbone routed with OSPF so for
the logs to make any sense, they should be merged.
rasort -v -r log1 -r log2 -r log3 -r log4 -w mergedlog.tmp
ragator -r mergedlog.tmp -w mergedlog
(only using ragator with multiple logfiles gives me one merged log, but
not in order)
The problem is that rasort consumes way to much resources and from what
I can read from the sources this is because it is implemented for
general sorting on pretty much any values (and stores everything in
memory), but in my case I want to merge four already sorted logs which
is pretty much trivial, but a special case for rasort. rasort seemed to
allocate several times more memory than the total size of my logs...
My question is:
How do other others do it? I'm sure I'm not the only one out there with
this problem. If there is no such tool (or option that have missed),
I'm considering writing it myself.
I have also considered shortening the interval, but I would rather not.
More information about the argus