[ARGUS] sorting larger logs
Peter Van Epp
vanepp at sfu.ca
Tue Mar 23 11:35:36 EST 2004
While I thought that ragator should do this, common sense tells me its
unlikely unless ragator is going to sort the 4 files, since the clients
typically read the input files in sequence. I suspect the best bet is to write
a client program that takes the 4 (or n) input files and writes the oldest time
stamp from any of the n input files to the output file which I think will do
what you want without an unneeded in memory sort. I'm not aware of such a
client currently (although someone else on list may have one).
Peter Van Epp / Operations and Technical Support
Simon Fraser University, Burnaby, B.C. Canada
On Tue, Mar 23, 2004 at 11:14:32AM +0100, Thorbjörn Axelsson wrote:
> I need to merge and sort somewhat large logs from argus, but rasort is
> a bit to general and needs lots of resources.
> Background: I'm working with four logs from argus (from eight sources)
> that I need to merge into one log. They are 30 - 150MB each (one hour
> worth of capturing). This is from a backbone routed with OSPF so for
> the logs to make any sense, they should be merged.
> Current solution:
> rasort -v -r log1 -r log2 -r log3 -r log4 -w mergedlog.tmp
> ragator -r mergedlog.tmp -w mergedlog
> (only using ragator with multiple logfiles gives me one merged log, but
> not in order)
> The problem is that rasort consumes way to much resources and from what
> I can read from the sources this is because it is implemented for
> general sorting on pretty much any values (and stores everything in
> memory), but in my case I want to merge four already sorted logs which
> is pretty much trivial, but a special case for rasort. rasort seemed to
> allocate several times more memory than the total size of my logs...
> My question is:
> How do other others do it? I'm sure I'm not the only one out there with
> this problem. If there is no such tool (or option that have missed),
> I'm considering writing it myself.
> I have also considered shortening the interval, but I would rather not.
More information about the argus