[ARGUS] sorting larger logs

[Peter Van Epp]

	While I thought that ragator should do this, common sense tells me its
unlikely unless ragator is going to sort the 4 files, since the clients 
typically read the input files in sequence. I suspect the best bet is to write 
a client program that takes the 4 (or n) input files and writes the oldest time 
stamp from any of the n input files to the output file which I think will do 
what you want without an unneeded in memory sort. I'm not aware of such a 
client currently (although someone else on list may have one).

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Tue, Mar 23, 2004 at 11:14:32AM +0100, Thorbjörn Axelsson wrote:
> I need to merge and sort somewhat large logs from argus, but rasort is 
> a bit to general and needs lots of resources.
> 
> Background: I'm working with four logs from argus (from eight sources) 
> that I need to merge into one log. They are 30 - 150MB each (one hour 
> worth of capturing). This is from a backbone routed with OSPF so for 
> the logs to make any sense, they should be merged.
> 
> Current solution:
> 
> rasort -v -r log1 -r log2 -r log3 -r log4 -w mergedlog.tmp
> ragator -r mergedlog.tmp -w mergedlog
> 
> (only using ragator with multiple logfiles gives me one merged log, but 
> not in order)
> 
> The problem is that rasort consumes way to much resources and from what 
> I can read from the sources this is because it is implemented for 
> general sorting on pretty much any values (and stores everything in 
> memory), but in my case I want to merge four already sorted logs which 
> is pretty much trivial, but a special case for rasort. rasort seemed to 
> allocate several times more memory than the total size of my logs...
> 
> My question is:
> 
> How do other others do it? I'm sure I'm not the only one out there with 
> this problem. If there is no such tool (or option that have missed), 
> I'm considering writing it myself.
> 
> I have also considered shortening the interval, but I would rather not.
> 
> regards,
> 
> /Thorbjörn
>