[ARGUS] sorting larger logs

[Carter Bullard]

Hey Thorbjörn,
   Most people avoid this problem by keeping their
files small in size.

   Yes, a smarter sorter will be the answer for sorting
very large files.  The argus client library has support
for multiple passes, and I suspect that first parsing
the files to see that they are sorted, or just to see
how much data needs to be buffered in order to accomplish
the sort in partially sorted files, will take care of
the resources problem, and then opening multiple files
and interleaving the reads makes sense.

   Let me get the current release out, and then we can
investigate a possible solution.

Carter



-----Original Message-----
From: owner-argus-info at lists.andrew.cmu.edu
[mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of Thorbjörn
Axelsson
Sent: Tuesday, March 23, 2004 5:15 AM
To: argus-info at lists.andrew.cmu.edu
Subject: [ARGUS] sorting larger logs

I need to merge and sort somewhat large logs from argus, but rasort is 
a bit to general and needs lots of resources.

Background: I'm working with four logs from argus (from eight sources) 
that I need to merge into one log. They are 30 - 150MB each (one hour 
worth of capturing). This is from a backbone routed with OSPF so for 
the logs to make any sense, they should be merged.

Current solution:

rasort -v -r log1 -r log2 -r log3 -r log4 -w mergedlog.tmp
ragator -r mergedlog.tmp -w mergedlog

(only using ragator with multiple logfiles gives me one merged log, but 
not in order)

The problem is that rasort consumes way to much resources and from what 
I can read from the sources this is because it is implemented for 
general sorting on pretty much any values (and stores everything in 
memory), but in my case I want to merge four already sorted logs which 
is pretty much trivial, but a special case for rasort. rasort seemed to 
allocate several times more memory than the total size of my logs...

My question is:

How do other others do it? I'm sure I'm not the only one out there with 
this problem. If there is no such tool (or option that have missed), 
I'm considering writing it myself.

I have also considered shortening the interval, but I would rather not.

regards,

/Thorbjörn

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 2934 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20040323/69eb7019/attachment.bin>