[ARGUS] sorting larger logs
carter at qosient.com
Tue Mar 23 18:51:41 EST 2004
Most people avoid this problem by keeping their
files small in size.
Yes, a smarter sorter will be the answer for sorting
very large files. The argus client library has support
for multiple passes, and I suspect that first parsing
the files to see that they are sorted, or just to see
how much data needs to be buffered in order to accomplish
the sort in partially sorted files, will take care of
the resources problem, and then opening multiple files
and interleaving the reads makes sense.
Let me get the current release out, and then we can
investigate a possible solution.
From: owner-argus-info at lists.andrew.cmu.edu
[mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of Thorbjörn
Sent: Tuesday, March 23, 2004 5:15 AM
To: argus-info at lists.andrew.cmu.edu
Subject: [ARGUS] sorting larger logs
I need to merge and sort somewhat large logs from argus, but rasort is
a bit to general and needs lots of resources.
Background: I'm working with four logs from argus (from eight sources)
that I need to merge into one log. They are 30 - 150MB each (one hour
worth of capturing). This is from a backbone routed with OSPF so for
the logs to make any sense, they should be merged.
rasort -v -r log1 -r log2 -r log3 -r log4 -w mergedlog.tmp
ragator -r mergedlog.tmp -w mergedlog
(only using ragator with multiple logfiles gives me one merged log, but
not in order)
The problem is that rasort consumes way to much resources and from what
I can read from the sources this is because it is implemented for
general sorting on pretty much any values (and stores everything in
memory), but in my case I want to merge four already sorted logs which
is pretty much trivial, but a special case for rasort. rasort seemed to
allocate several times more memory than the total size of my logs...
My question is:
How do other others do it? I'm sure I'm not the only one out there with
this problem. If there is no such tool (or option that have missed),
I'm considering writing it myself.
I have also considered shortening the interval, but I would rather not.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 2934 bytes
Desc: not available
More information about the argus