Argus Database.

[Carter Bullard]

The only problem with putting argus data into a database,
in order to sort, is that scaling is an issue.  At some
argus sites, they are generating 5-10K records per second,
and mysql isn't going to keep up with that kind of
transaction load.

And when using bulk inserts, to get the best performance,
you need to pre-sort the data based on your key strategy
before you insert, so, ......, you don't really win that way.

I would be interested in doing the right thing on the
sort algorithm.

Carter



-----Original Message-----
From: owner-argus-info at lists.andrew.cmu.edu
[mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of David Nolan
Sent: Sunday, March 13, 2005 7:12 PM
To: argus-info at lists.andrew.cmu.edu
Subject: Re: [ARGUS] Argus Database.



--On Saturday, March 12, 2005 8:14 PM -0800 Peter Van Epp <vanepp at sfu.ca> 
wrote:

> I've come to the conclusion (which Russell suggested long ago :-)) that
> instead of fighting with memory exhaustion in perl scripts post
> processing  argus output its time to let mysql do it for me.

While this isn't a bad idea, I think you should try some simple approaches 
to solve the memory problem before going the whole way to using mysql. 
(And I'm a big mysql guy, so this isn't just mysql bashing.)

In particular, if you're storing lots of data in hashes, try tie'ing those 
hashes to files on disk, so they don't eat up your memory.  You may have to 
restructure your data format a bit to do this, if you're currently using 
nested hashes, but it may be worth the effort.  tie'ing to a file actually 
gets around some memory (mis)management problems with perl.  We've seen 
code that was running a machine out of memory with an in memory hash result 
in only a few megabtye file on disk when tie'd.

If you still want to go the database approach, I found this page in google 
that indicates that someone else may have already done a bunch of the work 
you're looking for:
<http://article.gmane.org/gmane.network.argus/2626>


-David

David Nolan                    <*>                    vitroth+ at cmu.edu
curses: May you be forced to grep the termcap of an unclean yacc while
      a herd of rogue emacs fsck your troff and vgrind your pathalias!