Argus Database.

[Peter Van Epp]

	Currently on my argi, once an hour when archiving takes place perl 
scripts also run that post process the data for the last hour. At 6 AM that
also fires off a summary process that looks over the last 24 hours for 
interesting things. My thought is that the hourly perl (for now) process will
post process to a mysql bulk load format and bulk load the data in to mysql.
Then lots of things become possible / easy such as processing the last 24 hours
every hour (by dropping the oldest hour and adding the new hour). In theory
since mysql is using disk for its files I won't run in to the memory problems
that perl does (especially during port scans) because if things get too large
it will be possible to stop, do the bulk load to mysql, delete the memory and
continue processing.

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Mon, Mar 14, 2005 at 01:42:05AM +1100, Chris Keladis wrote:
> Hi Peter,
> 
> Thinking about it some more, there are a few other ways to do this.
> 
> What may be more efficient (although less current) could be to load 
> Argus data into the DB as chunks.
> 
> Say, running ragator to compress the flows, then grabbing your 
> interested time-window of data, CSV-ifying (or TSV) it, and using 
> something like MySQLs 'mysqlimport' on the Argus host to bulk-load the 
> data into the DB.
> 
> mysqlimport also supports network-compression (assuming the server 
> supports it) which may give a performance boost as well, given that the 
> information is all text.
> 
> Loads could be done at whatever frequency is necessary, perhaps ra's 
> '-t' option would come in handy to feed you the dataset of the time your 
> interested in, to save extra processing costs.
> 
> And (as most have noted in this thread) if we are only taking the output 
> of 'ra' and not the full Argus flow record, it would be possible to 
> store quite a bit of data.
> 
> Just thinking out aloud. Thoughts, comments?
> 
> 
> 
> 
> Regards,
> 
> Chris.
> 
> Peter Van Epp wrote:
> 
> >	I've come to the conclusion (which Russell suggested long ago :-)) 
> >	that
> >instead of fighting with memory exhaustion in perl scripts post processing 
> >argus output its time to let mysql do it for me. So far thats gotten as 
> >far as
> >me actually installing mysql on one of my test machines and no further. Its
> >my intent to start with the ra fields my scripts are currently using (which
> >are a subset) and see what happens but I expect progress to be slow. As 
> >noted
> >the start will be ra -> perl -> mysql but assuming I find something that 
> >works
> >the correct answer would be a new r client that writes directly to mysql. 
> >The
> >performace issues can probably be most easily beaten by being able to split
> >argus streams across multiple boxes (I don't currently have enough volume 
> >to need to do that, but the scripts as they stand can do it). Then 
> >assuming that
> >you are doing once a day summarization as I am you can combine the outputs
> >of multiple merged streams again in to a single output database.
> >
> >Peter Van Epp / Operations and Technical Support 
> >Simon Fraser University, Burnaby, B.C. Canada
> >
> >On Sat, Mar 12, 2005 at 05:07:05PM +1100, Chris Keladis wrote:
> >
> >>Hi all,
> >>
> >>I know this topic has come up before, but i was wondering how work was 
> >>going in adding database support for Argus output?
> >>
> >>I've played around with raxml and managed to use a python script to 
> >>create a MySQL schema from the XML DTD (although it is very inefficient, 
> >>it's got the basic structure).
> >>
> >>I was thinking about performance with database output and have been 
> >>thinking it might be best to use the same method Snort (IDS) uses to 
> >>support high-speed monitoring, with database output.
> >>
> >>Snort employs a high-speed outfile format called unified output, which 
> >>is read by a post-processor, and using checkpoints, writes the data into 
> >>the RDBMS, leaving Snort free to handle the task of performing IDS.
> >>
> >>Perhaps a similar tool would be useful with Argus?
> >>
> >>Would appreciate your thoughts.
> >>
> >>
> >>
> >>
> >>
> >>Regards,
> >>
> >>Chris.