Argus and Sguil

[Richard Bejtlich]

Hi guys,

Bamm asked me to post this -- he's subscribed to the list now but
somehow I agreed to be the messenger here.  :)

From: Bamm Visscher <bamm.visscher at gmail.com>
Date: Sep 19, 2006 9:57 AM
Subject: Re: [ARGUS] Graph of the Week at http://qosient.com/argus
To: carter at qosient.com
Cc: Richard Bejtlich <taosecurity at gmail.com>, Olaf Gellert
<olaf.gellert at intrusion-lab.net>, Argus
<argus-info at lists.andrew.cmu.edu>

I don't have a network where I can test SANCP and the 1 million plus
records/hour. About the most I have dealt w/personally is ~15
million/day. I don't think getting the data into the DB is the tough
part. MySQL (MyISAM) is pretty quick at loading it assuming your HW
can handle it (although I can understand concerns about this being
"safe").

Management of the DB with that many records can be a nightmare though.
At some point you are probably going to hit a HW or SW wall, whether
it be limited disk space or mysql performance and that is why we
started using MERGE tables. We've found the performance is acceptable
(I have users with over 1 billion </austinpowers> records in their
sancp tables) and management of the data is improved immensely. For
example, say you hit some HW or SW limit and needed to remove 30 days
or millions of records of data. With a non-MERGE DB, you would have to
perform a DELETE FROM foo WHERE bar (and maybe a CREATE TABLE foo ...
SELECT bar if you want to archive). The DELETE and CREATE...SELECT
might not be too bad but you will have to analyze and optimize the
table to gain anything and this can lock the table/DB for an
unexceptably long time.  With MERGE you can simple delete or move the
specific tables (sancp_sensor_20060101), restart mysql, and restart
sguild letting it quickly rebuild a new MERGE definition.

So, why do we want flow data in a DB anyway?  For starters, it gives
us a centralized location to query flow data from multiple sensors.
It's a simple query if I want to know if any systems from any of the
networks I monitor made a connection out to bad.ip.org.  We also get
the power of SQL (did I just right that) to do some data mining. Here
are some examples of stuff users are doing:

http://www.inliniac.net/blog/?p=24
http://infosecpotpourri.blogspot.com/2006/06/tracking-your-most-active-network.html
http://infosecpotpourri.blogspot.com/2006/05/traffic-analysis-approach-to-detecting.html

I should also point out that I am not sure how well SANCP will scale
in large bandwidth environments. A good friend of mine wrote it,
mainly out of my request, but I know he also uses it in his own work.
I cc'd him on this reply, maybe he has more insight.

Finally, I really don't have a lot of experience with Argus. I am
going to subscribe to the mailing lists after I send this out. I've
talked with Rich and others about how best to use/access Argus data
from the Sguil interface. We just haven't been able to come up with a
final solution. Do we simply provide a hook for the analyst to easily
run the ra client and grok data associated with a highlighted alert?
Do we take the time to parse the data and put it into a DB (on the
sensor or centralized)?   Do you have any comments on how you use
Argus in correlation with security events?

Final, finally ;)   I apologize if this email seems to bounce around, I
didn't want to wait to reply so I've been composing it in between
various other projects and my wife is convinced I don't multi-task
well.

Bammkkkk