Graph of the Week at http://qosient.com/argus

[Richard Bejtlich]

On 9/15/06, carter at qosient.com <carter at qosient.com> wrote:
> Hey Richard, et al,
> Why doesn't squil eat Argus records yet ;o)
>

Hi Carter,

This is an issue we have debated.  Maybe if I explain our current
situation you can imagine a solution?

Currently we use SANCP (www.metre.net/sancp.html) in the following
manner.  SANCP watches traffic and writes results to files with text
data like the following:

1|4960894957268645250|2006-08-08 14:46:06|2006-08-08
14:46:16|10|6|1167053256|57239|1123635987|443|9|1469|10|2103|27|27
1|4960894957268571650|2006-08-08 14:46:06|2006-08-08
14:46:17|11|6|1167053256|57238|1123636051|443|11|5063|9|1940|27|27

These records are bi-directional and generally unique for each session.

A Sguil component (sensor_agent.tcl) periodically checks the directory
into which the SANCP records are written, reads the files, and then
inserts them into a MySQL database like the following:

mysql> describe sancp;
+------------+----------------------+------+-----+---------+-------+
| Field      | Type                 | Null | Key | Default | Extra |
+------------+----------------------+------+-----+---------+-------+
| sid        | int(10) unsigned     | NO   | MUL | NULL    |       |
| sancpid    | bigint(20) unsigned  | NO   |     | NULL    |       |
| start_time | datetime             | NO   | MUL | NULL    |       |
| end_time   | datetime             | NO   |     | NULL    |       |
| duration   | int(10) unsigned     | NO   |     | NULL    |       |
| ip_proto   | tinyint(3) unsigned  | NO   |     | NULL    |       |
| src_ip     | int(10) unsigned     | YES  | MUL | NULL    |       |
| src_port   | smallint(5) unsigned | YES  | MUL | NULL    |       |
| dst_ip     | int(10) unsigned     | YES  | MUL | NULL    |       |
| dst_port   | smallint(5) unsigned | YES  | MUL | NULL    |       |
| src_pkts   | int(10) unsigned     | NO   |     | NULL    |       |
| src_bytes  | int(10) unsigned     | NO   |     | NULL    |       |
| dst_pkts   | int(10) unsigned     | NO   |     | NULL    |       |
| dst_bytes  | int(10) unsigned     | NO   |     | NULL    |       |
| src_flags  | tinyint(3) unsigned  | NO   |     | NULL    |       |
| dst_flags  | tinyint(3) unsigned  | NO   |     | NULL    |       |
+------------+----------------------+------+-----+---------+-------+
16 rows in set (0.02 sec)

I guess we grapple with Argus for a few reasons.  One, support for
SANCP is built into Sguil.  We haven't built an API to accept other
data sources, although Bamm is considering it.  When an API is in
place (maybe Sguil 2.0?) we would aim for accepting Argus, NetFlow,
etc.

Two, we're not sure how best to accommodate Argus' record creation
model, where data is written to a non-text format with potentially
multiple records for the same session.  Do we let Argus write records,
run ra against them, output to a text file, and then parse the results
for insertion into the database?  Or do we avoid a db entirely and
have Sguil invoke ra against Argus records?

In any case we would appreciate insights on how best to accommodate
Argus with Sguil, since obviously several of us use Argus alongside
Sguil components already.

Thank you,

RIchard