Argus vs SiLK

[John Gerth]

On 4/16/10 1:37 PM, John Kennedy wrote:
> Is there someone that might have some experience with SiLK who could
> provide a brief toe to toe comparison? 
> 
 The description below is based entirely on my experience and so is extremely anecdotal.
 In order to be brief, I will also be frank and since I have far more practical
 experience with Argus than SiLK, I'm undoubtedly not "fair and balanced".

 At our site, I started out with Cisco netflow but have been running argus
 for about 5 years. For the last two years I've had contracts with
 DHS organizations which were using SiLK tools so I've had to learn
 a fair amount about the data although my actual experience with the
 tools is limited.

SilK is described well at the CMU SEI site:
  http://tools.netsa.cert.org/
especially the tutorial
  http://tools.netsa.cert.org/silk/analysis-handbook.pdf

SEI also runs "FloCon" which normally has several papers
with a SiLK focus so you'll want to poke around at:
  http://www.cert.org/flocon/
as they do post all the presentations.

Formally, SiLK is a suite of applications like the argus clients
although it does separate itself into a set of 'packing tools' which
accept IPFIX or netflow records and then store those in the
SiLK repositories as a prescursor to feeding them into the
'analysis tools'. They've invested a fair amount in doing aggressive
data compression.

As a practical matter, both argus and SiLK have their own flow
sensors. In the case of SiLK, it's YAF which creates unidirectional
IPFIX flow records very similar to netflow whereas the argus daemon
normally creates bi-directional flow records.  argus (via radium)
also provides very nice realtime sockets that remote applications
can hook up to and most argus applications run inside a wrapper
which can hook up to data from files or sockets

uni vs. bi directionality can be somewhat of a religious issue
in the flow community.  The big argument for uni-directional
is that it's typical in large sites for traffic to be
asymmetrically routed which makes it difficult, perhaps
impossible, to have a sensor which can generate bi-directional
records.  Furthermore, the typical bi-directional record has
only one set of timestamps so there's an information loss
compared to uni  However, argus 3 allows for separate sets of
timestamps in each direction and also can generate uni-directional flows.

Personally, my work is in network security and I find that bi-flows
are essential.  When I've had to use uni-flows, I first turn them
into bi-flows before going further.

In general, though, the basic data in flows is pretty much the same;
the 5-tuple key, counts of packets, bytes, a few ancilliary fields.
For TCP, argus records the logical or of the flags by each end.
YAF does that and has a separate field for the initial flags which
is *really* important for uni-flows if you're going figure how which
IP is the client and which is the server. Both have a way to decorate
flows with country-codes.

Beyond the basic data, things diverge a bit. YAF has provision for
a flowtype and application fields which are only partially configurable
by the user. Historically, argus has had a lot more fields for
assessing QOS issues like loss and jitter.  argus3 has a very flexible
flow labeling system which looks to have a promising future.

For both argus and SiLK the main application tools I've used are
the commands to do filtered extractions from the flows.  The
argus application wrapper gives you this all for free and so
I've written a number of my own apps inside the wrapper when I need
to have high efficiency, but "ra" is the exemplar.  For SiLK one
generally uses "rwfilter" at the head of an analysis pipeline
and if that is piped into "rwcut" you can do many of the
things that "ra" does. AFAIK, writing tools to process the SiLK
repository data is not encouraged as they don't provide anything
nearly as convenient as the argus client libraries and I've
not found much documentation on the SiLK data structures, but they
do distribute all the source.

I have only limited experience with the more advanced clustering
and aggregating appplications in either system. I do mostly forensic
work so the devil is in the details.  Generally I load flows into a SQL
database and build apps which run off that. (argus 3 now has support for sql).

Leveraging the argus socket support, I have done some minor situational
awarenees work. Our peak flow volumes are only a couple of thousand/sec
so we can insert them into SQL in realtime. I have also used ra, ratop
and other apps in instances of "hot pursuit". I've never seen a
claim that SiLK will do realtime analysis, but I haven't
looked very hard either.

/John