Argus vs SiLK

[Carter Bullard]

Hey Chris,
This is not pointed at you, I'm taking the opportunity to get something off my chest,
and this seems like a good time.  No disrespect intended in any way.

The conversation regarding Argus and YAF+SiLK is not just a technical thread.  Both
Argus and YAF+SiLK were developed at CMU's CERT, just decades apart.

For those in the audience that may not know, Argus was first developed by me
in 1990, when I was the CERTs network security guy, and was used as a
standard security incident data handling tool for many years.  Argus was the
principal tool of the network vulnerability lab that we built at the CERT in
1990-1993, (a couple of machines a few routers, and a few really good
technical guys from the SEI, CMU and the CERT) where we discovered many
of the first real network vulnerabilities, like source routing, fragmentation
offset, and routing protocol vulnerabilities.   After I left the CERT, the use of
Argus  waned, as you would expect.  But for many years Argus was supported
and recommended by the CERT as a forensics tool, and mentioned in many CERT
advisories, such as the first DDoS attack advisories.  Many years later, what around
2006, Brian Trammell started YAF,  curiously mentioning argus in his initial
developers notes.  During those early years, YAF borrowed much from argus.  
I was in direct communication with Brian, off and on, and we talked frankly about
the CERT's IPFIX strategy.  I stressed to him that the file format was more
critical to the community than the transport protocol, and suggested strategies
for how the CERT could get that accomplished.  Brian has moved on, and new
groups have since been working flow monitor ideas at the CERT.  But its good to
see that the IPFIX file format is coming along!!!!

OK, enough history.  The key differences between Argus and YAF/SiLK are, I believe:

Argus  has more forensics capabilities than YAF/SiLK,  (we have more information),
Argus decodes more protocols, not just IP, Argus's flow tracking model is more
sophisticated, ( we are biflow all the way, which is extremely important especially
in asymmetric architectures ),  Argus handles many levels of encapsulation, it
performs faster than YAF/SiLK on most tasks, has been ported to more platforms, 
has integrated strong authentication and encryption support for flow data access, 
supports many transport strategies, both push and pull, Argus supports multicast
transport of flow records, which IPFIX is incapable of, Argus has a more extensive
data anonymization strategies, Argus has flexible record labeling, and geo and
net-spatial information support.  

Argus is being used at many more sites, has a very active global community and
a public mailing list talking about concepts in flow monitoring, that goes
back almost 10 years, and has been used in more academic research
investigations and referenced in more network security publications than
YAF/SiLK.......

Argus-3.0 has a database model, YAF/SiLK does not.

YAF has a better data reduction model than Argus.  Not clear that that is an
advantage, as systems based on the reduction have been reported as
poor performers for search etc....   SiLK is focused on the forensic's analyst, so
its interface concepts are much different than the ra* clients model, and there
are a lot of YAF/SiLK tutorials and technical documents, although I would score
the user documentation for both to be about the same, the API documentation
for SiLK is much better.

I think Argus is a good technology, one that the CERT should recognize and support.
Argus supports the CERT, with innovative flow monitoring concepts and proof of concept
implementations, an open development effort and a massive user base, compared
to YAF's user base.

I believe that the CERT, in its YAF/SiLK efforts and "total commitment to IPFIX", discredits
Argus in a lot of ways.  I have always presented Argus as a superset of IPFIX, with data
elements and transport concepts that IPFIX can't support.  Why doesn't the CERT
challenge IPFIX to handle these types of data, such as non-IP flows?

I think Argus is the only place that flow technology innovation is being done.
It really does need some support, and the CERT is a place that should be
supporting flow technology innovation.

Carter



On Jul 26, 2010, at 6:32 AM, Chris Inacio wrote:

> 
> 
> This is a follow up to a thread some 12 weeks ago.  (Yes, I'm generally running that far behind - actually, I had need of looking something else up and found the thread...)
> 
> First, the comparison is probably more appropriately, YAF+SiLK vs. Argus; but that's more a nit.
> 
> Biflow versus uniflow:  YAF is actually a biflow meter.  So YAF, in its default state will in fact export biflow IPFIX based records.  SiLK, however, is not biflow.  Part of that is historical, and part of that is related to the fact that the networks we usually deal with are extremely large and show a lot of asymmetry.  So in the end, John's statement that it is uniflow is at least half true.  That said, we have other applications where we store the output of YAF in something other than SiLK, in which case we do have biflows.
> 
> Flow labeling:  YAF is getting better in this area, but it has always had the ability for the user to add more, via PCRE based regular expressions.  That said, YAF 1.2 is just about to be released with a new set of DPI tools and more ability for users to expand this capability.  It does come with the caveat that a poorly written regex will kill the system performance.
> 
> YAF 1.2 (release any day now,) does have something similar to the Argus socket support based on the Spread toolkit / library.  (http://www.spread.org) (*1) I'll leave all the details as an exercise to the reader.  So admittedly, that is a very recent change - and somewhat against the IPFIX grain.  But that said, YAF has always supported the IPFIX mediator model which allows middlemen in between the meter and the collector.  We have some mediators built, but they are very specific to special needs, and we don't generally release that code.  There is a set of code we keep lying around to easily build new ones though.
> 
> As to writing apps directly on top of SiLK records, yes we support that.  It is, in fact, well documented in the SiLK documentation, and yes it does in fact drive us nuts that no one uses the SiLK C API to do that type of thing.  Using rwcut and going to ASCII is a total performance kill.  It is also a reason we built PySiLK, because then you can build quick extensions in Python that add to SiLK.  That capability is getting pretty advanced actually.
> 
> As to realtime analysis - if you're interested, stay tuned - something is coming.
> 
> 
> So, I can't really do a good job commenting on all the differences, because I don't know Argus.  I know Carter a bit though. :)
> 
> <really this isn't bait for Carter>
> As to, "why didn't CERT/SEI just use Argus?"  That's a much longer story, if you capture me in a corner sometime you can come ask me.  The shortest version is though, that we believe in open standards, and seeding the market in that regard - and so we believe in IPFIX.  So we built another flow meter that used the open standard.  
> </really this isn't bait for Carter>
> 
> 
> regards,
> Chris Inacio
> inacio at cert.org
> 
> (*1) Mostly based on my guess about the Argus socket "thing", because actually, I don't know too much about it.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100726/d1ebeca1/attachment.bin>