Fwd: Some useful Argus scripts for detecting traffic anomalies in ranges of data

[Phillip G Deneault]

Certainly an idea who's time has come...

I did a presentation on this kind of topic several years back, and 
actually wrote code for it!  However, there were lots of gotchas in the 
2.0 codebase that has since been fixed.  That combined with the 
inclusion of IPv6 requires me to perform a rewrite of the existing code, 
something that is on the back burner.

My presentation is here, if anyone is curious.
http://connect.educause.edu/Library/Abstract/MiningFlowsforIntrusionDa/43716

I was doing this using hourly snapshots, transfered via scp and crond, 
across multiple processing boxes dumping into a central database.  The 
whole thing was viewed via a very poorly written web front-end.  I 
eventually had to turn my attention to other issues, which required me 
to take down the flow processing and the system has been dormant since. :-(

Phil

Terry Burton wrote:
> Oops. Forgot to cc: the list...
> 
> ---------- Forwarded message ----------
> From: Terry Burton <tez at terryburton.co.uk>
> Date: Thu, Feb 21, 2008 at 5:00 PM
> Subject: Re: [ARGUS] Some useful Argus scripts for detecting traffic
> anomalies in ranges of data
> To: Carter Bullard <carter at qosient.com>
> 
> 
> On Sat, Feb 9, 2008 at 3:59 PM, Carter Bullard <carter at qosient.com> wrote:
>  >  These are great!!!  Simple and are doing a very specific job.
>  >  We will have a ./contrib directory in the distribution, and as long
>  >  as the code can be released under a GNU license (that's what
>  >  argus is distributed under) then all is cool.
> 
>  Hi Carter,
> 
>  Thanks for your thorough responses. Apologies for not responding to
>  them sooner - simply due to lack of time.
> 
>  Feel free to distribute them under any GNU or OSI-approved license
>  that suits the project.
> 
> 
>  >  What I would like is discussion on what is the intent/goal, so we can
>  >  know if there are faster, more efficient ways of getting these numbers.
>  >  Is this something that would lend itself to near realtime alert and
>  >  alarming, is there any opportunity for persistent intermediate data
>  >  to be available in realtime for high performance streams processing!!!!
> 
>  I am operating Argus on a university network that has thousands of
>  hosts with varying degrees of host management:
> 
>  Tightly managed - central services where we have full administrative control
>  Loosely managed - departmental services where our administrative
>  control extends as far as the centrally managed switch port that is in
>  closest proximity to the device, but for which we have a defined point
>  of contact with which to raise issues.
>  Not managed - private laptops, hosts on the wireless network where the
>  above description applies, but be have no particular point of contact
>  responsible for the host.
> 
>  In this environment I am using Argus to supplement the signature-based
>  analysis (Snort) that we perform at the gateway and at several
>  strategic points within the network. The overall goal would be to
>  provide (as near as possible to) real-time alerting of strange traffic
>  flows within the network in order to pro-actively reduce threat to and
>  from our network.
> 
>  The scripts that I have provided help to identify the "strange"
>  traffic patterns and so allow us to take prompt (sometimes automatic)
>  action against bad traffic, both in and out of our network. For
>  incoming traffic we can automatically add temporary drop rules into
>  our campus firewall (to thwart scanning), generate abuse emails based
>  on whois data, etc. We can do similar things for outgoing traffic but
>  rather than block the traffic entirely we might automatically
>  rate-limit it until the traffic is reviewed by an alerted network
>  administrator who can decide on the best course of action.
> 
>  Having real-time analysis and alarming to achieve these ends would be
>  excellent since there are substantial CPU and IO requirements for our
>  current processing of 15 mins of flow data every 5 mins.
> 
>  You can get a feel for the kinds of activities that I am interesting
>  from the script: port scanning (in and out), frequent connections to
>  uncommon ports, etc. I'm sure that there are plenty of other
>  activities that I will want to detect in time. One future extension of
>  our monitoring that has recently come to mind but not yet
>  crystalised...
> 
>  Roughly speaking our /16 network is divided into /8 subnets (separated
>  by VLANS) that serve individual departments, services, etc. We can
>  export the netflow information for most routed connections between
>  these subnets. Some of these subnets host central services, whilst
>  others host no services or services that are specific to just a few
>  other subnets. It would be useful to analyse these inter-subnet flows
>  to alert on suddenly increases in usage between unrelated subnets. Has
>  anybody previously used Argus to do something similar to this?
> 
> 
>  >  Best example is the arg_scan_hit_darkhost().  The $AUTH_IP_FILE is
>  >  intermediate data of the active IP addresses.  We can have other scripts
>  >  that generate and maintain a near real-time list of the "lit" net.
> 
>  A while ago I had the idea for rafilteraddr to be capable of
>  re-reading the filter file whenever it is updated. In our case, our
>  device registration procedure determines that all network devices must
>  be pre-registered before connection to the network, and this process
>  generates an accurate $AUTH_IP_FILE.
> 
> 
>  >  But, you're really using racluster() to modify the flow model of the
>  >  data
>  >  as its piped through.  I could write a simple fast program that does
>  >  that
>  >  on a stream.
> 
>  I'd be interested in helping to test this application. Along these
>  lines, are any of the current stream processors near to having the
>  ability to operate on a fixed-interval sliding window of data so that
>  you can alert as soon as any aggregate parameters exceed a pre-defined
>  threshold within any 5 minute period, say, and then to rest the
>  alerting mechanism for a period of time? Or is there another way that
>  you anticipate performing real-time threshold analysis for aggregate
>  data?
> 
> 
>  On Tue, Feb 12, 2008 at 4:35 PM, Carter Bullard <carter at qosient.com> wrote:
>  >  The direction aspects to this can be taken care of in a number
>  >  of ways, the most full-proof is to track the smac and dmac fields,
>  >  and to provide filters that deal with all the combinations.  The reason
>  >  this is a good strategy, is generally because the number of ethernet
>  >  addresses a give probe sees is small, so the list of filters is going
>  >  to be bounded, but, ...., it does mean that filters are very specific
>  >  for
>  >  each installation, which makes maintenance etc... a bit of a chore.
>  <...snip...>
> 
>>  This will give me any address that hit a dark address that is not
>  >  one of mine.   But I do end up using filters like:
>  >     (not src net $localnet and dst net $localnet)
> 
>  Agreed. Protocol specific direction is a red-herring and my current
>  scripts detect only connect style scans, rather than the more
>  sophisticated non-syn-based scans. smac and dmac based direction
>  determination is a much better solution that I will try.
> 
> 
>  >  In order to figure out did local address ever respond to this scanner,
>  >  which I think is the real question.  Who cares if someone is scanning,
>  >  the question is did they find anything, ..., new.
> 
>  To some extent I agree, but I would ultimate rather take preventative
>  action against a obvious scanner by blocking them wherever possible
>  *before* they find enumerate a substantial proportion of our network.
> 
> 
>  >  In support of your scripts, having a script that generates the darkhost
>  >  list or the inverse, the lithost list would be very helpful for some.
>  >  Do
>  >  you have a script, or do we need to generate one?  My radark.pl
>  >  script does some of this, and we can carve out that piece to provide
>  >  a ralight() like program?
> 
>  Our network registration system generates a reliable list of lit hosts
>  (assuming good housekeeping) and we invert the lookup using the
>  rafilteraddress "-v" flag to select dark hosts. We do nothing more
>  intelligent that that, I'm afraid!
> 
>  Thanks again for your detailed responses.
> 
> 
>  Warm regards,
> 
>  Tez