Fwd: Some useful Argus scripts for detecting traffic anomalies in ranges of data

[Terry Burton]

Oops. Forgot to cc: the list...

---------- Forwarded message ----------
From: Terry Burton <tez at terryburton.co.uk>
Date: Thu, Feb 21, 2008 at 5:00 PM
Subject: Re: [ARGUS] Some useful Argus scripts for detecting traffic
anomalies in ranges of data
To: Carter Bullard <carter at qosient.com>


On Sat, Feb 9, 2008 at 3:59 PM, Carter Bullard <carter at qosient.com> wrote:
 >  These are great!!!  Simple and are doing a very specific job.
 >  We will have a ./contrib directory in the distribution, and as long
 >  as the code can be released under a GNU license (that's what
 >  argus is distributed under) then all is cool.

 Hi Carter,

 Thanks for your thorough responses. Apologies for not responding to
 them sooner - simply due to lack of time.

 Feel free to distribute them under any GNU or OSI-approved license
 that suits the project.


 >  What I would like is discussion on what is the intent/goal, so we can
 >  know if there are faster, more efficient ways of getting these numbers.
 >  Is this something that would lend itself to near realtime alert and
 >  alarming, is there any opportunity for persistent intermediate data
 >  to be available in realtime for high performance streams processing!!!!

 I am operating Argus on a university network that has thousands of
 hosts with varying degrees of host management:

 Tightly managed - central services where we have full administrative control
 Loosely managed - departmental services where our administrative
 control extends as far as the centrally managed switch port that is in
 closest proximity to the device, but for which we have a defined point
 of contact with which to raise issues.
 Not managed - private laptops, hosts on the wireless network where the
 above description applies, but be have no particular point of contact
 responsible for the host.

 In this environment I am using Argus to supplement the signature-based
 analysis (Snort) that we perform at the gateway and at several
 strategic points within the network. The overall goal would be to
 provide (as near as possible to) real-time alerting of strange traffic
 flows within the network in order to pro-actively reduce threat to and
 from our network.

 The scripts that I have provided help to identify the "strange"
 traffic patterns and so allow us to take prompt (sometimes automatic)
 action against bad traffic, both in and out of our network. For
 incoming traffic we can automatically add temporary drop rules into
 our campus firewall (to thwart scanning), generate abuse emails based
 on whois data, etc. We can do similar things for outgoing traffic but
 rather than block the traffic entirely we might automatically
 rate-limit it until the traffic is reviewed by an alerted network
 administrator who can decide on the best course of action.

 Having real-time analysis and alarming to achieve these ends would be
 excellent since there are substantial CPU and IO requirements for our
 current processing of 15 mins of flow data every 5 mins.

 You can get a feel for the kinds of activities that I am interesting
 from the script: port scanning (in and out), frequent connections to
 uncommon ports, etc. I'm sure that there are plenty of other
 activities that I will want to detect in time. One future extension of
 our monitoring that has recently come to mind but not yet
 crystalised...

 Roughly speaking our /16 network is divided into /8 subnets (separated
 by VLANS) that serve individual departments, services, etc. We can
 export the netflow information for most routed connections between
 these subnets. Some of these subnets host central services, whilst
 others host no services or services that are specific to just a few
 other subnets. It would be useful to analyse these inter-subnet flows
 to alert on suddenly increases in usage between unrelated subnets. Has
 anybody previously used Argus to do something similar to this?


 >  Best example is the arg_scan_hit_darkhost().  The $AUTH_IP_FILE is
 >  intermediate data of the active IP addresses.  We can have other scripts
 >  that generate and maintain a near real-time list of the "lit" net.

 A while ago I had the idea for rafilteraddr to be capable of
 re-reading the filter file whenever it is updated. In our case, our
 device registration procedure determines that all network devices must
 be pre-registered before connection to the network, and this process
 generates an accurate $AUTH_IP_FILE.


 >  But, you're really using racluster() to modify the flow model of the
 >  data
 >  as its piped through.  I could write a simple fast program that does
 >  that
 >  on a stream.

 I'd be interested in helping to test this application. Along these
 lines, are any of the current stream processors near to having the
 ability to operate on a fixed-interval sliding window of data so that
 you can alert as soon as any aggregate parameters exceed a pre-defined
 threshold within any 5 minute period, say, and then to rest the
 alerting mechanism for a period of time? Or is there another way that
 you anticipate performing real-time threshold analysis for aggregate
 data?


 On Tue, Feb 12, 2008 at 4:35 PM, Carter Bullard <carter at qosient.com> wrote:
 >  The direction aspects to this can be taken care of in a number
 >  of ways, the most full-proof is to track the smac and dmac fields,
 >  and to provide filters that deal with all the combinations.  The reason
 >  this is a good strategy, is generally because the number of ethernet
 >  addresses a give probe sees is small, so the list of filters is going
 >  to be bounded, but, ...., it does mean that filters are very specific
 >  for
 >  each installation, which makes maintenance etc... a bit of a chore.
 <...snip...>

>  This will give me any address that hit a dark address that is not
 >  one of mine.   But I do end up using filters like:
 >     (not src net $localnet and dst net $localnet)

 Agreed. Protocol specific direction is a red-herring and my current
 scripts detect only connect style scans, rather than the more
 sophisticated non-syn-based scans. smac and dmac based direction
 determination is a much better solution that I will try.


 >  In order to figure out did local address ever respond to this scanner,
 >  which I think is the real question.  Who cares if someone is scanning,
 >  the question is did they find anything, ..., new.

 To some extent I agree, but I would ultimate rather take preventative
 action against a obvious scanner by blocking them wherever possible
 *before* they find enumerate a substantial proportion of our network.


 >  In support of your scripts, having a script that generates the darkhost
 >  list or the inverse, the lithost list would be very helpful for some.
 >  Do
 >  you have a script, or do we need to generate one?  My radark.pl
 >  script does some of this, and we can carve out that piece to provide
 >  a ralight() like program?

 Our network registration system generates a reliable list of lit hosts
 (assuming good housekeeping) and we invert the lookup using the
 rafilteraddress "-v" flag to select dark hosts. We do nothing more
 intelligent that that, I'm afraid!

 Thanks again for your detailed responses.


 Warm regards,

 Tez