Flow aggregation..

[carter at qosient.com]

Hey Rick,
Glad to see your mail/tome!!!  racluster() is suppose to be better, so if you can't do something you want to do, we'll fix it!!
I changed the name because it is a different beast, but we can use the ragator name again, very difficult to kill a dragon you know!!!!

You should be able to do your bidding using the 'net x.y.z.w/a.b.c.d' filter syntax.
The idea is that the 'filter' approach should allow you to have huge control over matching statements, and the 'model' approach should give you huge flexibility on how you merge the records. If you can specify your traffic with an argus filter, you can use it in racluster to pick the traffic and cluster.  So, ..., let me try to specify your examples, if they don't work for you, we'll make it work!!

To " match all flows on the local /16 network but aggregate them into
their sub /24 ranges..  for locally and remotely originated
connections",  does this do it?

Filter="src net 192.168.0.0/16" model="saddr/24 proto dport"
Filter="dst net 192.168.0.0/16" model="daddr/24 proto dport"

You're suppose to be able to provide a non-contiguous mask by putting a real mask instead of a digit, does this work?

Filter="src net 192.168.0.0/192.168.0.255 and dst port 53" model="saddr/255.255.0.255 proto dport"

Sorry if I got that wrong, I currently don't have access to the code!!!

Well have no sadness!!!!!

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: "Denton, Rick" <rick.denton at cybertrust.com>
Date: Thu, 5 Oct 2006 12:48:06 
To:<argus-info at lists.andrew.cmu.edu>
Subject: [ARGUS] Flow aggregation..


Hi,

long time listener (well user), first time caller.. :)

i've recently been looking at argus 3 to see what is going on and to
attempt to plan an upgrade path..

great work in general with argus.. much kudos to all concerned.. and
awfully good to see ipv6 support arrive :) i can now use argus at home
:)

i have one main concern with argus 3 however.. the distinct lack of
ragator(1).. in its place there seems to be racluster(1) which seems to
be less flexible in some ways than ragator(1).... ragator aggregating
until it hit 2^32 and spitting multiple entries so you then had to
aggregate the aggregate was a bit annoying but you will always hit some
ceiling where this will have to be done anyway i guess.. :/

i would use ragator, the friendly dragon, to aggregate flows based on
subnets of a larger address space by applying a netmask to the addresses
in the ragator flow config file.. this functionality seems to be missing
from (the more boringly named ;)) racluster.. some very rough / simple
examples..


flow 100 ip  192.168.0.0/16	0.0.0.0/0		*	*
*	200	x y
flow 101 ip	 0.0.0.0/0		192.168.0.0/16	*	*
*	201	x y

model 200 ip 255.255.255.0	0.0.0.0		yes	no	yes
model 201 ip 0.0.0.0		255.255.255.0	yes	no	yes


would match all flows on the local /16 network but aggregate them into
their sub /24 ranges..  for locally and remotely originated
connections.. which is useful when suballocating a larger address space
to subordinate networks that need to be accounted for individually in a
simple auto generatable ragator flow config :)

from what i can tell with racluster(1) this would have to be 256
flow/model lines or am i missing something obvious as there seems to be
no way to apply a mask to the match?


another useful thing i have attempted to match in the past (i suspect
this failed due to a bug / assumption made in the application of the
mask in 2.x tho i never really followed it up as wasn't that concerned
at the time) was using a noncontiguous netmask.. for example in the case
here say a set of dns servers on distinct network legs through an
infrastructure say each leg has a /24 (for simplicity) all dns servers
exist on .53 of each leg... you want to aggregate all connections
inbound and outbound based on protocol and service (dport) across all
dns servers.. bung a noncontiguous netmask in to match the servers
across all networks.. ie a 255.255.0.255 netmask in this case..

ala:

flow 100 ip 192.168.0.53	0.0.0.0/0		*	*
*	200	x y
flow 101 ip 0.0.0.0		192.168.0.53	*	*	*
200	x y

model 200 ip 255.255.0.255	0.0.0.0		yes	no	yes
model 201 ip 0.0.0.0		255.255.0.255	yes	no	yes

you would probably want to handle different services etc and possibly
explicitly separate 53 out with a 'no' on the protocol match in the
corresponding model to aggregate all (udp/tcp) dns traffic into one
aggregate etc etc.... for brevity and the purpose of the example.. will
just aggregate on 'dns servers' and proto/service triplet..

ragator (the friendly dragon) was an awfully useful/powerful tool once
you grok it's nuances ;)

it would be of great use in this environment to be able to easily auto
generate flow/model ragator configs to easily create protocol breakdowns
of traffic flows based on subnets of a larger range.. ie to be able to
pull assigned subordinate network ranges aggregate on those networks and
do so splitting on port or proto/port to be able to automate the process
of providing totals aswell as proto breakdown aggregates for the purpose
of accounting the traffic..

further usefulness here comes from having a real network instead of a
cidr as occasionally in the case of applying masks for the purposes of
aggregation (and for other bizaar circumstances) to be able to apply non
contiguous masks to easily achieve these sorts of outcomes for
example...

i see that the introduction of ipv6 would have seriously hindered
ragator and suspect that is why the newer (and more simplistic) matching
of racluster(1) was introduced.. which is fair enough..

real netmasks on ipv6 (rarely ever seen over prefix length style) would
also apply very usefully but would be a bugger to process as dealing
with the 128 bit values and masking would get a little tedious but would
be an awfully nice feature..

of course the above example with the dns servers split over multiple
legs should probably have a layer 4 switch on at least the public side
of it and if the argus probe was in front of that then a lot of the
problem would be alleviated as you would only ever see the one address
anyway.. but then internally if you wanted to track the data inside the
infrastructure you want visibility of the individual servers also and if
you sat it inside the layer 4 and could aggregate as above you could
have the best of both worlds...

to this end it saddens me to see the untimely death of ragator, the
friendly dragon :(