Argus & IPFIX?

Tue Oct 24 13:25:09 EDT 2017

For us near real-time processing isn't really an issue- so at least for now
I'll be aiming to leverage the second approach you've described which also
sounds to be the most common with the scenario we are in at the moment.
I've been reading/studying the radium/rastream/racluster documentation/man
pages etc. but still have questions related to configuration where there
are gaps.

Being that this approach will be collecting IPFIX data directed to my
server which will be running radium|rastream|racluster etc.  I'm hoping
someone can help me better understand/confirm the best configuration
options combination available via radium in order to do so?  I may be
overthinking/over-complicating this a bit but my main confusion is
surrounding how to tell radium to listen for that IPFIX data properly.  I'm
getting incoming data on udp/9995 which is confirmed via tcpdump...

I'm currently telling radium to listen for what the docs refer to as Cisco
Netflow (cmdline -C) but I am using radium.conf and the current settings
I'm using in the config file are as follows:

RADIUM_DAEMON=yes
RADIUM_DEBUG_LEVEL=0
RADIUM_MAR_STATUS_INTERVAL=60
RADIUM_CISCONETFLOW_PORT=192.xx.xx.xx:9995
RADIUM_ACCESS_PORT=561
RADIUM_BIND_IP=127.0.0.1
RADIUM_OUTPUT_FILE=/flowdata/radium/radium.out
RADIUM_SETUSER_ID=argus
RADIUM_SETGROUP_ID=argus

Invoked with.... radium -f /etc/radium.conf

Does radium log somewhere other than /var/log/messages?  Is it required to
manually specify the ability to generate debug logs when compiling for that
to be available? I'm not seeing anything mentioning that specifically via
./configure --help for argus-clients.

I'm trying to test that radium is processing the data I've confirmed is
coming via udp/9995 using rastream like:

rastream -S 127.0.0.01:561 -w -

When I invoke rastream in this fashion (as root from a root shell) should I
not see the converted argus formatted data being spewed out to stdout in
unidirectional argus format?

I do see the log entries from radium confirming the connection from
rastream, does this look normal?:

Oct 24 12:56:16  radium: radium[14196]: 12:56:16.271560 connect from
localhost[127.0.0.1]
Oct 24 12:56:16  radium[14196]: 12:56:16.271560 connect from
localhost[127.0.0.1]
Oct 24 12:56:30  radium: radium[14196]: 12:56:30.781854
ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
Oct 24 12:56:30  radium[14196]: 12:56:30.781854 ArgusCheckClientMessage:
client localhost[127.0.0.1] sent DONE

However, I'm not seeing anything being printed out to stdout though, also
the output file I've configured via radium doesn't seem to be doing much
(as far as I can tell at least), it's incrementing in size but not nearly
to the degree I would expect but then again it does appear to be a DBase 3
data file which appears to be a database that is showing a lot of records
already according to the output of the file command:

$ file /flowdata/radium/radium.out
/flowdata/radium/radium.out: DBase 3 data file with memo(s) (2097152
records)

Should I be using another instance of radium to test the incoming data is
being processed rather than trying to use rastream for that?  Does this
make sense and/or is that doable?

Still very much learning all these tools- help is greatly appreciated, many
thanks in advance.

-Drew

---------- Forwarded message ----------
From: Carter Bullard <carter at qosient.com>
Date: Tue, Oct 17, 2017 at 10:10 AM
Subject: Re: [ARGUS] Argus & IPFIX?
To: Drew Dixon <dwdixon at umich.edu>

Hey Drew,

The open source clients have all the basic functionality to build a system
that can handle complex asymmetric monitoring problems, but they not
designed specifically to do this, nor are they designed to go very fast...

If you have any open source argus questions, please send them to the public
mailing list.

If you want to merge asymmetric flows from multiple sources, the best
approach is going to be dependent on how quickly you want to know about
issues on the wire.  If time is important, you will want to use radium() to
collect real-time streaming data, and then use programs like rabins() to
aggregate the uni-directional streams into bi-directional streams, and keep
the processing pipeline going.  There are time issues to work out, but the
concepts in radium and rabins are what are needed to provide near-realtime
multi-source aggregation.

If near real-time processing is not an issue, then having radium collect
the IPFIX data, covert it to argus data and use a program like rastream()
to break the stream up into files, and then post process the files with
racluster() to merge the records to form the bi-directional flow data.
That is the most common way of doing what you’ve described.  The open
source racluster() is a bit of a memory hog, so you’ll want give your
system a bit of RAM, and then tune the process to handle the number of
flows that you are processing.  So far in our experience, 100G doesn’t
significantly increase the numbers of simultaneous flows, but rather
supports bigger elephants co-existing with the traditional mix of data
(research university networks).  You may find that in your specific
network, that the flow data processing requirements aren’t much different
than 10G, but …, then again you may be more like a wireless cellular
network, where 100G really means 10x the flows seen at 10G … just all
depends on your network design and customer base.

Good luck, and keep us posted ….
Carter

[image: QoSient]        <http://qosient.com/>
Carter Bullard  <carter at qosient.com>• CTO
150 E 57th Street, Suite 12D
<https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
New York, New York 10022
<https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
-2795
Phone +1.212.588.9133 • Mobile +1.917.497.9494

On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <carter at qosient.com> wrote:

> Hey Drew,
> Argus should be able to read most/any IPFIX TCP/UDP data source, at least
> that is the goal.  To that end, if you have some IPFIX data that the ra*
> programs can’t read, I’ll spend some time making it work.  So if your using
> Juniper, have it export UDP IPFIX, and we should be able to read them, as
> the router advertises the templates in a reasonable timeframe, as we need
> to see the templates before we can decode the records (really terrible
> design flaw).
>
> We, of course recommend that you generate your own flow records rather
> than read from integrated IPFIX, especially if you’re network is going
> particularly fast.  QoSient has 1g, 10g, 40g and 100g argus sensor
> appliances for sale, so if you’re looking to do the do for real, think
> about generating your own data.
>
> Hope all is most excellent,
> Carter
>
> [image: QoSient] <http://qosient.com>
> Carter Bullard <carter at qosient.com> • CTO
> 150 E 57th Street, Suite 12D
> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> New York, New York 10022
> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> -2795
> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>
>
>
> On Oct 16, 2017, at 11:18 AM, Drew Dixon <dwdixon at umich.edu> wrote:
>
> Hello,
>
> I'm wondering what the current status of Argus' support of reading IPFIX
> and if there might be any relevant information/updates on that front which
> someone could share?
>
> I did some quick searching online and see mention of IPFIX in relation to
> Argus but nothing really stating that it's officially supported at this
> time etc.
>
> Thank you!
>
> -Drew
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20171024/33c1e150/attachment.html>