Argus & IPFIX?

Tue Oct 24 13:51:48 EDT 2017

Not sure what version you are using, but you should be able to read IPFIX data from an ra* program using the same strategy as radium.
Does this work ???

   ra -S ipfix://192.xx.xx.xx:9995

Carter

> On Oct 24, 2017, at 1:25 PM, Drew Dixon <dwdixon at umich.edu> wrote:
> 
> For us near real-time processing isn't really an issue- so at least for now I'll be aiming to leverage the second approach you've described which also sounds to be the most common with the scenario we are in at the moment.  I've been reading/studying the radium/rastream/racluster documentation/man pages etc. but still have questions related to configuration where there are gaps.
> 
> Being that this approach will be collecting IPFIX data directed to my server which will be running radium|rastream|racluster etc.  I'm hoping someone can help me better understand/confirm the best configuration options combination available via radium in order to do so?  I may be overthinking/over-complicating this a bit but my main confusion is surrounding how to tell radium to listen for that IPFIX data properly.  I'm getting incoming data on udp/9995 which is confirmed via tcpdump...
> 
> I'm currently telling radium to listen for what the docs refer to as Cisco Netflow (cmdline -C) but I am using radium.conf and the current settings I'm using in the config file are as follows:
> 
> RADIUM_DAEMON=yes
> RADIUM_DEBUG_LEVEL=0
> RADIUM_MAR_STATUS_INTERVAL=60
> RADIUM_CISCONETFLOW_PORT=192.xx.xx.xx:9995
> RADIUM_ACCESS_PORT=561
> RADIUM_BIND_IP=127.0.0.1
> RADIUM_OUTPUT_FILE=/flowdata/radium/radium.out
> RADIUM_SETUSER_ID=argus
> RADIUM_SETGROUP_ID=argus
> 
> Invoked with.... radium -f /etc/radium.conf
> 
> Does radium log somewhere other than /var/log/messages?  Is it required to manually specify the ability to generate debug logs when compiling for that to be available? I'm not seeing anything mentioning that specifically via ./configure --help for argus-clients.
> 
> I'm trying to test that radium is processing the data I've confirmed is coming via udp/9995 using rastream like:
> 
> rastream -S 127.0.0.01:561 <http://127.0.0.1:561/> -w -
> 
> When I invoke rastream in this fashion (as root from a root shell) should I not see the converted argus formatted data being spewed out to stdout in unidirectional argus format?  
> 
> I do see the log entries from radium confirming the connection from rastream, does this look normal?:
> 
> Oct 24 12:56:16  radium: radium[14196]: 12:56:16.271560 connect from localhost[127.0.0.1]
> Oct 24 12:56:16  radium[14196]: 12:56:16.271560 connect from localhost[127.0.0.1]
> Oct 24 12:56:30  radium: radium[14196]: 12:56:30.781854 ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
> Oct 24 12:56:30  radium[14196]: 12:56:30.781854 ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
> 
> However, I'm not seeing anything being printed out to stdout though, also the output file I've configured via radium doesn't seem to be doing much (as far as I can tell at least), it's incrementing in size but not nearly to the degree I would expect but then again it does appear to be a DBase 3 data file which appears to be a database that is showing a lot of records already according to the output of the file command:
> 
> $ file /flowdata/radium/radium.out
> /flowdata/radium/radium.out: DBase 3 data file with memo(s) (2097152 records)
> 
> Should I be using another instance of radium to test the incoming data is being processed rather than trying to use rastream for that?  Does this make sense and/or is that doable?
> 
> Still very much learning all these tools- help is greatly appreciated, many thanks in advance.
> 
> -Drew
> 
> ---------- Forwarded message ----------
> From: Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>>
> Date: Tue, Oct 17, 2017 at 10:10 AM
> Subject: Re: [ARGUS] Argus & IPFIX?
> To: Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>>
> 
> 
> Hey Drew,
> 
> The open source clients have all the basic functionality to build a system that can handle complex asymmetric monitoring problems, but they not designed specifically to do this, nor are they designed to go very fast...
> 
> If you have any open source argus questions, please send them to the public mailing list.
> 
> If you want to merge asymmetric flows from multiple sources, the best approach is going to be dependent on how quickly you want to know about issues on the wire.  If time is important, you will want to use radium() to collect real-time streaming data, and then use programs like rabins() to aggregate the uni-directional streams into bi-directional streams, and keep the processing pipeline going.  There are time issues to work out, but the concepts in radium and rabins are what are needed to provide near-realtime multi-source aggregation.
> 
> If near real-time processing is not an issue, then having radium collect the IPFIX data, covert it to argus data and use a program like rastream() to break the stream up into files, and then post process the files with racluster() to merge the records to form the bi-directional flow data.  That is the most common way of doing what you’ve described.  The open source racluster() is a bit of a memory hog, so you’ll want give your system a bit of RAM, and then tune the process to handle the number of flows that you are processing.  So far in our experience, 100G doesn’t significantly increase the numbers of simultaneous flows, but rather supports bigger elephants co-existing with the traditional mix of data (research university networks).  You may find that in your specific network, that the flow data processing requirements aren’t much different than 10G, but …, then again you may be more like a wireless cellular network, where 100G really means 10x the flows seen at 10G … just all depends on your network design and customer base.
> 
> Good luck, and keep us posted ….
> Carter
> 
>         <http://qosient.com/>     	 	
> Carter Bullard  <mailto:carter at qosient.com>• CTO
> 150 E 57th Street, Suite 12D <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> New York, New York 10022 <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>-2795
> Phone +1.212.588.9133 • Mobile +1.917.497.9494
> 
> 
> On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
> Hey Drew,
> Argus should be able to read most/any IPFIX TCP/UDP data source, at least that is the goal.  To that end, if you have some IPFIX data that the ra* programs can’t read, I’ll spend some time making it work.  So if your using Juniper, have it export UDP IPFIX, and we should be able to read them, as the router advertises the templates in a reasonable timeframe, as we need to see the templates before we can decode the records (really terrible design flaw).
> 
> We, of course recommend that you generate your own flow records rather than read from integrated IPFIX, especially if you’re network is going particularly fast.  QoSient has 1g, 10g, 40g and 100g argus sensor appliances for sale, so if you’re looking to do the do for real, think about generating your own data.
> 
> Hope all is most excellent,
> Carter
> 
>          <http://qosient.com/>     	 	
> Carter Bullard  <mailto:carter at qosient.com>• CTO
> 150 E 57th Street, Suite 12D <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> New York, New York 10022 <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>-2795
> Phone +1.212.588.9133 • Mobile +1.917.497.9494
> 
>  
> 
>> On Oct 16, 2017, at 11:18 AM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>> 
>> Hello,
>> 
>> I'm wondering what the current status of Argus' support of reading IPFIX and if there might be any relevant information/updates on that front which someone could share?  
>> 
>> I did some quick searching online and see mention of IPFIX in relation to Argus but nothing really stating that it's officially supported at this time etc.
>> 
>> Thank you!
>> 
>> -Drew
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20171024/f927c467/attachment.html>