Argus & IPFIX?
Carter Bullard
carter at qosient.com
Wed Oct 25 10:56:09 EDT 2017
Also, there isn’t anything special about getting these packets. Is the interface working ??? Can you ping in and out of it ???
If we decide that it could be the card driver, is there a standard ethernet interface you can use for testing on this box ???
Carter
> On Oct 25, 2017, at 10:48 AM, Carter Bullard <carter at qosient.com> wrote:
>
> Hey Drew,
> So lets get this working and then we can get to the best way to do it.
> Here is the problem in your debug output …
>
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192553 ArgusReadConnection(0x78d85010, 2) reading cisco wire format
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192566 ArgusReadConnection(0x78d85010, 2) returning 0
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192620 ArgusHandleRecord (0x7fef78d85228, 0x7fef78f17808) returning -1
>
> The ArgusHandleRecord() which is the routine that is parses the data, is returning -1.
> If you have a packet capture of the data, I can take a look to see what your routers are sending that we’re not parsing.
>
> Carter
>
>> On Oct 25, 2017, at 10:03 AM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>
>> Thanks Carter, replies in-line below:
>>
>> On Tue, Oct 24, 2017 at 11:02 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>> The timeout is a 1 second timeout routine that is called every second, regardless of what is going on …
>>
>> Ah I see...good to know.
>>
>> Try running ra.1 instead of radium to see if you can’t get the clients to read your packets, and you can up the -D option higher (9) to see if you’re getting any input.
>>
>> I did try ra as well radium, I probably should have explained a bit more...but here goes below:
>>
>> If you can use tcpdump/wireshark to capture the packets you expect on the interface that ra* is opening, that would help in diagnosis.
>>
>> So the single system I'm testing the ra* clients on is being directly sent IPFIX data and that has been confirmed via tcpdump, the same system has a network interface assigned the 192.xx.xx.xx IP address which is a NIC local to this same server which is the destination IP (192.xx.xx.xx:9995) our remote IPFIX data source is configured to send the IPFIX data to (192.xx.xx.xx:9995) which I've confirmed already via tcpdump is indeed coming in steadily at that IP and udp port.
>>
>> Reacmember, you’re not capturing from the remote host, you’re opening a specific interface on your sensor, where IPFIX packets are being sent.
>>
>> Correct, I am wanting to use whichever proper ra* client program you would recommend that would be best to bind to the specific local interface on my same system (which is the 192.xx.xx.xx IP address) and collect/convert that data. I'm still bit cloudy on what the best ra* client tool and proper combination of command line/config file options/syntax is best to properly open/bind to this specific interface on my system that's assigned 192.xx.xx.xx and receiving raw IPFIX data at that IP/interface via udp port 9995.
>>
>> This NIC is a Myricom running the latest version of SNFv3 drivers...(SNFv3.0.12) if that matters or could be creating issues...
>>
>> I've tried both radium with the config file options I shared earlier and also ra using the command line options you recommended:
>>
>> ra -D6 -S ipfix://192.xx.xx.xx:9995 <>
>>
>> <>
>> As well as another which seems to be doing more than the ipfix://
>>
>> ra -D6 -S cisco://192.xx.xx.xx:9995 <cisco://192.xx.xx.xx:9995>
>>
>> Should either or both of these be working to read the raw IPFIX data coming into this local (192.xx.xx.xx:9995) network interface IP on the same box? I tried these both again now with -D9/-D10 as you just mentioned and it isn't giving me too much more to go on (at least that I can make sense of without knowing the code) as far as if the client tools are seeing the raw IPFIX data on the 192.xx.xx.xx:9995 interface and port they are binding to- it seems they are not seeing the data for some reason.
>>
>> In the man pages I see a lot of reference to "binding to remote hosts" but in my case I want to bind to a local but externally facing (Non-localhost/127.0.0.1 <http://127.0.0.1/>) interface on my system to collect the incoming raw IPFIX data and convert it to argus data and then post-process further with additional client tools once this first piece is working and collecting raw IPFIX and converting to Argus format but I can't get any clients to read the incoming IPFIX data it seems.
>>
>>
>> *************************************************************************************
>> ra -D10 -S ipfix://192.xx.xx.xx:9995 <ipfix://192.xx.xx.xx:9995> #Just seems to die immediately on its own after executing:
>>
>>
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543716 ArgusCalloc (1, 461728) returning 0xdfe37010
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543785 ArgusAddToQueue (0x2873190, 0x7fcfdfe37010) returning 1
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543803 ArgusAddHostList (0xdfea8010, ipfix://192.xx.xx.xx:9995 <ipfix://192.xx.xx.xx:9995>, 64, 6) returning 1
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543826 main: reading files completed
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543842 ArgusCalloc (1, 80) returning 0x2873850
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543856 ArgusNewQueue () returning 0x2873850
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543871 ArgusPopQueue (0x2873190) returning 0x7fcfdfe37010
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543889 ArgusGetServerSocket (0x7fcfdfe37010) returning -1
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543927 ArgusAddToQueue (0x2873850, 0x7fcfdfe37010) returning 1
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543941 ArgusPopQueue (0x2873190) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543954 ArgusPopQueue (0x2873850) returning 0x7fcfdfe37010
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543978 ArgusAddToQueue (0x2873190, 0x7fcfdfe37010) returning 1
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.543991 ArgusPopQueue (0x2873850) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544016 ArgusPopQueue (0x2873850) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544042 ArgusFree (0x2873850)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544055 ArgusDeleteQueue (0x2873850) returning
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544072 ArgusShutDown (0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544095 ArgusPopQueue (0x2873190) returning 0x7fcfdfe37010
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544128 ArgusFree (0x7fcfdfe37010)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544146 ArgusPopQueue (0x2873190) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544162 ArgusFree (0x2873190)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544175 ArgusDeleteQueue (0x2873190) returning
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544190 ArgusPopQueue (0x28731f0) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544206 ArgusPopQueue (0x28731f0) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544222 ArgusFree (0x28731f0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544235 ArgusDeleteQueue (0x28731f0) returning
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544249 RaParseComplete(caught signal 0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544263 ArgusWindowClose () returning
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544278 RaParseComplete(caught signal 0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544293 RaParseComplete(caught signal 0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544308 ArgusShutDown (0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544324 ArgusPopQueue ((nil)) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544339 ArgusPopQueue ((nil)) returning (nil)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544649 ArgusFree (0x2873050)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544667 ArgusDeleteList (0x2873050, 4) returning
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544684 ArgusFree (0x28730f0)
>> ra[44769.40c7fedfcf7f0000]: 00:34:25.544699 ArgusDeleteList (0x28730f0, 4) returning
>> *************************************************************************************
>>
>>
>> *************************************************************************************
>> ra -D10 -S cisco://192.xx.xx.xx:9995 <cisco://192.xx.xx.xx:9995> #Runs but doesn't seem to do anything, here is more verbose output:
>>
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173465 ArgusCalloc (1, 461728) returning 0x78d85010
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173544 ArgusAddToQueue (0x16fb190, 0x7fef78d85010) returning 1
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173563 ArgusAddHostList (0x78df6010, cisco://192.xx.xx.xx:9995 <cisco://192.xx.xx.xx:9995>, 16, 17) returning 1
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173602 main: reading files completed
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173620 ArgusCalloc (1, 80) returning 0x16fb850
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173636 ArgusNewQueue () returning 0x16fb850
>> ra[44711.40a7f378ef7f0000]: 00:25:53.173653 ArgusPopQueue (0x16fb190) returning 0x7fef78d85010
>> ra[44711]: 00:25:53.174131 Binding 192.xx.xx.xx:9995 Expecting Netflow records
>> ra[44711.40a7f378ef7f0000]: 00:25:53.174318 receiving
>> ra[44711.40a7f378ef7f0000]: 00:25:53.174339 ArgusGetServerSocket (0x7fef78d85010) returning 3
>> ra[44711.40a7f378ef7f0000]: 00:25:53.174380 ArgusCalloc (1, 4194304) returning 0x77c02010
>> ra[44711.40a7f378ef7f0000]: 00:25:53.174413 ArgusCalloc (1, 262144) returning 0x77bc1010
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192504 ArgusInitAddrtoname (0x7fef78df6010, 0x0, 0x0)
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192540 ArgusParseInit(0x7fef78df6010 0x7fef78d85010
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192553 ArgusReadConnection(0x78d85010, 2) reading cisco wire format
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192566 ArgusReadConnection(0x78d85010, 2) returning 0
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192620 ArgusHandleRecord (0x7fef78d85228, 0x7fef78f17808) returning -1
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192634 ArgusAddToQueue (0x16fb1f0, 0x7fef78d85010) returning 1
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192661 ArgusPopQueue (0x16fb190) returning (nil)
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192674 ArgusPopQueue (0x16fb850) returning (nil)
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192687 ArgusPopQueue (0x16fb850) returning (nil)
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192699 ArgusFree (0x16fb850)
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192711 ArgusDeleteQueue (0x16fb850) returning
>> ra[44711.40a7f378ef7f0000]: 00:25:53.192736 ArgusReadStream(0x7fef78df6010) starting
>> ra[44711.40a7f378ef7f0000]: 00:25:53.443019 ArgusAdjustGlobalTime real 1508905553.443019 global 1508905553.443019
>> ra[44711.40a7f378ef7f0000]: 00:25:53.693329 ArgusAdjustGlobalTime real 1508905553.693329 global 1508905553.693329
>> ra[44711.40a7f378ef7f0000]: 00:25:53.943634 ArgusAdjustGlobalTime real 1508905553.943634 global 1508905553.943634
>> ra[44711.40a7f378ef7f0000]: 00:25:54.193936 ArgusAdjustGlobalTime real 1508905554.193935 global 1508905554.193935
>> ra[44711.40a7f378ef7f0000]: 00:25:54.444239 ArgusAdjustGlobalTime real 1508905554.444239 global 1508905554.444239
>> ra[44711.40a7f378ef7f0000]: 00:25:54.444276 ArgusClientTimeout()
>> ra[44711.40a7f378ef7f0000]: 00:25:54.694571 ArgusAdjustGlobalTime real 1508905554.694571 global 1508905554.694571
>> ra[44711.40a7f378ef7f0000]: 00:25:54.944887 ArgusAdjustGlobalTime real 1508905554.944887 global 1508905554.944887
>> ra[44711.40a7f378ef7f0000]: 00:25:55.195191 ArgusAdjustGlobalTime real 1508905555.195191 global 1508905555.195191
>> ra[44711.40a7f378ef7f0000]: 00:25:55.445495 ArgusAdjustGlobalTime real 1508905555.445495 global 1508905555.445495
>> ra[44711.40a7f378ef7f0000]: 00:25:55.445570 ArgusClientTimeout()
>> ra[44711.40a7f378ef7f0000]: 00:25:55.695858 ArgusAdjustGlobalTime real 1508905555.695858 global 1508905555.695858
>> ra[44711.40a7f378ef7f0000]: 00:25:55.946168 ArgusAdjustGlobalTime real 1508905555.946168 global 1508905555.946168
>> ra[44711.40a7f378ef7f0000]: 00:25:56.196477 ArgusAdjustGlobalTime real 1508905556.196476 global 1508905556.196476
>> ra[44711.40a7f378ef7f0000]: 00:25:56.446785 ArgusAdjustGlobalTime real 1508905556.446785 global 1508905556.446785
>> ra[44711.40a7f378ef7f0000]: 00:25:56.446822 ArgusClientTimeout()
>> ^Cra[44711.40a7f378ef7f0000]: 00:25:56.531281 ArgusShutDown (2)
>> ****************************************************************************************
>>
>>
>> Thank you,
>>
>> -Drew
>>
>>
>>
>> Carter
>>
>>
>>
>>> On Oct 24, 2017, at 5:47 PM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>>
>>> Thanks Carter, I re-compiled as you directed and now have debug functionality running argus-clients-3.0.8.2...still seeing IPFIX data coming in via 192.xx.xx.xx:9995 (UDP) in a steady stream via tcpdump.
>>>
>>> I ran radium again with debug set to 6 and using the config file parameters I mentioned earlier in the thread. I keep getting repeated "ArgusClientTimeout()" after radium binds to the 192.xx.xx.xx:9995 network interface on the same server to listen for/collect the IPFIX data. I also tried running ra as you suggested, with ipfix:// ra just exits immediately, with cisco:// ra also repeatedly gives me "ArgusClientTimeout()" after binding to the server NIC's IP on udp/9995...all debug log results are below (level 6).
>>>
>>> Not certain but guessing this means it's not receiving/recognizing any of the incoming IPFIX data or something else?
>>>
>>>
>>> ***********************************************************************
>>> Oct 24 17:20:54 argus1 systemd: Started radium Service.
>>> Oct 24 17:20:54 argus1 systemd: Starting radium Service...
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582224 ArgusCalloc (1, 560) returning 0x3113e80
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582308 ArgusCalloc (1, 112) returning 0x31137c0
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582319 ArgusCalloc (1, 80) returning 0x3113840
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582327 ArgusNewQueue () returning 0x3113840
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582336 ArgusCalloc (1, 56) returning 0x31138a0
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582349 ArgusCalloc (65536, 8) returning 0xe0bd8010
>>> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582357 ArgusNewHashTable (65536) returning 0x31138a0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582566 setArgusMarReportInterval(15) returning
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582854 ArgusCalloc (1, 461728) returning 0xdfdf5010
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582894 ArgusAddHostList (0xe0c59010, 192.xx.xx.xx:9995, 16, 17) returning 1
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582929 RadiumParseResourceFile: ArgusBindAddr "127.0.0.1"
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582959 ArgusCalloc (1, 144) returning 0x3114390
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.582990 ArgusNewList () returning 0x3114390
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.583032 ArgusCalloc (1, 296) returning 0x3114460
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.583060 ArgusPushFrontList (0x3114390, 0x3114460, 1) returning 0x3114970
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.583538 RadiumParseResourceFile (/etc/radium.conf) returning 0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.583589 ArgusEstablishListen(0xe0c59010, 561, 127.0.0.1) binding: 127.0.0.1:561 <http://127.0.0.1:561/>
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584032 ArgusEstablishListen(0xe0c59010, 561, 127.0.0.1) returning 3
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584073 ArgusCalloc (1, 392) returning 0x31140c0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584102 ArgusCalloc (1, 80) returning 0x3114b40
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584129 ArgusNewQueue () returning 0x3114b40
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584153 ArgusNewOutput() returning retn 0x31140c0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584178 ArgusCalloc (1, 144) returning 0x3114250
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584202 ArgusNewList () returning 0x3114250
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584227 ArgusCalloc (1, 128) returning 0x3115da0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584251 ArgusGenerateInitialMar() returning
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584282 ArgusCalloc (1, 168) returning 0x3115e30
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584355 ArgusCalloc (1, 65792) returning 0x3115ee0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584388 ArgusCalloc (1, 144) returning 0x3125ff0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584412 ArgusNewList () returning 0x3125ff0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584436 ArgusNewSocket (4) returning 0x3115ee0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584479 ArgusPushBackList (0x3114390, 0x3114460, 1) returning 1
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.584965 ArgusInitOutput() done
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585011 main: reading files completed
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585040 ArgusCalloc (1, 80) returning 0x31262d0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585065 ArgusNewQueue () returning 0x31262d0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585121 ArgusFree (0x31262d0)
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585149 ArgusDeleteQueue (0x31262d0) returning
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:54.585174 ArgusReadStream(0x7fc7e0c59010) starting
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.585224 ArgusConnectRemote(0x7fc7dfdf5010) starting
>>> Oct 24 17:20:54 argus1 radium: radium[41360]: 17:20:54.585349 Binding 192.xx.xx.xx:9995 Expecting Netflow records
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.585433 receiving
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.585464 ArgusGetServerSocket (0x7fc7dfdf5010) returning 5
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.585513 ArgusCalloc (1, 4194304) returning 0xce7de010
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.586086 ArgusCalloc (1, 262144) returning 0xce79d010
>>> Oct 24 17:20:54 argus1 radium[41360]: 17:20:54.585349 Binding 192.xx.xx.xx:9995 Expecting Netflow records
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602162 ArgusInitAddrtoname (0x7fc7e0c59010, 0x0, 0x0)
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602202 ArgusParseInit(0x7fc7e0c59010 0x7fc7dfdf5010
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602223 ArgusReadConnection(0xdfdf5010, 2) reading cisco wire format
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602245 ArgusReadConnection(0xdfdf5010, 2) returning 0
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602265 ArgusConnectRemote(0x7fc7dfdf5010) connected to 192.xx.xx.xx
>>> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]: 17:20:54.602285 ArgusConnectRemote() done!
>>> Oct 24 17:20:55 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:55.636383 ArgusClientTimeout()
>>> Oct 24 17:20:56 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:56.637568 ArgusClientTimeout()
>>> Oct 24 17:20:57 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:57.638727 ArgusClientTimeout()
>>> Oct 24 17:20:58 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:58.639887 ArgusClientTimeout()
>>> Oct 24 17:20:59 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:20:59.641141 ArgusClientTimeout()
>>> Oct 24 17:21:00 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:21:00.642402 ArgusClientTimeout()
>>> Oct 24 17:21:01 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]: 17:21:01.643565 ArgusClientTimeout()
>>> Oct 24 17:21:01 argus1 systemd: Stopping radium Service...
>>> ***********************************************************************
>>>
>>>
>>> ra using ipfix://
>>> ***********************************************************************
>>> # ra -D6 -S ipfix://192.xx.xx.xx:9995 <>
>>>
>>> ra[41491.4037f00ea37f0000]: 17:32:00.930991 ArgusCalloc (1, 461728) returning 0xed4e010
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931065 ArgusAddHostList (0xedbf010, ipfix://192.xx.xx.xx:9995 <>, 64, 6) returning 1
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931091 main: reading files completed
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931136 ArgusCalloc (1, 80) returning 0x1545840
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931157 ArgusNewQueue () returning 0x1545840
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931185 ArgusGetServerSocket (0x7fa30ed4e010) returning -1
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931217 ArgusFree (0x1545840)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931237 ArgusDeleteQueue (0x1545840) returning
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931269 ArgusShutDown (0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931298 ArgusFree (0x7fa30ed4e010)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931319 ArgusFree (0x1545190)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931336 ArgusDeleteQueue (0x1545190) returning
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931355 ArgusFree (0x15451f0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931371 ArgusDeleteQueue (0x15451f0) returning
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931389 RaParseComplete(caught signal 0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931404 ArgusWindowClose () returning
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931420 RaParseComplete(caught signal 0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931439 RaParseComplete(caught signal 0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931457 ArgusShutDown (0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931789 ArgusFree (0x1545050)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931808 ArgusDeleteList (0x1545050, 4) returning
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931829 ArgusFree (0x15450f0)
>>> ra[41491.4037f00ea37f0000]: 17:32:00.931842 ArgusDeleteList (0x15450f0, 4) returning
>>> ***********************************************************************
>>>
>>> ra using cisco://
>>> ***********************************************************************
>>> # ra -D6 -S cisco://192.xx.xx.xx:9995 <>
>>> ra[41498.407743631c7f0000]: 17:33:07.958512 ArgusCalloc (1, 461728) returning 0x63282010
>>> ra[41498.407743631c7f0000]: 17:33:07.958588 ArgusAddHostList (0x632f3010, cisco://192.xx.xx.xx:9995 <>, 16, 17) returning 1
>>> ra[41498.407743631c7f0000]: 17:33:07.958612 main: reading files completed
>>> ra[41498.407743631c7f0000]: 17:33:07.958633 ArgusCalloc (1, 80) returning 0x18dc840
>>> ra[41498.407743631c7f0000]: 17:33:07.958664 ArgusNewQueue () returning 0x18dc840
>>> ra[41498]: 17:33:07.958737 Binding 192.xx.xx.xx:9995 Expecting Netflow records
>>> ra[41498.407743631c7f0000]: 17:33:07.959292 receiving
>>> ra[41498.407743631c7f0000]: 17:33:07.959312 ArgusGetServerSocket (0x7f1c63282010) returning 3
>>> ra[41498.407743631c7f0000]: 17:33:07.959354 ArgusCalloc (1, 4194304) returning 0x620ff010
>>> ra[41498.407743631c7f0000]: 17:33:07.959384 ArgusCalloc (1, 262144) returning 0x620be010
>>> ra[41498.407743631c7f0000]: 17:33:07.977130 ArgusInitAddrtoname (0x7f1c632f3010, 0x0, 0x0)
>>> ra[41498.407743631c7f0000]: 17:33:07.977161 ArgusParseInit(0x7f1c632f3010 0x7f1c63282010
>>> ra[41498.407743631c7f0000]: 17:33:07.977174 ArgusReadConnection(0x63282010, 2) reading cisco wire format
>>> ra[41498.407743631c7f0000]: 17:33:07.977186 ArgusReadConnection(0x63282010, 2) returning 0
>>> ra[41498.407743631c7f0000]: 17:33:07.977209 ArgusHandleRecord (0x7f1c63282228, 0x7f1c63414808) returning -1
>>> ra[41498.407743631c7f0000]: 17:33:07.977227 ArgusFree (0x18dc840)
>>> ra[41498.407743631c7f0000]: 17:33:07.977240 ArgusDeleteQueue (0x18dc840) returning
>>> ra[41498.407743631c7f0000]: 17:33:07.977253 ArgusReadStream(0x7f1c632f3010) starting
>>> ra[41498.407743631c7f0000]: 17:33:09.228565 ArgusClientTimeout()
>>> ra[41498.407743631c7f0000]: 17:33:10.229714 ArgusClientTimeout()
>>> ra[41498.407743631c7f0000]: 17:33:11.230838 ArgusClientTimeout()
>>> ***********************************************************************
>>>
>>> Thank you,
>>>
>>> -Drew
>>>
>>> On Tue, Oct 24, 2017 at 3:45 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>> When you run the ./configure program, like many open source distros, you can create files, like .debug or .devel, that ./configure looks for, and if they exist, it will configure the package to add specific functionality. If there is a ./.debug file, the ./conifgure script will compile in support for printing debug information on the command line …. Try this …
>>>
>>> % cd /full/path/to/your/argus/clients/distribution
>>> % touch .debug
>>> % ./configure
>>> % make clean; make
>>>
>>> Now all your ra* programs will be able to print detail debugging information, using the “-D debuglevel” option on the command line.
>>> You can also turn on this type of support like this …
>>>
>>> % ./configure —enable-debug=yes
>>>
>>> I prefer the .debug file, as you can set it and forget that you need the switch on the configure line.
>>> Please keep all email on the mailing list … Thanks...
>>>
>>> Carter
>>>
>>>
>>>
>>>> On Oct 24, 2017, at 3:33 PM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>>>
>>>> Whoops...apologies with that- I noticed it said on the downloads page that 3.0.8.1 was the current stable version and never thought I had a need to look on the main page.
>>>>
>>>> Please forgive my ignorance but I'm not 100% sure what you mean by "with the .debug tag" I've listed out my options using ./configure --help but don't see anything related to enabling debugging. Only thing that I can see is --enable-FEATURE[=ARG] include FEATURE [ARG=yes] is this what you're referring to?
>>>>
>>>> --enable-debug=yes ?
>>>>
>>>> Or somehow with referencing the files below?
>>>>
>>>> /argus-clients-3.0.8.2/common/._argus_debug.h
>>>> /argus-clients-3.0.8.2/common/argus_debug.h
>>>> /argus-clients-3.0.8.2/include/._argus_debug.h
>>>> /argus-clients-3.0.8.2/include/argus_debug.h
>>>>
>>>> I'm not certain on how to compile to enable debugging properly in the fashion you've described.
>>>>
>>>> I very much appreciate your help.
>>>>
>>>> Sincerely,
>>>>
>>>> -Drew
>>>>
>>>> On Tue, Oct 24, 2017 at 3:09 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>>> How about the link to argus-clients-3.0.8.2 from the main page …
>>>>
>>>> http://qosient.com/argus/ <http://qosient.com/argus/>
>>>>
>>>> You should ./configure and compile with the .debug tag in the home directory, and then run your ra command with the -D4 or -D5 option to see if you are receiving any packets.
>>>> Carter
>>>>
>>>> <http://qosient.com/>
>>>> Carter Bullard <mailto:carter at qosient.com>• CTO
>>>> 150 E 57th Street, Suite 12D <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York&entry=gmail&source=g>
>>>> New York, New York <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York&entry=gmail&source=g> 10022-2795
>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>
>>>>
>>>>
>>>>> On Oct 24, 2017, at 2:15 PM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>>>>
>>>>> I'm using the latest stable version (argus-clients-3.0.8) downloaded from the website (http://www.qosient.com/argus/downloads.shtml <http://www.qosient.com/argus/downloads.shtml>).
>>>>>
>>>>> That command doesn't appear to be working upon running it, it appears to just immediately exit and throw me back into a command prompt.
>>>>>
>>>>> In comparison if I run the following:
>>>>>
>>>>> ra -S cisco://192.xx.xx.xx:9995 <>
>>>>>
>>>>> <>
>>>>> It appears to stay running and sits on a new line with a flashing cursor waiting to print something possibly? But nothing is printed to stdout. <>
>>>>>
>>>>> <>
>>>>> Thank you, <>
>>>>>
>>>>> <>
>>>>> -Drew <>
>>>>>
>>>>> On Tue, Oct 24, 2017 at 1:51 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>>>> Not sure what version you are using, but you should be able to read IPFIX data from an ra* program using the same strategy as radium.
>>>>> Does this work ???
>>>>>
>>>>> ra -S ipfix://192.xx.xx.xx:9995 <>
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>>> On Oct 24, 2017, at 1:25 PM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>>>>>
>>>>>> For us near real-time processing isn't really an issue- so at least for now I'll be aiming to leverage the second approach you've described which also sounds to be the most common with the scenario we are in at the moment. I've been reading/studying the radium/rastream/racluster documentation/man pages etc. but still have questions related to configuration where there are gaps.
>>>>>>
>>>>>> Being that this approach will be collecting IPFIX data directed to my server which will be running radium|rastream|racluster etc. I'm hoping someone can help me better understand/confirm the best configuration options combination available via radium in order to do so? I may be overthinking/over-complicating this a bit but my main confusion is surrounding how to tell radium to listen for that IPFIX data properly. I'm getting incoming data on udp/9995 which is confirmed via tcpdump...
>>>>>>
>>>>>> I'm currently telling radium to listen for what the docs refer to as Cisco Netflow (cmdline -C) but I am using radium.conf and the current settings I'm using in the config file are as follows:
>>>>>>
>>>>>> RADIUM_DAEMON=yes
>>>>>> RADIUM_DEBUG_LEVEL=0
>>>>>> RADIUM_MAR_STATUS_INTERVAL=60
>>>>>> RADIUM_CISCONETFLOW_PORT=192.xx.xx.xx:9995
>>>>>> RADIUM_ACCESS_PORT=561
>>>>>> RADIUM_BIND_IP=127.0.0.1
>>>>>> RADIUM_OUTPUT_FILE=/flowdata/radium/radium.out
>>>>>> RADIUM_SETUSER_ID=argus
>>>>>> RADIUM_SETGROUP_ID=argus
>>>>>>
>>>>>> Invoked with.... radium -f /etc/radium.conf
>>>>>>
>>>>>> Does radium log somewhere other than /var/log/messages? Is it required to manually specify the ability to generate debug logs when compiling for that to be available? I'm not seeing anything mentioning that specifically via ./configure --help for argus-clients.
>>>>>>
>>>>>> I'm trying to test that radium is processing the data I've confirmed is coming via udp/9995 using rastream like:
>>>>>>
>>>>>> rastream -S 127.0.0.01:561 <http://127.0.0.1:561/> -w -
>>>>>>
>>>>>> When I invoke rastream in this fashion (as root from a root shell) should I not see the converted argus formatted data being spewed out to stdout in unidirectional argus format?
>>>>>>
>>>>>> I do see the log entries from radium confirming the connection from rastream, does this look normal?:
>>>>>>
>>>>>> Oct 24 12:56:16 radium: radium[14196]: 12:56:16.271560 connect from localhost[127.0.0.1]
>>>>>> Oct 24 12:56:16 radium[14196]: 12:56:16.271560 connect from localhost[127.0.0.1]
>>>>>> Oct 24 12:56:30 radium: radium[14196]: 12:56:30.781854 ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
>>>>>> Oct 24 12:56:30 radium[14196]: 12:56:30.781854 ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
>>>>>>
>>>>>> However, I'm not seeing anything being printed out to stdout though, also the output file I've configured via radium doesn't seem to be doing much (as far as I can tell at least), it's incrementing in size but not nearly to the degree I would expect but then again it does appear to be a DBase 3 data file which appears to be a database that is showing a lot of records already according to the output of the file command:
>>>>>>
>>>>>> $ file /flowdata/radium/radium.out
>>>>>> /flowdata/radium/radium.out: DBase 3 data file with memo(s) (2097152 records)
>>>>>>
>>>>>> Should I be using another instance of radium to test the incoming data is being processed rather than trying to use rastream for that? Does this make sense and/or is that doable?
>>>>>>
>>>>>> Still very much learning all these tools- help is greatly appreciated, many thanks in advance.
>>>>>>
>>>>>> -Drew
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>>
>>>>>> Date: Tue, Oct 17, 2017 at 10:10 AM
>>>>>> Subject: Re: [ARGUS] Argus & IPFIX?
>>>>>> To: Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>>
>>>>>>
>>>>>>
>>>>>> Hey Drew,
>>>>>>
>>>>>> The open source clients have all the basic functionality to build a system that can handle complex asymmetric monitoring problems, but they not designed specifically to do this, nor are they designed to go very fast...
>>>>>>
>>>>>> If you have any open source argus questions, please send them to the public mailing list.
>>>>>>
>>>>>> If you want to merge asymmetric flows from multiple sources, the best approach is going to be dependent on how quickly you want to know about issues on the wire. If time is important, you will want to use radium() to collect real-time streaming data, and then use programs like rabins() to aggregate the uni-directional streams into bi-directional streams, and keep the processing pipeline going. There are time issues to work out, but the concepts in radium and rabins are what are needed to provide near-realtime multi-source aggregation.
>>>>>>
>>>>>> If near real-time processing is not an issue, then having radium collect the IPFIX data, covert it to argus data and use a program like rastream() to break the stream up into files, and then post process the files with racluster() to merge the records to form the bi-directional flow data. That is the most common way of doing what you’ve described. The open source racluster() is a bit of a memory hog, so you’ll want give your system a bit of RAM, and then tune the process to handle the number of flows that you are processing. So far in our experience, 100G doesn’t significantly increase the numbers of simultaneous flows, but rather supports bigger elephants co-existing with the traditional mix of data (research university networks). You may find that in your specific network, that the flow data processing requirements aren’t much different than 10G, but …, then again you may be more like a wireless cellular network, where 100G really means 10x the flows seen at 10G … just all depends on your network design and customer base.
>>>>>>
>>>>>> Good luck, and keep us posted ….
>>>>>> Carter
>>>>>>
>>>>>> <http://qosient.com/>
>>>>>> Carter Bullard <mailto:carter at qosient.com>• CTO
>>>>>> 150 E 57th Street, Suite 12D <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>> New York, New York 10022 <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>-2795
>>>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>>>>> Hey Drew,
>>>>>> Argus should be able to read most/any IPFIX TCP/UDP data source, at least that is the goal. To that end, if you have some IPFIX data that the ra* programs can’t read, I’ll spend some time making it work. So if your using Juniper, have it export UDP IPFIX, and we should be able to read them, as the router advertises the templates in a reasonable timeframe, as we need to see the templates before we can decode the records (really terrible design flaw).
>>>>>>
>>>>>> We, of course recommend that you generate your own flow records rather than read from integrated IPFIX, especially if you’re network is going particularly fast. QoSient has 1g, 10g, 40g and 100g argus sensor appliances for sale, so if you’re looking to do the do for real, think about generating your own data.
>>>>>>
>>>>>> Hope all is most excellent,
>>>>>> Carter
>>>>>>
>>>>>> <http://qosient.com/>
>>>>>> Carter Bullard <mailto:carter at qosient.com>• CTO
>>>>>> 150 E 57th Street, Suite 12D <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>> New York, New York 10022 <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>-2795
>>>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 16, 2017, at 11:18 AM, Drew Dixon <dwdixon at umich.edu <mailto:dwdixon at umich.edu>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm wondering what the current status of Argus' support of reading IPFIX and if there might be any relevant information/updates on that front which someone could share?
>>>>>>>
>>>>>>> I did some quick searching online and see mention of IPFIX in relation to Argus but nothing really stating that it's officially supported at this time etc.
>>>>>>>
>>>>>>> Thank you!
>>>>>>>
>>>>>>> -Drew
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20171025/89d83569/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4045 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20171025/89d83569/attachment.bin>
More information about the argus
mailing list