Argus & IPFIX?

Drew Dixon dwdixon at umich.edu
Wed Oct 25 10:03:29 EDT 2017 

Thanks Carter, replies in-line below:

On Tue, Oct 24, 2017 at 11:02 PM, Carter Bullard <carter at qosient.com> wrote:

> The timeout is a 1 second timeout routine that is called every second,
> regardless of what is going on …
>

Ah I see...good to know.


> Try running ra.1 instead of radium to see if you can’t get the clients to
> read your packets, and you can up the -D option higher (9) to see if you’re
> getting any input.
>

I did try ra as well radium, I probably should have explained a bit
more...but here goes below:


> If you can use tcpdump/wireshark to capture the packets you expect on the
> interface that ra* is opening, that would help in diagnosis.
>

So the single system I'm testing the ra* clients on is being directly sent
IPFIX data and that has been confirmed via tcpdump, the same system has a
network interface assigned the 192.xx.xx.xx IP address which is a NIC local
to this same server which is the destination IP (192.xx.xx.xx:9995) our
remote IPFIX data source is configured to send the IPFIX data to
(192.xx.xx.xx:9995) which I've confirmed already via tcpdump is indeed
coming in steadily at that IP and udp port.


> Reacmember, you’re not capturing from the remote host, you’re opening a
> specific interface on your sensor, where IPFIX packets are being sent.
>

Correct, I am wanting to use whichever proper ra* client program you would
recommend that would be best to bind to the specific local interface on my
same system (which is the 192.xx.xx.xx IP address) and collect/convert that
data.  I'm still bit cloudy on what the best ra* client tool and proper
combination of command line/config file options/syntax is best to properly
open/bind to this specific interface on my system that's assigned
192.xx.xx.xx and receiving raw IPFIX data at that IP/interface via udp port
9995.

This NIC is a Myricom running the latest version of SNFv3
drivers...(SNFv3.0.12) if that matters or could be creating issues...

I've tried both radium with the config file options I shared earlier and
also ra using the command line options you recommended:

ra -D6 -S ipfix://192.xx.xx.xx:9995

As well as another which seems to be doing more than the ipfix://

ra -D6 -S cisco://192.xx.xx.xx:9995

Should either or both of these be working to read the raw IPFIX data coming
into this local (192.xx.xx.xx:9995) network interface IP on the same box?
I tried these both again now with -D9/-D10 as you just mentioned and it
isn't giving me too much more to go on (at least that I can make sense of
without knowing the code) as far as if the client tools are seeing the raw
IPFIX data on the 192.xx.xx.xx:9995 interface and port they are binding to-
it seems they are not seeing the data for some reason.

In the man pages I see a lot of reference to "binding to *remote* hosts"
but in my case I want to bind to a local but externally facing
(Non-localhost/127.0.0.1) interface on my system to collect the incoming
raw IPFIX data and convert it to argus data and then post-process further
with additional client tools once this first piece is working and
collecting raw IPFIX and converting to Argus format but I can't get any
clients to read the incoming IPFIX data it seems.


************************************************************
*************************
 ra -D10 -S ipfix://192.xx.xx.xx:9995  #Just seems to die immediately on
its own after executing:


ra[44769.40c7fedfcf7f0000]: 00:34:25.543716 ArgusCalloc (1, 461728)
returning 0xdfe37010
ra[44769.40c7fedfcf7f0000]: 00:34:25.543785 ArgusAddToQueue (0x2873190,
0x7fcfdfe37010) returning 1
ra[44769.40c7fedfcf7f0000]: 00:34:25.543803 ArgusAddHostList (0xdfea8010,
ipfix://192.xx.xx.xx:9995, 64, 6) returning 1
ra[44769.40c7fedfcf7f0000]: 00:34:25.543826 main: reading files completed
ra[44769.40c7fedfcf7f0000]: 00:34:25.543842 ArgusCalloc (1, 80) returning
0x2873850
ra[44769.40c7fedfcf7f0000]: 00:34:25.543856 ArgusNewQueue () returning
0x2873850
ra[44769.40c7fedfcf7f0000]: 00:34:25.543871 ArgusPopQueue (0x2873190)
returning 0x7fcfdfe37010
ra[44769.40c7fedfcf7f0000]: 00:34:25.543889 ArgusGetServerSocket
(0x7fcfdfe37010) returning -1
ra[44769.40c7fedfcf7f0000]: 00:34:25.543927 ArgusAddToQueue (0x2873850,
0x7fcfdfe37010) returning 1
ra[44769.40c7fedfcf7f0000]: 00:34:25.543941 ArgusPopQueue (0x2873190)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.543954 ArgusPopQueue (0x2873850)
returning 0x7fcfdfe37010
ra[44769.40c7fedfcf7f0000]: 00:34:25.543978 ArgusAddToQueue (0x2873190,
0x7fcfdfe37010) returning 1
ra[44769.40c7fedfcf7f0000]: 00:34:25.543991 ArgusPopQueue (0x2873850)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544016 ArgusPopQueue (0x2873850)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544042 ArgusFree (0x2873850)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544055 ArgusDeleteQueue (0x2873850)
returning
ra[44769.40c7fedfcf7f0000]: 00:34:25.544072 ArgusShutDown (0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544095 ArgusPopQueue (0x2873190)
returning 0x7fcfdfe37010
ra[44769.40c7fedfcf7f0000]: 00:34:25.544128 ArgusFree (0x7fcfdfe37010)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544146 ArgusPopQueue (0x2873190)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544162 ArgusFree (0x2873190)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544175 ArgusDeleteQueue (0x2873190)
returning
ra[44769.40c7fedfcf7f0000]: 00:34:25.544190 ArgusPopQueue (0x28731f0)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544206 ArgusPopQueue (0x28731f0)
returning (nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544222 ArgusFree (0x28731f0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544235 ArgusDeleteQueue (0x28731f0)
returning
ra[44769.40c7fedfcf7f0000]: 00:34:25.544249 RaParseComplete(caught signal 0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544263 ArgusWindowClose () returning
ra[44769.40c7fedfcf7f0000]: 00:34:25.544278 RaParseComplete(caught signal 0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544293 RaParseComplete(caught signal 0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544308 ArgusShutDown (0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544324 ArgusPopQueue ((nil)) returning
(nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544339 ArgusPopQueue ((nil)) returning
(nil)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544649 ArgusFree (0x2873050)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544667 ArgusDeleteList (0x2873050, 4)
returning
ra[44769.40c7fedfcf7f0000]: 00:34:25.544684 ArgusFree (0x28730f0)
ra[44769.40c7fedfcf7f0000]: 00:34:25.544699 ArgusDeleteList (0x28730f0, 4)
returning
************************************************************
*************************


************************************************************
*************************
ra -D10 -S cisco://192.xx.xx.xx:9995  #Runs but doesn't seem to do
anything, here is more verbose output:

ra[44711.40a7f378ef7f0000]: 00:25:53.173465 ArgusCalloc (1, 461728)
returning 0x78d85010
ra[44711.40a7f378ef7f0000]: 00:25:53.173544 ArgusAddToQueue (0x16fb190,
0x7fef78d85010) returning 1
ra[44711.40a7f378ef7f0000]: 00:25:53.173563 ArgusAddHostList (0x78df6010,
cisco://192.xx.xx.xx:9995, 16, 17) returning 1
ra[44711.40a7f378ef7f0000]: 00:25:53.173602 main: reading files completed
ra[44711.40a7f378ef7f0000]: 00:25:53.173620 ArgusCalloc (1, 80) returning
0x16fb850
ra[44711.40a7f378ef7f0000]: 00:25:53.173636 ArgusNewQueue () returning
0x16fb850
ra[44711.40a7f378ef7f0000]: 00:25:53.173653 ArgusPopQueue (0x16fb190)
returning 0x7fef78d85010
ra[44711]: 00:25:53.174131 Binding 192.xx.xx.xx:9995 Expecting Netflow
records
ra[44711.40a7f378ef7f0000]: 00:25:53.174318 receiving
ra[44711.40a7f378ef7f0000]: 00:25:53.174339 ArgusGetServerSocket
(0x7fef78d85010) returning 3
ra[44711.40a7f378ef7f0000]: 00:25:53.174380 ArgusCalloc (1, 4194304)
returning 0x77c02010
ra[44711.40a7f378ef7f0000]: 00:25:53.174413 ArgusCalloc (1, 262144)
returning 0x77bc1010
ra[44711.40a7f378ef7f0000]: 00:25:53.192504 ArgusInitAddrtoname
(0x7fef78df6010, 0x0, 0x0)
ra[44711.40a7f378ef7f0000]: 00:25:53.192540 ArgusParseInit(0x7fef78df6010
0x7fef78d85010
ra[44711.40a7f378ef7f0000]: 00:25:53.192553 ArgusReadConnection(0x78d85010,
2) reading cisco wire format
ra[44711.40a7f378ef7f0000]: 00:25:53.192566 ArgusReadConnection(0x78d85010,
2) returning 0
ra[44711.40a7f378ef7f0000]: 00:25:53.192620 ArgusHandleRecord
(0x7fef78d85228, 0x7fef78f17808) returning -1
ra[44711.40a7f378ef7f0000]: 00:25:53.192634 ArgusAddToQueue (0x16fb1f0,
0x7fef78d85010) returning 1
ra[44711.40a7f378ef7f0000]: 00:25:53.192661 ArgusPopQueue (0x16fb190)
returning (nil)
ra[44711.40a7f378ef7f0000]: 00:25:53.192674 ArgusPopQueue (0x16fb850)
returning (nil)
ra[44711.40a7f378ef7f0000]: 00:25:53.192687 ArgusPopQueue (0x16fb850)
returning (nil)
ra[44711.40a7f378ef7f0000]: 00:25:53.192699 ArgusFree (0x16fb850)
ra[44711.40a7f378ef7f0000]: 00:25:53.192711 ArgusDeleteQueue (0x16fb850)
returning
ra[44711.40a7f378ef7f0000]: 00:25:53.192736 ArgusReadStream(0x7fef78df6010)
starting
ra[44711.40a7f378ef7f0000]: 00:25:53.443019 ArgusAdjustGlobalTime real
1508905553.443019 global 1508905553.443019
ra[44711.40a7f378ef7f0000]: 00:25:53.693329 ArgusAdjustGlobalTime real
1508905553.693329 global 1508905553.693329
ra[44711.40a7f378ef7f0000]: 00:25:53.943634 ArgusAdjustGlobalTime real
1508905553.943634 global 1508905553.943634
ra[44711.40a7f378ef7f0000]: 00:25:54.193936 ArgusAdjustGlobalTime real
1508905554.193935 global 1508905554.193935
ra[44711.40a7f378ef7f0000]: 00:25:54.444239 ArgusAdjustGlobalTime real
1508905554.444239 global 1508905554.444239
ra[44711.40a7f378ef7f0000]: 00:25:54.444276 ArgusClientTimeout()
ra[44711.40a7f378ef7f0000]: 00:25:54.694571 ArgusAdjustGlobalTime real
1508905554.694571 global 1508905554.694571
ra[44711.40a7f378ef7f0000]: 00:25:54.944887 ArgusAdjustGlobalTime real
1508905554.944887 global 1508905554.944887
ra[44711.40a7f378ef7f0000]: 00:25:55.195191 ArgusAdjustGlobalTime real
1508905555.195191 global 1508905555.195191
ra[44711.40a7f378ef7f0000]: 00:25:55.445495 ArgusAdjustGlobalTime real
1508905555.445495 global 1508905555.445495
ra[44711.40a7f378ef7f0000]: 00:25:55.445570 ArgusClientTimeout()
ra[44711.40a7f378ef7f0000]: 00:25:55.695858 ArgusAdjustGlobalTime real
1508905555.695858 global 1508905555.695858
ra[44711.40a7f378ef7f0000]: 00:25:55.946168 ArgusAdjustGlobalTime real
1508905555.946168 global 1508905555.946168
ra[44711.40a7f378ef7f0000]: 00:25:56.196477 ArgusAdjustGlobalTime real
1508905556.196476 global 1508905556.196476
ra[44711.40a7f378ef7f0000]: 00:25:56.446785 ArgusAdjustGlobalTime real
1508905556.446785 global 1508905556.446785
ra[44711.40a7f378ef7f0000]: 00:25:56.446822 ArgusClientTimeout()
^Cra[44711.40a7f378ef7f0000]: 00:25:56.531281 ArgusShutDown (2)
************************************************************
****************************


Thank you,

-Drew



> Carter
>
>
>
> On Oct 24, 2017, at 5:47 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>
> Thanks Carter, I re-compiled as you directed and now have debug
> functionality running argus-clients-3.0.8.2...still seeing IPFIX data
> coming in via 192.xx.xx.xx:9995 (UDP) in a steady stream via tcpdump.
>
> I ran radium again with debug set to 6 and using the config file
> parameters I mentioned earlier in the thread.  I keep getting repeated
> "ArgusClientTimeout()" after radium binds to the 192.xx.xx.xx:9995 network
> interface on the same server to listen for/collect the IPFIX data.  I also
> tried running ra as you suggested, with ipfix:// ra just exits immediately,
> with cisco:// ra also repeatedly gives me "ArgusClientTimeout()" after
> binding to the server NIC's IP on udp/9995...all debug log results are
> below (level 6).
>
> Not certain but guessing this means it's not receiving/recognizing any of
> the incoming IPFIX data or something else?
>
>
> ***********************************************************************
> Oct 24 17:20:54 argus1 systemd: Started radium Service.
> Oct 24 17:20:54 argus1 systemd: Starting radium Service...
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582224 ArgusCalloc (1, 560) returning 0x3113e80
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582308 ArgusCalloc (1, 112) returning 0x31137c0
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582319 ArgusCalloc (1, 80) returning 0x3113840
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582327 ArgusNewQueue () returning 0x3113840
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582336 ArgusCalloc (1, 56) returning 0x31138a0
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582349 ArgusCalloc (65536, 8) returning 0xe0bd8010
> Oct 24 17:20:54 argus1 radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582357 ArgusNewHashTable (65536) returning 0x31138a0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582566 setArgusMarReportInterval(15) returning
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582854 ArgusCalloc (1, 461728) returning 0xdfdf5010
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582894 ArgusAddHostList (0xe0c59010, 192.xx.xx.xx:9995, 16, 17)
> returning 1
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582929 RadiumParseResourceFile: ArgusBindAddr "127.0.0.1"
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582959 ArgusCalloc (1, 144) returning 0x3114390
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.582990 ArgusNewList () returning 0x3114390
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.583032 ArgusCalloc (1, 296) returning 0x3114460
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.583060 ArgusPushFrontList (0x3114390, 0x3114460, 1) returning
> 0x3114970
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.583538 RadiumParseResourceFile (/etc/radium.conf) returning 0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.583589 ArgusEstablishListen(0xe0c59010, 561, 127.0.0.1) binding:
> 127.0.0.1:561
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584032 ArgusEstablishListen(0xe0c59010, 561, 127.0.0.1)
> returning 3
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584073 ArgusCalloc (1, 392) returning 0x31140c0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584102 ArgusCalloc (1, 80) returning 0x3114b40
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584129 ArgusNewQueue () returning 0x3114b40
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584153 ArgusNewOutput() returning retn 0x31140c0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584178 ArgusCalloc (1, 144) returning 0x3114250
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584202 ArgusNewList () returning 0x3114250
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584227 ArgusCalloc (1, 128) returning 0x3115da0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584251 ArgusGenerateInitialMar() returning
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584282 ArgusCalloc (1, 168) returning 0x3115e30
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584355 ArgusCalloc (1, 65792) returning 0x3115ee0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584388 ArgusCalloc (1, 144) returning 0x3125ff0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584412 ArgusNewList () returning 0x3125ff0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584436 ArgusNewSocket (4) returning 0x3115ee0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584479 ArgusPushBackList (0x3114390, 0x3114460, 1) returning 1
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.584965 ArgusInitOutput() done
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585011 main: reading files completed
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585040 ArgusCalloc (1, 80) returning 0x31262d0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585065 ArgusNewQueue () returning 0x31262d0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585121 ArgusFree (0x31262d0)
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585149 ArgusDeleteQueue (0x31262d0) returning
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:54.585174 ArgusReadStream(0x7fc7e0c59010) starting
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.585224 ArgusConnectRemote(0x7fc7dfdf5010) starting
> Oct 24 17:20:54 argus1 radium: radium[41360]: 17:20:54.585349 Binding
> 192.xx.xx.xx:9995 Expecting Netflow records
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.585433 receiving
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.585464 ArgusGetServerSocket (0x7fc7dfdf5010) returning 5
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.585513 ArgusCalloc (1, 4194304) returning 0xce7de010
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.586086 ArgusCalloc (1, 262144) returning 0xce79d010
> Oct 24 17:20:54 argus1 radium[41360]: 17:20:54.585349 Binding
> 192.xx.xx.xx:9995 Expecting Netflow records
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602162 ArgusInitAddrtoname (0x7fc7e0c59010, 0x0, 0x0)
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602202 ArgusParseInit(0x7fc7e0c59010 0x7fc7dfdf5010
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602223 ArgusReadConnection(0xdfdf5010, 2) reading cisco wire
> format
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602245 ArgusReadConnection(0xdfdf5010, 2) returning 0
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602265 ArgusConnectRemote(0x7fc7dfdf5010) connected to
> 192.xx.xx.xx
> Oct 24 17:20:54 argus1 radium: radium: radium[41360.00f73dcfc77f0000]:
> 17:20:54.602285 ArgusConnectRemote() done!
> Oct 24 17:20:55 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:55.636383 ArgusClientTimeout()
> Oct 24 17:20:56 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:56.637568 ArgusClientTimeout()
> Oct 24 17:20:57 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:57.638727 ArgusClientTimeout()
> Oct 24 17:20:58 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:58.639887 ArgusClientTimeout()
> Oct 24 17:20:59 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:20:59.641141 ArgusClientTimeout()
> Oct 24 17:21:00 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:21:00.642402 ArgusClientTimeout()
> Oct 24 17:21:01 argus1 radium: radium: radium[41360.40d7d9e0c77f0000]:
> 17:21:01.643565 ArgusClientTimeout()
> Oct 24 17:21:01 argus1 systemd: Stopping radium Service...
> ***********************************************************************
>
>
> ra using ipfix://
> ***********************************************************************
> # ra -D6 -S ipfix://192.xx.xx.xx:9995
>
> ra[41491.4037f00ea37f0000]: 17:32:00.930991 ArgusCalloc (1, 461728)
> returning 0xed4e010
> ra[41491.4037f00ea37f0000]: 17:32:00.931065 ArgusAddHostList (0xedbf010,
> ipfix://192.xx.xx.xx:9995, 64, 6) returning 1
> ra[41491.4037f00ea37f0000]: 17:32:00.931091 main: reading files completed
> ra[41491.4037f00ea37f0000]: 17:32:00.931136 ArgusCalloc (1, 80) returning
> 0x1545840
> ra[41491.4037f00ea37f0000]: 17:32:00.931157 ArgusNewQueue () returning
> 0x1545840
> ra[41491.4037f00ea37f0000]: 17:32:00.931185 ArgusGetServerSocket
> (0x7fa30ed4e010) returning -1
> ra[41491.4037f00ea37f0000]: 17:32:00.931217 ArgusFree (0x1545840)
> ra[41491.4037f00ea37f0000]: 17:32:00.931237 ArgusDeleteQueue (0x1545840)
> returning
> ra[41491.4037f00ea37f0000]: 17:32:00.931269 ArgusShutDown (0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931298 ArgusFree (0x7fa30ed4e010)
> ra[41491.4037f00ea37f0000]: 17:32:00.931319 ArgusFree (0x1545190)
> ra[41491.4037f00ea37f0000]: 17:32:00.931336 ArgusDeleteQueue (0x1545190)
> returning
> ra[41491.4037f00ea37f0000]: 17:32:00.931355 ArgusFree (0x15451f0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931371 ArgusDeleteQueue (0x15451f0)
> returning
> ra[41491.4037f00ea37f0000]: 17:32:00.931389 RaParseComplete(caught signal
> 0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931404 ArgusWindowClose () returning
> ra[41491.4037f00ea37f0000]: 17:32:00.931420 RaParseComplete(caught signal
> 0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931439 RaParseComplete(caught signal
> 0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931457 ArgusShutDown (0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931789 ArgusFree (0x1545050)
> ra[41491.4037f00ea37f0000]: 17:32:00.931808 ArgusDeleteList (0x1545050, 4)
> returning
> ra[41491.4037f00ea37f0000]: 17:32:00.931829 ArgusFree (0x15450f0)
> ra[41491.4037f00ea37f0000]: 17:32:00.931842 ArgusDeleteList (0x15450f0, 4)
> returning
> ***********************************************************************
>
> ra using cisco://
> ***********************************************************************
> # ra -D6 -S cisco://192.xx.xx.xx:9995
> ra[41498.407743631c7f0000]: 17:33:07.958512 ArgusCalloc (1, 461728)
> returning 0x63282010
> ra[41498.407743631c7f0000]: 17:33:07.958588 ArgusAddHostList (0x632f3010,
> cisco://192.xx.xx.xx:9995, 16, 17) returning 1
> ra[41498.407743631c7f0000]: 17:33:07.958612 main: reading files completed
> ra[41498.407743631c7f0000]: 17:33:07.958633 ArgusCalloc (1, 80) returning
> 0x18dc840
> ra[41498.407743631c7f0000]: 17:33:07.958664 ArgusNewQueue () returning
> 0x18dc840
> ra[41498]: 17:33:07.958737 Binding 192.xx.xx.xx:9995 Expecting Netflow
> records
> ra[41498.407743631c7f0000]: 17:33:07.959292 receiving
> ra[41498.407743631c7f0000]: 17:33:07.959312 ArgusGetServerSocket
> (0x7f1c63282010) returning 3
> ra[41498.407743631c7f0000]: 17:33:07.959354 ArgusCalloc (1, 4194304)
> returning 0x620ff010
> ra[41498.407743631c7f0000]: 17:33:07.959384 ArgusCalloc (1, 262144)
> returning 0x620be010
> ra[41498.407743631c7f0000]: 17:33:07.977130 ArgusInitAddrtoname
> (0x7f1c632f3010, 0x0, 0x0)
> ra[41498.407743631c7f0000]: 17:33:07.977161 ArgusParseInit(0x7f1c632f3010
> 0x7f1c63282010
> ra[41498.407743631c7f0000]: 17:33:07.977174 ArgusReadConnection(0x63282010,
> 2) reading cisco wire format
> ra[41498.407743631c7f0000]: 17:33:07.977186 ArgusReadConnection(0x63282010,
> 2) returning 0
> ra[41498.407743631c7f0000]: 17:33:07.977209 ArgusHandleRecord
> (0x7f1c63282228, 0x7f1c63414808) returning -1
> ra[41498.407743631c7f0000]: 17:33:07.977227 ArgusFree (0x18dc840)
> ra[41498.407743631c7f0000]: 17:33:07.977240 ArgusDeleteQueue (0x18dc840)
> returning
> ra[41498.407743631c7f0000]: 17:33:07.977253 ArgusReadStream(0x7f1c632f3010)
> starting
> ra[41498.407743631c7f0000]: 17:33:09.228565 ArgusClientTimeout()
> ra[41498.407743631c7f0000]: 17:33:10.229714 ArgusClientTimeout()
> ra[41498.407743631c7f0000]: 17:33:11.230838 ArgusClientTimeout()
> ***********************************************************************
>
> Thank you,
>
> -Drew
>
> On Tue, Oct 24, 2017 at 3:45 PM, Carter Bullard <carter at qosient.com>
> wrote:
>
>> When you run the ./configure program, like many open source distros, you
>> can create files, like .debug or .devel,  that ./configure looks for, and
>> if they exist, it will configure the package to add specific
>> functionality.  If there is a ./.debug file, the ./conifgure script will
>> compile in support for printing debug information on the command line ….
>> Try this …
>>
>>    % cd /full/path/to/your/argus/clients/distribution
>>    % touch .debug
>>    % ./configure
>>    % make clean; make
>>
>> Now all your ra* programs will be able to print detail debugging
>> information, using the “-D debuglevel” option on the command line.
>> You can also turn on this type of support like this …
>>
>>    % ./configure —enable-debug=yes
>>
>> I prefer the .debug file, as you can set it and forget that you need the
>> switch on the configure line.
>> Please keep all email on the mailing list … Thanks...
>>
>> Carter
>>
>>
>>
>> On Oct 24, 2017, at 3:33 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>>
>> Whoops...apologies with that- I noticed it said on the downloads page
>> that 3.0.8.1 was the current stable version and never thought I had a need
>> to look on the main page.
>>
>> Please forgive my ignorance but I'm not 100% sure what you mean by "with
>> the .debug tag" I've listed out my options using ./configure --help but
>> don't see anything related to enabling debugging.  Only thing that I can
>> see is  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes] is this what
>> you're referring to?
>>
>> --enable-debug=yes ?
>>
>> Or somehow with referencing the files below?
>>
>> /argus-clients-3.0.8.2/common/._argus_debug.h
>> /argus-clients-3.0.8.2/common/argus_debug.h
>> /argus-clients-3.0.8.2/include/._argus_debug.h
>> /argus-clients-3.0.8.2/include/argus_debug.h
>>
>> I'm not certain on how to compile to enable debugging properly in the
>> fashion you've described.
>>
>> I very much appreciate your help.
>>
>> Sincerely,
>>
>> -Drew
>>
>> On Tue, Oct 24, 2017 at 3:09 PM, Carter Bullard <carter at qosient.com>
>> wrote:
>>
>>> How about the link to argus-clients-3.0.8.2 from the main page …
>>>
>>> http://qosient.com/argus/
>>>
>>> You should ./configure and compile with the .debug tag in the home
>>> directory, and then run your ra command with the -D4 or -D5 option to see
>>> if you are receiving any packets.
>>> Carter
>>>
>>> [image: QoSient] <http://qosient.com/>
>>> Carter Bullard <carter at qosient.com> • CTO
>>> 150 E 57th Street, Suite 12D
>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York&entry=gmail&source=g>
>>> New York, New York
>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York&entry=gmail&source=g>
>>> 10022-2795
>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>
>>>
>>>
>>> On Oct 24, 2017, at 2:15 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>>>
>>> I'm using the latest stable version (argus-clients-3.0.8) downloaded
>>> from the website (http://www.qosient.com/argus/downloads.shtml).
>>>
>>> That command doesn't appear to be working upon running it, it appears to
>>> just immediately exit and throw me back into a command prompt.
>>>
>>> In comparison if I run the following:
>>>
>>> ra -S cisco://192.xx.xx.xx:9995
>>>
>>> It appears to stay running and sits on a new line with a flashing cursor
>>> waiting to print something possibly?  But nothing is printed to stdout.
>>>
>>> Thank you,
>>>
>>> -Drew
>>>
>>> On Tue, Oct 24, 2017 at 1:51 PM, Carter Bullard <carter at qosient.com>
>>> wrote:
>>>
>>>> Not sure what version you are using, but you should be able to read
>>>> IPFIX data from an ra* program using the same strategy as radium.
>>>> Does this work ???
>>>>
>>>>    ra -S ipfix://192.xx.xx.xx:9995
>>>>
>>>> Carter
>>>>
>>>>
>>>> On Oct 24, 2017, at 1:25 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>>>>
>>>> For us near real-time processing isn't really an issue- so at least for
>>>> now I'll be aiming to leverage the second approach you've described which
>>>> also sounds to be the most common with the scenario we are in at the
>>>> moment.  I've been reading/studying the radium/rastream/racluster
>>>> documentation/man pages etc. but still have questions related to
>>>> configuration where there are gaps.
>>>>
>>>> Being that this approach will be collecting IPFIX data directed to my
>>>> server which will be running radium|rastream|racluster etc.  I'm hoping
>>>> someone can help me better understand/confirm the best configuration
>>>> options combination available via radium in order to do so?  I may be
>>>> overthinking/over-complicating this a bit but my main confusion is
>>>> surrounding how to tell radium to listen for that IPFIX data properly.  I'm
>>>> getting incoming data on udp/9995 which is confirmed via tcpdump...
>>>>
>>>> I'm currently telling radium to listen for what the docs refer to as
>>>> Cisco Netflow (cmdline -C) but I am using radium.conf and the current
>>>> settings I'm using in the config file are as follows:
>>>>
>>>> RADIUM_DAEMON=yes
>>>> RADIUM_DEBUG_LEVEL=0
>>>> RADIUM_MAR_STATUS_INTERVAL=60
>>>> RADIUM_CISCONETFLOW_PORT=192.xx.xx.xx:9995
>>>> RADIUM_ACCESS_PORT=561
>>>> RADIUM_BIND_IP=127.0.0.1
>>>> RADIUM_OUTPUT_FILE=/flowdata/radium/radium.out
>>>> RADIUM_SETUSER_ID=argus
>>>> RADIUM_SETGROUP_ID=argus
>>>>
>>>> Invoked with.... radium -f /etc/radium.conf
>>>>
>>>> Does radium log somewhere other than /var/log/messages?  Is it required
>>>> to manually specify the ability to generate debug logs when compiling for
>>>> that to be available? I'm not seeing anything mentioning that specifically
>>>> via ./configure --help for argus-clients.
>>>>
>>>> I'm trying to test that radium is processing the data I've confirmed is
>>>> coming via udp/9995 using rastream like:
>>>>
>>>> rastream -S 127.0.0.01:561 <http://127.0.0.1:561/> -w -
>>>>
>>>> When I invoke rastream in this fashion (as root from a root shell)
>>>> should I not see the converted argus formatted data being spewed out to
>>>> stdout in unidirectional argus format?
>>>>
>>>> I do see the log entries from radium confirming the connection from
>>>> rastream, does this look normal?:
>>>>
>>>> Oct 24 12:56:16  radium: radium[14196]: 12:56:16.271560 connect from
>>>> localhost[127.0.0.1]
>>>> Oct 24 12:56:16  radium[14196]: 12:56:16.271560 connect from
>>>> localhost[127.0.0.1]
>>>> Oct 24 12:56:30  radium: radium[14196]: 12:56:30.781854
>>>> ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
>>>> Oct 24 12:56:30  radium[14196]: 12:56:30.781854
>>>> ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
>>>>
>>>> However, I'm not seeing anything being printed out to stdout though,
>>>> also the output file I've configured via radium doesn't seem to be doing
>>>> much (as far as I can tell at least), it's incrementing in size but not
>>>> nearly to the degree I would expect but then again it does appear to be a
>>>> DBase 3 data file which appears to be a database that is showing a lot of
>>>> records already according to the output of the file command:
>>>>
>>>> $ file /flowdata/radium/radium.out
>>>> /flowdata/radium/radium.out: DBase 3 data file with memo(s) (2097152
>>>> records)
>>>>
>>>> Should I be using another instance of radium to test the incoming data
>>>> is being processed rather than trying to use rastream for that?  Does this
>>>> make sense and/or is that doable?
>>>>
>>>> Still very much learning all these tools- help is greatly appreciated,
>>>> many thanks in advance.
>>>>
>>>> -Drew
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Carter Bullard <carter at qosient.com>
>>>> Date: Tue, Oct 17, 2017 at 10:10 AM
>>>> Subject: Re: [ARGUS] Argus & IPFIX?
>>>> To: Drew Dixon <dwdixon at umich.edu>
>>>>
>>>>
>>>> Hey Drew,
>>>>
>>>> The open source clients have all the basic functionality to build a
>>>> system that can handle complex asymmetric monitoring problems, but they not
>>>> designed specifically to do this, nor are they designed to go very fast...
>>>>
>>>> If you have any open source argus questions, please send them to the
>>>> public mailing list.
>>>>
>>>> If you want to merge asymmetric flows from multiple sources, the best
>>>> approach is going to be dependent on how quickly you want to know about
>>>> issues on the wire.  If time is important, you will want to use radium() to
>>>> collect real-time streaming data, and then use programs like rabins() to
>>>> aggregate the uni-directional streams into bi-directional streams, and keep
>>>> the processing pipeline going.  There are time issues to work out, but the
>>>> concepts in radium and rabins are what are needed to provide near-realtime
>>>> multi-source aggregation.
>>>>
>>>> If near real-time processing is not an issue, then having radium
>>>> collect the IPFIX data, covert it to argus data and use a program like
>>>> rastream() to break the stream up into files, and then post process the
>>>> files with racluster() to merge the records to form the bi-directional flow
>>>> data.  That is the most common way of doing what you’ve described.  The
>>>> open source racluster() is a bit of a memory hog, so you’ll want give your
>>>> system a bit of RAM, and then tune the process to handle the number of
>>>> flows that you are processing.  So far in our experience, 100G doesn’t
>>>> significantly increase the numbers of simultaneous flows, but rather
>>>> supports bigger elephants co-existing with the traditional mix of data
>>>> (research university networks).  You may find that in your specific
>>>> network, that the flow data processing requirements aren’t much different
>>>> than 10G, but …, then again you may be more like a wireless cellular
>>>> network, where 100G really means 10x the flows seen at 10G … just all
>>>> depends on your network design and customer base.
>>>>
>>>> Good luck, and keep us posted ….
>>>> Carter
>>>>
>>>> [image: QoSient]        <http://qosient.com/>
>>>> Carter Bullard  <carter at qosient.com>• CTO
>>>> 150 E 57th Street, Suite 12D
>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>> New York, New York 10022
>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>> -2795
>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>
>>>>
>>>> On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <carter at qosient.com>
>>>> wrote:
>>>>
>>>>> Hey Drew,
>>>>> Argus should be able to read most/any IPFIX TCP/UDP data source, at
>>>>> least that is the goal.  To that end, if you have some IPFIX data that the
>>>>> ra* programs can’t read, I’ll spend some time making it work.  So if your
>>>>> using Juniper, have it export UDP IPFIX, and we should be able to read
>>>>> them, as the router advertises the templates in a reasonable timeframe, as
>>>>> we need to see the templates before we can decode the records (really
>>>>> terrible design flaw).
>>>>>
>>>>> We, of course recommend that you generate your own flow records rather
>>>>> than read from integrated IPFIX, especially if you’re network is going
>>>>> particularly fast.  QoSient has 1g, 10g, 40g and 100g argus sensor
>>>>> appliances for sale, so if you’re looking to do the do for real, think
>>>>> about generating your own data.
>>>>>
>>>>> Hope all is most excellent,
>>>>> Carter
>>>>>
>>>>> [image: QoSient] <http://qosient.com/>
>>>>> Carter Bullard <carter at qosient.com> • CTO
>>>>> 150 E 57th Street, Suite 12D
>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>> New York, New York 10022
>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>> -2795
>>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>>
>>>>>
>>>>>
>>>>> On Oct 16, 2017, at 11:18 AM, Drew Dixon <dwdixon at umich.edu> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm wondering what the current status of Argus' support of reading
>>>>> IPFIX and if there might be any relevant information/updates on that front
>>>>> which someone could share?
>>>>>
>>>>> I did some quick searching online and see mention of IPFIX in relation
>>>>> to Argus but nothing really stating that it's officially supported at this
>>>>> time etc.
>>>>>
>>>>> Thank you!
>>>>>
>>>>> -Drew
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20171025/641130b8/attachment.html>

More information about the argus mailing list