Multi-Instanced Argus

[Reynolds, Jeffrey]

First, before we dive into to it too deep, how is the performance ??

This actually seems like a great place to start.  Before getting too heavy into PF_RING integration, maybe I should offer a bit of backstory.  Our main goal is just to archive traffic.  We have a server running CentOS 6 that receives traffic from two SPAN ports.  The only thing we want to accomplish is to maintain a copy of that traffic for some period of time.  Argus was used because it seemed to be the best tool for the price, and it comes with a lot of great features that while we may not use now, we may use later (again, for right now all we want is a copy of the traffic to be able to perform forensics on).

Now, I put up a single instance of Argus and pointed it at the interface that was the master of our two bonded physical NICs (eth0 and eth1 are bonded to bond0).  I let it run for an hour to get some preliminary numbers.  I ran an recount against my output file and got the following stats:

racount -t 2014y3m12d05h -r argus-out
racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
sum 14236180 187526800 98831765 88695035 212079839908 102889789820 109190050088

However, the switch the switch sending that traffic reported that it had sent a total of 421,978,297 packets to both interfaces, and a total of 371,307,051,815 bytes for that time frame.  I could be interpreting something incorrectly, so maybe the best first thing for me to confirm is that we are in fact losing a lot of traffic.  But it seems that a single argus instance can’t keep up with the traffic.  I’ve seen this happen with Snort, and our solution was to plug Snort into PF_RING to allow the traffic to be intelligently forwarded via the Snort Data Acquisition Library (DAQ).  From the perspective of someone who hasn’t had a lot of exposure to this level of hardware configuration, it was relatively easy to plug the configuration parameters in at the Snort command line to have them all point at the same traffic source so that each individual process didn’t run through the same traffic.  My hope was that there might just be some parameters to set within the argus.conf file which would tell each process to pull from a single PF_RING source.  However, it looks like this might not be as easy as I had once thought.

Am I on the right track or does this make even a little sense?

Thanks,

Jeff



From: Carter Bullard <carter at qosient.com<mailto:carter at qosient.com>>
Date: Wednesday, March 12, 2014 at 9:54 AM
To: "Reynolds, Jeffrey" <JReynolds at utdallas.edu<mailto:JReynolds at utdallas.edu>>
Cc: Argus <argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>>
Subject: Re: [ARGUS] Multi-Instanced Argus

Hey Jeffrey,
I am very interested in this approach, but I have no experience with this PF_RING feature, so I’ll have to give you the "design response”.  Hopefully, we can get this to where its doing exactly what anyone would want it to do, and get us a really fast argus, on the cheap.

First, before we dive into to it too deep, how is the performance ??  Are you getting bi-directional flows out of this scheme ??  Are you seeing all the traffic ???  If so, then congratulations !!!  If the performance is good, your seeing all the traffic, but you’re only getting uni-directional flows, then we may have some work to do, but still congratulations !!!  If you’re not getting all the traffic then we have some real work to do, as one of the purposes of argus is to monitor all the traffic.

OK, so my understanding is that the PF_RING can do some packet routing to a non-overlapping set of tap interfaces.  Routing is based on some classification scheme, designed to make this usable. The purpose is to provide coarse grain parallelism for packet processing.  The idea, as much as I can tell, is to prevent multiple readers from having to read from the same queue; eliminating locking issues, which kills performance etc…

So, I’m not sure what you mean by “pulling from the same queue”.  If you do have multiple argi reading the same packet, you will end up counting a single packet multiple times.  Not a terrible thing, but not recommended.  Its not that you’re creating multiple observation domains using this PF_RING technique.  You’re really splitting a single packet observation domain into a multi-sensor facility … eventually you will want to combine the total argus output into a single output stream, that represents the single packet observation domain.  At least that is my thinking, and I would recommend that you use radium to connect to all of your argus instances, rather than writing the argus output to a set of files.  Radium will generate a single argus data output stream, representing the argus data from the single observation domain.

The design issue of using the PF_RING function is “how is PF_RING classifying packets to do the routing?”.
We would like for it to send packets that belong to the same bi-directional flow to the same virtual interface, so argus can do its bi-directional thing.  PF_RING claims that you can provide your own classifier logic, which we can do to make this happen.  We have a pretty fast bidirectional hashing scheme which we can try out.

We have a number of people that are using netmap instead of PF_RING.  My understanding is that it also has this same type of feature.  If we can get some people talking about that, that would help a bit.

Carter



On Mar 12, 2014, at 1:03 AM, Reynolds, Jeffrey <JReynolds at utdallas.edu<mailto:JReynolds at utdallas.edu>> wrote:

Howdy All,

So after forever and a day, I’ve finally found time to start working on my multi-instanced argus configuration.  Here is my setup:

-CentOS 6.5 x64
-pfring driver compiled from source
-pfring capable Intel NICs (currently using the ixgbe driver version 3.15.1-k)
(these NICs are in a bonded configuration under a device named bond0)

I’ve configured my startup script to start 5 instances of Argus, each with there own /etc/argusX.conf file (argus1.conf, argus2.conf, etc).  The start up script correctly assigns the proper pid file to each instance, and everything starts and stops smoothly.  Each instance is writing an output file to /var/argus in the format of argusX.out.  When I first tried running my argus instances, I ran them with a version of PF_RING I had installed from an RPM obtained from the ntop repo.  Things didn’t seem to work correctly, so I tried again after I had compiled from source.  After compiling from source, I got the following output in /var/log/messages when I started argus:

Mar 11 17:48:16 argus kernel: No module found in object
Mar 11 17:49:16 argus kernel: [PF_RING] Welcome to PF_RING 5.6.3 ($Revision: 7358$)
Mar 11 17:49:16 argus kernel: (C) 2004-14 ntop.org<http://ntop.org>
Mar 11 17:49:16 argus kernel: [PF_RING] registered /proc/net/pf_ring/
Mar 11 17:49:16 argus kernel: NET: Registered protocol family 27
Mar 11 17:49:16 argus kernel: [PF_RING] Min # ring slots 4096
Mar 11 17:49:16 argus kernel: [PF_RING] Slot version     15
Mar 11 17:49:16 argus kernel: [PF_RING] Capture TX       Yes [RX+TX]
Mar 11 17:49:16 argus kernel: [PF_RING] Transparent Mode 0
Mar 11 17:49:16 argus kernel: [PF_RING] IP Defragment    No
Mar 11 17:49:16 argus kernel: [PF_RING] Initialized correctly
Mar 11 17:49:35 argus kernel: Bluetooth: Core ver 2.15
Mar 11 17:49:35 argus kernel: NET: Registered protocol family 31
Mar 11 17:49:35 argus kernel: Bluetooth: HCI device and connection manager initialized
Mar 11 17:49:35 argus kernel: Bluetooth: HCI socket layer initialized
Mar 11 17:49:35 argus kernel: Netfilter messages via NETLINK v0.30.
Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.643243 started
Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.693930 started
Mar 11 17:49:35 argus kernel: device bond0 entered promiscuous mode
Mar 11 17:49:35 argus kernel: device em1 entered promiscuous mode
Mar 11 17:49:35 argus kernel: device em2 entered promiscuous mode
Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.721490 ArgusGetInterfaceStatus: interface bond0 is up
Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.349202 started
Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.364625 started
Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.383623 ArgusGetInterfaceStatus: interface bond0 is up
Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.045224 started
Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.060689 started
Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.079706 ArgusGetInterfaceStatus: interface bond0 is up
Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.753278 started
Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.768613 started
Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.785691 ArgusGetInterfaceStatus: interface bond0 is up
Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.449229 started
Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.466365 started
Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.485675 ArgusGetInterfaceStatus: interface bond0 is up

Aside from the “No module found in object” error, everything seems like its working Ok.  The only problem is that I don’t seem to have my argus instances configured to pull traffic from the same queue.  In other words, I have five output files from five argus instances with like traffic in all of them.  I haven’t made any changes to my argus config files, aside from telling them to write to different locations and the name of the interface.  I know I’m missing something but I’m not quite sure what it is.  If someone might be able to tell me how to configure these five instances to pull from the same PF_RING queue, I’d be mighty obliged.  Let me know if I need to submit any additional information.

Thanks,

Jeff Reynolds