Multi-Instanced Argus

[Carter Bullard]

Hey Jeffrey,
I am very interested in this approach, but I have no experience with this PF_RING feature, so I’ll have to give you the "design response”.  Hopefully, we can get this to where its doing exactly what anyone would want it to do, and get us a really fast argus, on the cheap.

First, before we dive into to it too deep, how is the performance ??  Are you getting bi-directional flows out of this scheme ??  Are you seeing all the traffic ???  If so, then congratulations !!!  If the performance is good, your seeing all the traffic, but you’re only getting uni-directional flows, then we may have some work to do, but still congratulations !!!  If you’re not getting all the traffic then we have some real work to do, as one of the purposes of argus is to monitor all the traffic.

OK, so my understanding is that the PF_RING can do some packet routing to a non-overlapping set of tap interfaces.  Routing is based on some classification scheme, designed to make this usable. The purpose is to provide coarse grain parallelism for packet processing.  The idea, as much as I can tell, is to prevent multiple readers from having to read from the same queue; eliminating locking issues, which kills performance etc…

So, I’m not sure what you mean by “pulling from the same queue”.  If you do have multiple argi reading the same packet, you will end up counting a single packet multiple times.  Not a terrible thing, but not recommended.  Its not that you’re creating multiple observation domains using this PF_RING technique.  You’re really splitting a single packet observation domain into a multi-sensor facility … eventually you will want to combine the total argus output into a single output stream, that represents the single packet observation domain.  At least that is my thinking, and I would recommend that you use radium to connect to all of your argus instances, rather than writing the argus output to a set of files.  Radium will generate a single argus data output stream, representing the argus data from the single observation domain.

The design issue of using the PF_RING function is “how is PF_RING classifying packets to do the routing?”.
We would like for it to send packets that belong to the same bi-directional flow to the same virtual interface, so argus can do its bi-directional thing.  PF_RING claims that you can provide your own classifier logic, which we can do to make this happen.  We have a pretty fast bidirectional hashing scheme which we can try out.

We have a number of people that are using netmap instead of PF_RING.  My understanding is that it also has this same type of feature.  If we can get some people talking about that, that would help a bit.

Carter



On Mar 12, 2014, at 1:03 AM, Reynolds, Jeffrey <JReynolds at utdallas.edu> wrote:

> Howdy All,
> 
> So after forever and a day, I’ve finally found time to start working on my multi-instanced argus configuration.  Here is my setup:
> 
> -CentOS 6.5 x64
> -pfring driver compiled from source
> -pfring capable Intel NICs (currently using the ixgbe driver version 3.15.1-k)
> (these NICs are in a bonded configuration under a device named bond0)
> 
> I’ve configured my startup script to start 5 instances of Argus, each with there own /etc/argusX.conf file (argus1.conf, argus2.conf, etc).  The start up script correctly assigns the proper pid file to each instance, and everything starts and stops smoothly.  Each instance is writing an output file to /var/argus in the format of argusX.out.  When I first tried running my argus instances, I ran them with a version of PF_RING I had installed from an RPM obtained from the ntop repo.  Things didn’t seem to work correctly, so I tried again after I had compiled from source.  After compiling from source, I got the following output in /var/log/messages when I started argus:
> 
> Mar 11 17:48:16 argus kernel: No module found in object
> Mar 11 17:49:16 argus kernel: [PF_RING] Welcome to PF_RING 5.6.3 ($Revision: 7358$)
> Mar 11 17:49:16 argus kernel: (C) 2004-14 ntop.org
> Mar 11 17:49:16 argus kernel: [PF_RING] registered /proc/net/pf_ring/
> Mar 11 17:49:16 argus kernel: NET: Registered protocol family 27
> Mar 11 17:49:16 argus kernel: [PF_RING] Min # ring slots 4096
> Mar 11 17:49:16 argus kernel: [PF_RING] Slot version     15
> Mar 11 17:49:16 argus kernel: [PF_RING] Capture TX       Yes [RX+TX]
> Mar 11 17:49:16 argus kernel: [PF_RING] Transparent Mode 0
> Mar 11 17:49:16 argus kernel: [PF_RING] IP Defragment    No
> Mar 11 17:49:16 argus kernel: [PF_RING] Initialized correctly
> Mar 11 17:49:35 argus kernel: Bluetooth: Core ver 2.15
> Mar 11 17:49:35 argus kernel: NET: Registered protocol family 31
> Mar 11 17:49:35 argus kernel: Bluetooth: HCI device and connection manager initialized
> Mar 11 17:49:35 argus kernel: Bluetooth: HCI socket layer initialized
> Mar 11 17:49:35 argus kernel: Netfilter messages via NETLINK v0.30.
> Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.643243 started
> Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.693930 started
> Mar 11 17:49:35 argus kernel: device bond0 entered promiscuous mode
> Mar 11 17:49:35 argus kernel: device em1 entered promiscuous mode
> Mar 11 17:49:35 argus kernel: device em2 entered promiscuous mode
> Mar 11 17:49:35 argus argus[13918]: 11 Mar 14 17:49:35.721490 ArgusGetInterfaceStatus: interface bond0 is up
> Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.349202 started
> Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.364625 started
> Mar 11 17:49:36 argus argus[13922]: 11 Mar 14 17:49:36.383623 ArgusGetInterfaceStatus: interface bond0 is up
> Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.045224 started
> Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.060689 started
> Mar 11 17:49:37 argus argus[13926]: 11 Mar 14 17:49:37.079706 ArgusGetInterfaceStatus: interface bond0 is up
> Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.753278 started
> Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.768613 started
> Mar 11 17:49:37 argus argus[13930]: 11 Mar 14 17:49:37.785691 ArgusGetInterfaceStatus: interface bond0 is up
> Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.449229 started
> Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.466365 started
> Mar 11 17:49:38 argus argus[13934]: 11 Mar 14 17:49:38.485675 ArgusGetInterfaceStatus: interface bond0 is up
> 
> Aside from the “No module found in object” error, everything seems like its working Ok.  The only problem is that I don’t seem to have my argus instances configured to pull traffic from the same queue.  In other words, I have five output files from five argus instances with like traffic in all of them.  I haven’t made any changes to my argus config files, aside from telling them to write to different locations and the name of the interface.  I know I’m missing something but I’m not quite sure what it is.  If someone might be able to tell me how to configure these five instances to pull from the same PF_RING queue, I’d be mighty obliged.  Let me know if I need to submit any additional information.
> 
> Thanks,
> 
> Jeff Reynolds
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140312/acbd218a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140312/acbd218a/attachment.sig>