Argus with PF_RING DNA clusters

[Carter Bullard]

Hey Chris,
We have a few P1 / P2 flows where this pre-classification will not work.  They are primarily for ICMP mapping and multicast request, unicast respose tracking.  Any chance we can add them to the PF_RING classifier ?

So you aill have 1 radium collect flow records from all the argi, to generate a single output argus data stream.  No problems.

Carter


On Oct 5, 2012, at 12:59 PM, Chris Wakelin <c.d.wakelin at reading.ac.uk> wrote:

> On 05/10/12 15:55, Carter Bullard wrote:
>> Hey Chris,
>> Sorry for the delay. 
>> 
>> Argus is already structured for multiple threads to eat packets, but there
>> is work that needs to be done to enable many cores to get into 
>> the packet processing business for a single interface.  We can do that
>> kind of effort on the list if that is of interest.   Not hard, but will need some
>> re-architecting for argus to support multiple cores at this level.
>> 
>> There are two strategies possible here, that I see.  
>> 
>> First involves routing packets to the appropriate cores for processing.  Argus
>> has been ported to multi-core chips, like Tilera, and they use this technique,
>> where you run a separate image of argus on each core.  One core reads
>> packets, and processes the headers to the point where it can decide which 
>> argus context should process the packets, and then queues the packets for
>> the independent argi to process.  This is pretty good, as it avoids any form
>> of locking, but does have the problem of asymmetric work loads.
> 
> That was more or less what I had in mind. PF_RING DNA will split the
> traffic by flow into up to 32 queues bound to virtual ethernet
> interfaces (though in my case I might be limited to 10). Unlike RSS in
> the native ixgbe driver, the flows should be symmetrically balanced so
> that packets from X->Y go to the same queue as those from Y->X. The flow
> binding is via a user-customisable function, so it may be possible to
> arrange things to get a similar number of packets in each core.
> 
> I'm hoping to have my three applications each get their own view of each
> of the queues, but each bound to the same CPU for a particular queue.
> This should avoid cache-misses on each core, which is one of the
> problems, I believe, in multithreaded processing.
> 
> At the same time, it would be nice for all the flow records to end up in
> one Argus database, so running 10 independent copies of Argus could be a
> problem. I'm hoping that starting Argus with multiple virtual
> interfaces, and then setting the CPU affinities for the individual
> threads manually should avoid Argus having to do the flow balancing
> itself, but have one database.
> 
>> 
>> The other is where you have multiple cores doing the packet header processing,
>> getting to the point where you have established the flow cache keys, and
>> calculated the hash values, then a single core does the look ups, and updates
>> the metrics.  This type of processing has some advantages for hardware
>> implementations, but does have some issues in software scheduling, as the
>> amount of header processing can be different for each packet.
> 
> That would be fine if it was just ARGUS. I'm hoping to have PF_RING do
> the flow balancing, and then have each application process the packets
> in the appropriate queues and save their results (flow database, IDS
> alerts etc.) in one place each.
> 
>> If you're interested, and can help me with the PF_RING part, I can do
>> a lot of the multicore parts.
>> 
>> Carter
> 
> I'll do my best to help with PF_RING, but I'm not really a programmer!
> 
> Best Wishes,
> Chris
> 
> -- 
> --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
> Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
> IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 2908
> Whiteknights, Reading, RG6 6AF, UK              Fax: +44 (0)118 975 3094
>