Argus, CPU load and threading

[Jesse Bowling]

Hi Jesper,

I didn't see any details on the configuration of the box argus is running on; OS, NIC, any special drivers (I.e PF_RING), performance tuning...can you add some detail on that end?

Cheers,

Jesse

> On Feb 13, 2014, at 3:59 AM, Jesper Skou Jensen <jesper.skou.jensen at uni-c.dk> wrote:
> 
> Hi Carter (and the rest of you guys),
> 
> I'll just pick up on this somewhat old thread again.
> 
> Last summer I was a bit sidetracked and didn't get much time to look into this issue, but I have just started looking into it again, mainly because we recently upgraded the Internet connection and Argus will potentially receive a lot more data than it did earlier.
> 
> Once again, during normal traffic all appears to be just fine, but when we experience massive DDoS attacks (mostly UDP flood) and when we load-test our setup that this issue appears. But I would like to be able to log all the traffic and be sure we aren't losing any packets/log.
> 
> I'm 100% sure it's not the switch, nor is it the network cable and I've also ruled out the NIC on the Argus box that's the issue. I can say that I'm 100% sure because:
> 1. snmp monitoring of the switch shows a much larger load
> 2. nload running on the Argus box shows a much larger load
> 
> Both snmp and nload shows about 1/3 to 1/2 more bytes/pkts than Argus seems to have captured. I used "ragraph proto bytes -M 1s -r input.ra -w output.png" to generate graphs of the traffic.
> 
> Which again leads me to think Argus is to "blame".
> 
> Do you guys have any ideas on now to get to the bottom of this issue?
> 
> If it matters, argus is running with this commandline "argus -i eth4 -P 0 -S 5 -e 4 -w output.ra" and the output.ra is rotated/moved every 15 minutes and analyzed/archived later on.
> 
> 
> Regards
> Jesper
> 
>> On 26-06-2013 18:34, Carter Bullard wrote:
>> Hey Jesper,
>> Argus is multi-threaded, with a thread for packet processing,
>> flow modeling, queue management and output.  The more independent
>> packet sources, the more threads.  On this workstation, argus
>> has 5 threads running.  Now, we could definitely improve the
>> use of threads, but I think we're doing ok.
>> 
>> There are lots of things that could be different between the
>> two machines, that could impact performance.  Front and back bus,
>> memory bandwidth, L1 and L2 cache sizes, and type of integrated
>> ethernet chipsets.  All of these can affect performance.
>> 
>> With regard to packet loss, don't forget that the switch that
>> is doing the port mirroring is a primary source of packet loss.
>> There is going to be a difference between the interfaces
>> used on the switch.  So, in your testing, be sure and swap ports
>> for the packet sources, to see if the loss follows that.
>> 
>> As long as you're running the same version, I would attribute
>> the differences to the switch first, bus bandwidth second, CPU
>> speed third, ethernet chipset 4th…, hard to say which one at
>> this point.
>> 
>> Carter
>> 
>> 
>>> On Jun 26, 2013, at 11:00 AM, Jesper Skou Jensen <jesper.skou.jensen at uni-c.dk> wrote:
>>> 
>>> Hi guys,
>>> 
>>> Short version:
>>> In short, am I right to assume that Argus isn't particularly well suited for multi core/threads? As in, it doesn't use much more than two cores at a time, even if the CPU has many more cores that are idling?
>>> 
>>> Long version:
>>> I recently upgraded my Argus box from an older Dell (Quad Core Xeon E5410 - 2.33GHz) to a brand spanking new HP (2x Octa Core Xeon E5-2650 - 2GHz), in that process I expected Argus to perform a lot better, but that doesn't look like it's the case, and I'm wondering why.
>>> 
>>> Is Argus very dependent on Hz? I would expect the new CPU to blow the old one out of the water.
>>> 
>>> I still have the old and the new box running, both receiving the same mirror/monitor port traffic, and I've tried to compare the two. Both boxes are running Argus 3.0.6.1 with the same settings/options at the moment - Until I get 3.0.7.3 installed on my new box, having a few issues with it not compiling right.
>>> 
>>> During normal network load, they show a CPU load:
>>> OldBox: 23%
>>> NewBox: 35%
>>> and Argus captures all packets on both servers just fine.
>>> 
>>> During a heavy network load (DDoS) I have previously noticed the CPU load to hover around 180-190% on the old box, unfortunately I haven't observed the new box during a DDoS but I'm expecting it to be around the same numbers.
>>> 
>>> NOW for the important part...
>>> 
>>> It appears that the new box is dropping packets, compared to the old one. :(
>>> 
>>> The old one does drop packets during a DDoS, I know that for sure, but that the new one wouldn't be able to cope is a small mystery to me.
>>> 
>>> I have compared a few minutes before, during and after the DDoS in question.
>>> 
>>> :~$ racount -r argusfile.ra_oldbox
>>> racount   records     total_pkts     src_pkts       dst_pkts total_bytes        src_bytes          dst_bytes
>>>    sum   6221694     16046917       11623543       4423374 8917614173         2212376654         6705237519
>>> 
>>> :~$ racount -r argusfile.ra_newbox
>>> racount   records     total_pkts     src_pkts       dst_pkts total_bytes        src_bytes          dst_bytes
>>>    sum   5366008     15231963       10608231       4623732 8854037128         2159179022         6694858106
>>> 
>>> ragraph confirms it, there is a noticeable drop in bytes/sec, while pkts/sec appears almost the same.
>>> 
>>> Do you guys have any good ideas/explanations for this behavior?
>>> 
>>> 
>>> Regards
>>> Jesper
>>> 
>