Argus, CPU load and threading

[Jesper Skou Jensen]

Hi Carter (and the rest of you guys),

I'll just pick up on this somewhat old thread again.

Last summer I was a bit sidetracked and didn't get much time to look 
into this issue, but I have just started looking into it again, mainly 
because we recently upgraded the Internet connection and Argus will 
potentially receive a lot more data than it did earlier.

Once again, during normal traffic all appears to be just fine, but when 
we experience massive DDoS attacks (mostly UDP flood) and when we 
load-test our setup that this issue appears. But I would like to be able 
to log all the traffic and be sure we aren't losing any packets/log.

I'm 100% sure it's not the switch, nor is it the network cable and I've 
also ruled out the NIC on the Argus box that's the issue. I can say that 
I'm 100% sure because:
1. snmp monitoring of the switch shows a much larger load
2. nload running on the Argus box shows a much larger load

Both snmp and nload shows about 1/3 to 1/2 more bytes/pkts than Argus 
seems to have captured. I used "ragraph proto bytes -M 1s -r input.ra -w 
output.png" to generate graphs of the traffic.

Which again leads me to think Argus is to "blame".

Do you guys have any ideas on now to get to the bottom of this issue?

If it matters, argus is running with this commandline "argus -i eth4 -P 
0 -S 5 -e 4 -w output.ra" and the output.ra is rotated/moved every 15 
minutes and analyzed/archived later on.


Regards
Jesper

On 26-06-2013 18:34, Carter Bullard wrote:
> Hey Jesper,
> Argus is multi-threaded, with a thread for packet processing,
> flow modeling, queue management and output.  The more independent
> packet sources, the more threads.  On this workstation, argus
> has 5 threads running.  Now, we could definitely improve the
> use of threads, but I think we're doing ok.
>
> There are lots of things that could be different between the
> two machines, that could impact performance.  Front and back bus,
> memory bandwidth, L1 and L2 cache sizes, and type of integrated
> ethernet chipsets.  All of these can affect performance.
>
> With regard to packet loss, don't forget that the switch that
> is doing the port mirroring is a primary source of packet loss.
> There is going to be a difference between the interfaces
> used on the switch.  So, in your testing, be sure and swap ports
> for the packet sources, to see if the loss follows that.
>
> As long as you're running the same version, I would attribute
> the differences to the switch first, bus bandwidth second, CPU
> speed third, ethernet chipset 4th…, hard to say which one at
> this point.
>
> Carter
>
>
> On Jun 26, 2013, at 11:00 AM, Jesper Skou Jensen <jesper.skou.jensen at uni-c.dk> wrote:
>
>> Hi guys,
>>
>> Short version:
>> In short, am I right to assume that Argus isn't particularly well suited for multi core/threads? As in, it doesn't use much more than two cores at a time, even if the CPU has many more cores that are idling?
>>
>> Long version:
>> I recently upgraded my Argus box from an older Dell (Quad Core Xeon E5410 - 2.33GHz) to a brand spanking new HP (2x Octa Core Xeon E5-2650 - 2GHz), in that process I expected Argus to perform a lot better, but that doesn't look like it's the case, and I'm wondering why.
>>
>> Is Argus very dependent on Hz? I would expect the new CPU to blow the old one out of the water.
>>
>> I still have the old and the new box running, both receiving the same mirror/monitor port traffic, and I've tried to compare the two. Both boxes are running Argus 3.0.6.1 with the same settings/options at the moment - Until I get 3.0.7.3 installed on my new box, having a few issues with it not compiling right.
>>
>> During normal network load, they show a CPU load:
>> OldBox: 23%
>> NewBox: 35%
>> and Argus captures all packets on both servers just fine.
>>
>> During a heavy network load (DDoS) I have previously noticed the CPU load to hover around 180-190% on the old box, unfortunately I haven't observed the new box during a DDoS but I'm expecting it to be around the same numbers.
>>
>> NOW for the important part...
>>
>> It appears that the new box is dropping packets, compared to the old one. :(
>>
>> The old one does drop packets during a DDoS, I know that for sure, but that the new one wouldn't be able to cope is a small mystery to me.
>>
>> I have compared a few minutes before, during and after the DDoS in question.
>>
>> :~$ racount -r argusfile.ra_oldbox
>> racount   records     total_pkts     src_pkts       dst_pkts total_bytes        src_bytes          dst_bytes
>>     sum   6221694     16046917       11623543       4423374 8917614173         2212376654         6705237519
>>
>> :~$ racount -r argusfile.ra_newbox
>> racount   records     total_pkts     src_pkts       dst_pkts total_bytes        src_bytes          dst_bytes
>>     sum   5366008     15231963       10608231       4623732 8854037128         2159179022         6694858106
>>
>> ragraph confirms it, there is a noticeable drop in bytes/sec, while pkts/sec appears almost the same.
>>
>> Do you guys have any good ideas/explanations for this behavior?
>>
>>
>> Regards
>> Jesper
>>