lost flows and memory leak in radium

[Craig Merchant]

I tried rebooting the server with the label options commented out in radium.conf.  When the server came up, radium was running at 11% CPU and there were no pauses or loss of flows when clients connected.  I added the labeling config back to radium.conf and restarted.  The CPU ran at over 190% and the flow loss and pauses returned.

I commented those lines back out again and restarted radium.  Radium ran at around 150% with flow loss and pauses.  I rebooted the server again and it radium was back to normal.

From: argus-info-bounces+cmerchant=responsys.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+cmerchant=responsys.com at lists.andrew.cmu.edu] On Behalf Of Craig Merchant
Sent: Friday, January 25, 2013 4:44 PM
To: Argus (argus-info at lists.andrew.cmu.edu)
Subject: [ARGUS] lost flows and memory leak in radium

We've got one data center currently running argus on our IDS sensor (CentOS 6.2) and it listens on a DNA/libzero interface thanks to code from Chris Wakelin.  So, we do experience the bug in PF_RING where some select() call causes argusd to run at 100% CPU all the time.

We probably average between 4-8 Gbps of traffic.  A separate host is running radium and pulls the flows off of the sensor by connecting to tcp 561.  Top shows radium running at 190% CPU most of the time.

If I connect any of the ra clients to radium (such as ra -S radium:561), flows will appear for 10-30 seconds and then pause for 30-60 seconds.  If I connect the ra clients directly to the remote argusd instance, they work fine.  We'll be deploying argus in a second data center soon, so we'd really like to take advantage of radium's ability to dedup flows.

Radium's memory usage slowly climbed whether an ra client was connected or not.

I tried commenting out the two RADIUM_CLASSIFIER settings and restarted radium.  Our label file is something like 1500 lines long, so I thought that could be causing problems.    Radium uses about 30% less CPU and memory stays at 0.8%.  The intermittent pauses still happen though.

I then tried setting RADIUM_CLASSIFIER=no instead of commenting it out and the CPU went back up by 30% and the memory usage climbed steadily with no ra clients connected.  Does that not disable labeling in radium?

I'm not sure how to diagnose it any further.  My argus.conf and radium.conf are in the spreadsheet I sent you earlier.  Let me know what I can do to help diagnose this further.

Thanks.

Craig




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130126/6d0bb7c3/attachment.html>