Radium memory leak

[Craig Merchant]

That did it!  Thanks, Carter.  You're providing some of the best support I've seen for a free, open source project.

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Tuesday, March 26, 2013 11:35 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] Radium memory leak

Hey Craig,
Here is a patch to fix the leak.  Sloppy on my part, but we made some changes
in the library, and missed the change here.  My bad.

Carter

osiris:common carter$ diff -c argus_label.c.orig argus_label.c
*** argus_label.c.orig Tue Mar 26 14:30:39 2013
--- argus_label.c           Tue Mar 26 14:31:03 2013
***************
*** 3072,3077 ****
--- 3072,3078 ----
                    if ((saddr = RaFetchIPv4AddressLabel(parser, &flow->ip_flow.ip_src)) != NULL) {
                       int slen = strlen(RaAddressLabelBuffer);
                       snprintf (&RaAddressLabelBuffer[slen], 1024 - slen, "saddr=%s", saddr);
+                      free(saddr);
                       found++;
                    }
                    if ((daddr = RaFetchIPv4AddressLabel(parser, &flow->ip_flow.ip_dst)) != NULL) {
***************
*** 3081,3086 ****
--- 3082,3088 ----
                          slen++;
                       }
                       snprintf (&RaAddressLabelBuffer[slen], 1024 - slen, "daddr=%s", daddr);
+                      free(daddr);
                       found++;
                    }
                 }


On Mar 26, 2013, at 11:57 AM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:


I restarted radium last night around 9 pm.  12 hours later, it was using about 44% of 128 GB.

Other than cleaning up some of the entries in the ralabel file, the only change I made recently was increasing the time interval of rastream from 5m to 15m to (hopefully) reduce the volume of flow records a bit when that data was fed to racluster.

We seem to be generating about 10-20 GB of binary flow records per hour.  Let me know how much data you need for testing...

Thanks.

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com>]
Sent: Tuesday, March 26, 2013 4:54 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Radium memory leak

Hmmmmm, I really thought we had fixed that.
Any sense of the rate of memory loss ?
I may need some flow data from you, if my data doesn't
generate any leaks.

I'll valgrind it today.

Carter

On Mar 25, 2013, at 11:04 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:
Hey, Carter...

I tried installing the latest 3.0.7.7 client build and I'm still seeing a memory leak in radium when label files are enabled.

These are the searches that we launch when radium starts up:

        /usr/local/bin/ra -S 10.10.10.10:561 -n -u -c "," -s "+0dnstroke" | egrep -v "(^,|^0,)" >> /ssd/argus/splunk/reverse_keystroke.csv &

        /usr/local/bin/ralabel -S 10.10.10.10:561 -f /usr/local/argus/ralabel.conf -n -u -c "," -M dsrs="+label" label="blacklisted" -s +1dur,+label:200 - not dst net 10.0.0.0/8 and not dst net 12.130.140.0/24 and not dst port 25 and not dst port 53 >> /ssd/argus/splunk/blacklisted.csv &

        /usr/local/bin/ra -S 10.10.10.10:561 -n -u -c "," -M dsrs="+duser,+suser,+label" -u -e "^SSH-" -s "+1dur,+suser,+duser,+label:200" - not dst net 10.0.0.0/8 and not dst net 12.130.140.0/24 | egrep -v "whitelisted" >> /ssd/argus/splunk/suspicious_outbound_ssh.csv &

        /usr/local/bin/rastream -S 10.10.10.10:561 -M time 15m -B 10s -w /ssd/argus/%s.argus -f /usr/local/argus/rastream.sh &

The /usr/local/argus/rastream.sh launches four searches (with different filters) that look like the following:

racluster -r $FILES -M correct -m proto saddr daddr dport -c "@" -p 3 -u -Z b -n                                     -s "+0ltime,+trans,+dur,+runtime,+mean,+stddev,+sum,+sco,+dco,+spkts,+dpkts,+sb                                    ytes,+dbytes,+load,+sload,+dload,+loss,+sloss,+dloss,+ploss,+sploss,+dploss,+rat                                    e,+srate,+drate,+appbytes,+sappbytes,+dappbytes,+label:200"

I sent you the iana label file we're using a while back along with the ralabel.conf file.  Those are still pretty much the same (though I cleaned up some errors in the label file recently).  If you want the most recent version of the label file, let me know.  Or if you want some more samples of binary flow data.

Let me know how best to troubleshoot this...

Thx.

Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130328/156181ce/attachment.html>