rasqlinsert memory consumption
Carter Bullard
carter at qosient.com
Wed Sep 29 13:31:20 EDT 2010
Hey Javier,
So going through the complete set of memory uses for rasqlinsert(), there are a number of
caches that we manage that could be impacting your memory use. Of particular interest are
the IP address printing routines, that cache the actual string that you use to construct the
SQL statement. We cache them, thinking that most addresses are recurring. I can add
some logic to limit the memory use of this module, and age the entries if needed.
Do you have an idea of how many IP addresses you see in in a day? Many sites are seeing
around 1M , but if the IP addresses are an issue, then I'd think you would be seeing
over 200M or so?
Carter
On Sep 28, 2010, at 2:27 PM, Javier Almillategui wrote:
> Hi Carter,
>
> I would like to tell that it is 4G, but I don't note any specific memory consumption problem for files under 10G (which I rarely use for db loads) I usually use files that are 50G+ in size (about a week worth of flow data).
>
> I will test argus with a smaller files and let you know if there is a difference between 4G files and 10G.
>
> best,
>
> Javier
>
> On Sep 28, 2010, at 1:37 PM, Carter Bullard wrote:
>
>> Hey Javier,
>> So I have been running rasqlinsert() on some bigger files under valgrind, a memory checking system,
>> and can't find any memory issues that stand out. Since you are just poking records into the database
>> without processing them ("-m none"), there really shouldn't be any memory use, so I'm thinking that
>> we're using, but not leaking memory, for some reason.
>>
>> The culprits could be rasqlinsert(), the argus-client libraries or the mysql client libraries.
>> Just wanted you to know that I'm still pondering your issue and hopefully will have something
>> definitive to report later this week.
>>
>> You said that you didn't see appreciable memory use until a certain threshold of file size?
>> Any since of what that size maybe? isn't 4G by any chance?
>>
>> Carter
>>
>> On Sep 28, 2010, at 11:16 AM, Javier Almillategui wrote:
>>
>>> Hi Carter,
>>>
>>> just let me know when you have a version ready and I will test it on my system
>>>
>>> best,
>>>
>>> Javier
>>>
>>> On Sep 27, 2010, at 7:05 PM, Carter Bullard wrote:
>>>
>>>> Hey Javier,
>>>> I've been pondering your problem, and I'm making some changes to the way rasqlinsert() processes files.
>>>> I've been working under the assumption that files are different from reading from a continuous stream of
>>>> argus records, but your files are so large, that I shouldn't make that assumption.
>>>>
>>>> I should have something to test later today/tomorrow.
>>>> Sorry for the huge delay.
>>>>
>>>> Carter
>>>>
>>>> On Aug 17, 2010, at 11:35 AM, Javier Almillategui wrote:
>>>>
>>>>> I'm using a single file.
>>>>>
>>>>> Javier Almillategui
>>>>> Center for Secure Information Systems
>>>>> George Mason University
>>>>> Mobile: (703)309-2060
>>>>> Email: jalmilla at gmu.edu
>>>>>
>>>>> On Aug 17, 2010, at 9:50, carter at qosient.com wrote:
>>>>>
>>>>>> Hey Javier,
>>>>>> OK, and that is for a single file or multiple files in a run?
>>>>>> Carter
>>>>>> Sent from my Verizon Wireless BlackBerry
>>>>>>
>>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>>> Date: Tue, 17 Aug 2010 08:41:32 -0500
>>>>>> To: carter at qosient.com<carter at qosient.com>
>>>>>> Cc: Argus<argus-info at lists.andrew.cmu.edu>
>>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>>>
>>>>>> Hi Carter,
>>>>>>
>>>>>> I have been using argus files of 100+ GB in size. With smaller files, the memory consumption is not noted, because it doesn't run long enough.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Javier Almillategui
>>>>>>
>>>>>> Cell:(703)309-2060
>>>>>> E-mail: jalmillategui at me.com
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Aug 17, 2010, at 6:11, carter at qosient.com wrote:
>>>>>>
>>>>>>> Hey Javier,
>>>>>>> Does this happen on any file? Particularly large files?
>>>>>>> I'll look at this tonight, want to make sure I have all the information.
>>>>>>>
>>>>>>> Carter
>>>>>>> Sent from my Verizon Wireless BlackBerry
>>>>>>>
>>>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>>>> Date: Sun, 15 Aug 2010 12:51:26 -0500
>>>>>>> To: Carter Bullard<carter at qosient.com>
>>>>>>> Cc: argus-info at lists.andrew.cmu.edu<argus-info at lists.andrew.cmu.edu>
>>>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>>>>
>>>>>>> Hi Carter,
>>>>>>>
>>>>>>> I tested the development version and it shows the same behavior as 3.0.2
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Javier Almillategui
>>>>>>>
>>>>>>> Cell:(703)309-2060
>>>>>>> E-mail: jalmillategui at me.com
>>>>>>>
>>>>>>> On Aug 13, 2010, at 7:54, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>
>>>>>>>> Hey Javier,
>>>>>>>> The odd release numbers are development versions, so there will be a few compile warnings, so no worries.
>>>>>>>> However, the buffer overflow warnings you highlighted seem to be bogus. The first one is in a system
>>>>>>>> include file, which of course could happen, but in this case isn't, and the one in rasqlinsert.c is obviously
>>>>>>>> an error. We're using snprintf and the number of bytes copied (BUFSIZ, which should be 1K) is 1/4 the size
>>>>>>>> of the array (4K). These buffers are the full pathname of the archive variable, which we're not using at this
>>>>>>>> time.
>>>>>>>>
>>>>>>>> Go ahead and run the programs to see if all is better.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 13, 2010, at 6:52 AM, Javier Almillategui wrote:
>>>>>>>>
>>>>>>>>> Hi Carter,
>>>>>>>>>
>>>>>>>>> I have compiled version 3.0.3 on my system and got some interesting warnings regarding buffer overflows:
>>>>>>>>>
>>>>>>>>> making in ./common
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_main.c
>>>>>>>>> rm -f ../lib/argus_parse.a; ar qc ../lib/argus_parse.a argus_main.o
>>>>>>>>> ranlib ../lib/argus_parse.a
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_code.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_filter.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_util.c
>>>>>>>>> ./argus_util.c: In function ‘ArgusParseResourceFile’:
>>>>>>>>> ./argus_util.c:1680: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>>> ./argus_util.c: In function ‘ArgusLog’:
>>>>>>>>> ./argus_util.c:18124: warning: format not a string literal and no format arguments
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_auth.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_parser.c
>>>>>>>>> flex -Pargus_ -t scanner.l > $$.scanner.c; mv $$.scanner.c scanner.c
>>>>>>>>> bison -y -p argus_ -d grammar.y
>>>>>>>>> mv y.tab.c grammar.c
>>>>>>>>> mv y.tab.h tokdefs.h
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./scanner.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -Dyylval=argus_lval -c grammar.c
>>>>>>>>> sed -e 's/.*/char version[] = "&";/' ./../VERSION > version.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./version.c
>>>>>>>>> rm -f ../lib/argus_common.a; ar qc ../lib/argus_common.a argus_code.o argus_filter.o argus_util.o argus_auth.o argus_parser.o scanner.o grammar.o version.o
>>>>>>>>> ranlib ../lib/argus_common.a
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_client.c
>>>>>>>>> ./argus_client.c: In function ‘ArgusParseAggregator’:
>>>>>>>>> ./argus_client.c:11758: warning: ignoring return value of ‘strtol’, declared with attribute warn_unused_result
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_label.c
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_grep.c
>>>>>>>>> rm -f ../lib/argus_client.a; ar qc ../lib/argus_client.a argus_client.o argus_label.o argus_grep.o
>>>>>>>>> ranlib ../lib/argus_client.a
>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_event.c
>>>>>>>>> rm -f ../lib/argus_event.a; ar qc ../lib/argus_event.a argus_event.o
>>>>>>>>> ranlib ../lib/argus_event.a
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>>> making in ./include
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>>> make[1]: Nothing to be done for `all'.
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>>> making in ./clients
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ra.c
>>>>>>>>> gcc -O3 -o ../bin/ra ra.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racount.c
>>>>>>>>> gcc -O3 -o ../bin/racount racount.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasort.c
>>>>>>>>> gcc -O3 -o ../bin/rasort rasort.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasplit.c
>>>>>>>>> gcc -O3 -o ../bin/rasplit rasplit.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rabins.c
>>>>>>>>> gcc -O3 -o ../bin/rabins rabins.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racluster.c
>>>>>>>>> gcc -O3 -o ../bin/racluster racluster.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raconvert.c
>>>>>>>>> gcc -O3 -o ../bin/raconvert raconvert.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastrip.c
>>>>>>>>> gcc -O3 -o ../bin/rastrip rastrip.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rahisto.c
>>>>>>>>> gcc -O3 -o ../bin/rahisto rahisto.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ralabel.c
>>>>>>>>> gcc -O3 -o ../bin/ralabel ralabel.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapolicy.c
>>>>>>>>> gcc -O3 -o ../bin/rapolicy rapolicy.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ranonymize.c
>>>>>>>>> gcc -O3 -o ../bin/ranonymize ranonymize.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapath.c
>>>>>>>>> gcc -O3 -o ../bin/rapath rapath.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastream.c
>>>>>>>>> gcc -O3 -o ../bin/rastream rastream.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratree.c
>>>>>>>>> gcc -O3 -o ../bin/ratree ratree.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rafilteraddr.c
>>>>>>>>> gcc -O3 -o ../bin/rafilteraddr rafilteraddr.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratimerange.c
>>>>>>>>> gcc -O3 -o ../bin/ratimerange ratimerange.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' raports.pl > ../bin/raports
>>>>>>>>> chmod +x ../bin/raports
>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' rahosts.pl > ../bin/rahosts
>>>>>>>>> chmod +x ../bin/rahosts
>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' radark.pl > ../bin/radark
>>>>>>>>> chmod +x ../bin/radark
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rauserdata.c
>>>>>>>>> gcc -O3 -o ../bin/rauserdata rauserdata.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raservices.c
>>>>>>>>> gcc -O3 -o ../bin/raservices raservices.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raevent.c
>>>>>>>>> gcc -O3 -o ../bin/raevent raevent.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ragrep.c
>>>>>>>>> gcc -O3 -o ../bin/ragrep ragrep.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratemplate.c
>>>>>>>>> gcc -O3 -o ../bin/ratemplate ratemplate.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>>> making in ./ragraph
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>>> sed 's+ at BINDIR@+/usr/local/bin+' ragraph.pl | \
>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' > ../bin/ragraph
>>>>>>>>> chmod +x ../bin/ragraph
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>>> making in ./ratop
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratop.c
>>>>>>>>> gcc -O3 -o ../bin/ratop ratop.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>>> making in ./radium
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radium.c
>>>>>>>>> ./radium.c: In function ‘RadiumParseResourceFile’:
>>>>>>>>> ./radium.c:818: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>>> gcc -O3 -o radium radium.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lwrap -lnsl -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>>> making in ./radump
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radump.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-aodv.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-arp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-beep.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bfd.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bgp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bootp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-domain.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-krb.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ldp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-l2tp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-lmp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-msdp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ntp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pim.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pptp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-snmp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-syslog.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rip.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-telnet.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-tftp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isoclns.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ether.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-timed.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-radius.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rx.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-smb.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-stp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isakmp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-igmp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-dvmrp.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./smbutil.c
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./gmpls.c
>>>>>>>>> gcc -O3 -o ../bin/radump radump.o print-aodv.o print-arp.o print-beep.o print-bfd.o print-bgp.o print-bootp.o print-domain.o print-krb.o print-ldp.o print-l2tp.o print-lmp.o print-msdp.o print-ntp.o print-pim.o print-pptp.o print-snmp.o print-syslog.o print-rip.o print-telnet.o print-tftp.o print-isoclns.o print-ether.o print-timed.o print-radius.o print-rx.o print-smb.o print-stp.o print-isakmp.o print-igmp.o print-dvmrp.o smbutil.o gmpls.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>>> making in ./ramysql
>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasql.c
>>>>>>>>> In file included from /usr/include/string.h:640,
>>>>>>>>> from ./rasql.c:42:
>>>>>>>>> In function ‘bzero’:
>>>>>>>>> /usr/include/bits/string3.h:100: warning: call to__builtin___memset_chk will always overflow destination buffer
>>>>>>>>> gcc -O3 -o ../bin/rasql rasql.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqlinsert.c
>>>>>>>>> In file included from /usr/include/stdio.h:910,
>>>>>>>>> from ../include/argus_util.h:195,
>>>>>>>>> from ./rasqlinsert.c:53:
>>>>>>>>> In function ‘snprintf’,
>>>>>>>>> inlined from ‘ArgusProcessCommand’ at ./rasqlinsert.c:2531:
>>>>>>>>> /usr/include/bits/stdio2.h:65: warning: call to__builtin___snprintf_chk will always overflow destination buffer
>>>>>>>>> gcc -O3 -o ../bin/rasqlinsert rasqlinsert.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqltimeindex.c
>>>>>>>>> gcc -O3 -o ../bin/rasqltimeindex rasqltimeindex.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>>>
>>>>>>>>> I will look into this later and see what alternatives there are... I will also test the compiled binaries later today.
>>>>>>>>>
>>>>>>>>> best,
>>>>>>>>>
>>>>>>>>> Javier Almillategui
>>>>>>>>>
>>>>>>>>> On Aug 12, 2010, at 5:35 PM, Carter Bullard wrote:
>>>>>>>>>
>>>>>>>>>> Hey Javier,
>>>>>>>>>> The current client version to use that may fix this bug is argus-clients-3.0.3.17, which you can get here:
>>>>>>>>>> http://qosient.com/argus/dev/argus-clients-3.0.3.17.tar.gz
>>>>>>>>>>
>>>>>>>>>> I believe that there was a memory leak that we fixed. If you try the new code, could you please send
>>>>>>>>>> some email if the problem was fixed? That would be most helpful. If its not fixed, send more email.
>>>>>>>>>>
>>>>>>>>>> Carter
>>>>>>>>>>
>>>>>>>>>> On Aug 12, 2010, at 4:56 PM, Javier Almillategui wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I have noticed a strange behavior in rasqlinsert (using argus client 3.0.2 compiled from source) in ubuntu 10.04 x86_64
>>>>>>>>>>>
>>>>>>>>>>> after running the rasqlinsert to load a database using the following options:
>>>>>>>>>>>
>>>>>>>>>>> rasqlinsert -w mysql://xxx:xxx@localhost/anondb/xxxtap_%Y_%m_%d -r xxx_tap_20100521_prepared.argus -s +2seq -s +3ltime -s +4dur -s +loss -s +1srcid -s -record -m none -M time 1d
>>>>>>>>>>>
>>>>>>>>>>> the memory consumption of rasqlinsert continues to increase to the point of using a significant part of my systems memory.
>>>>>>>>>>>
>>>>>>>>>>> it went from consuming ~16 MB to consuming 712 MB (at this point).
>>>>>>>>>>>
>>>>>>>>>>> is there any known memory leak in rasqlinsert??
>>>>>>>>>>>
>>>>>>>>>>> best,
>>>>>>>>>>>
>>>>>>>>>>> Javier Almillategui
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>>> Carter Bullard
>>>> CEO/President
>>>> QoSient, LLC
>>>> 150 E 57th Street Suite 12D
>>>> New York, New York 10022
>>>>
>>>> +1 212 588-9133 Phone
>>>> +1 212 588-9134 Fax
>>>>
>>>>
>>>>
>>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100929/d433e7d0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100929/d433e7d0/attachment.bin>
More information about the argus
mailing list