rasqlinsert memory consumption

Tue Sep 28 14:27:53 EDT 2010

Hi Carter,

I would like to tell that it is 4G, but I don't note any specific memory consumption problem for files under 10G (which I rarely use for db loads) I usually use files that are 50G+ in size (about a week worth of flow data).

I will test argus with a smaller files and let you know if there is a difference between 4G files and 10G.

best,

Javier

On Sep 28, 2010, at 1:37 PM, Carter Bullard wrote:

> Hey Javier,
> So I have been running rasqlinsert() on some bigger files under valgrind, a memory checking system,
> and can't find any memory issues that stand out.  Since you are just poking records into the database
> without processing them ("-m none"), there really shouldn't be any memory use, so I'm thinking that
> we're using, but not leaking memory, for some reason.
> 
> The culprits could be rasqlinsert(), the argus-client libraries or the mysql client libraries.
> Just wanted you to know that I'm still pondering your issue and hopefully will have something
> definitive to report later this week.
> 
> You said that you didn't see appreciable memory use until a certain threshold of file size?
> Any since of what that size maybe?  isn't 4G by any chance?
> 
> Carter
> 
> On Sep 28, 2010, at 11:16 AM, Javier Almillategui wrote:
> 
>> Hi Carter,
>> 
>> just let me know when you have a version ready and I will test it on my system
>> 
>> best,
>> 
>> Javier
>> 
>> On Sep 27, 2010, at 7:05 PM, Carter Bullard wrote:
>> 
>>> Hey Javier,
>>> I've been pondering your problem, and I'm making some changes to the way rasqlinsert() processes files.
>>> I've been working under the assumption that files are different from reading from a continuous stream of
>>> argus records, but your files are so large, that I shouldn't make that assumption.  
>>> 
>>> I should have something to test later today/tomorrow.
>>> Sorry for the huge delay.
>>> 
>>> Carter
>>> 
>>> On Aug 17, 2010, at 11:35 AM, Javier Almillategui wrote:
>>> 
>>>> I'm using a single file.
>>>> 
>>>> Javier Almillategui
>>>> Center for Secure Information Systems
>>>> George Mason University
>>>> Mobile: (703)309-2060
>>>> Email: jalmilla at gmu.edu
>>>> 
>>>> On Aug 17, 2010, at 9:50, carter at qosient.com wrote:
>>>> 
>>>>> Hey Javier,
>>>>> OK, and that is for a single file or multiple files in a run?
>>>>> Carter
>>>>> Sent from my Verizon Wireless BlackBerry
>>>>> 
>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>> Date: Tue, 17 Aug 2010 08:41:32 -0500
>>>>> To: carter at qosient.com<carter at qosient.com>
>>>>> Cc: Argus<argus-info at lists.andrew.cmu.edu>
>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>> 
>>>>> Hi Carter,
>>>>> 
>>>>> I have been using argus files of 100+ GB in size.  With smaller files, the memory consumption is not noted, because it doesn't run long enough.
>>>>> 
>>>>> Best, 
>>>>> 
>>>>> Javier Almillategui
>>>>> 
>>>>> Cell:(703)309-2060
>>>>> E-mail: jalmillategui at me.com
>>>>> Sent from my iPhone
>>>>> 
>>>>> On Aug 17, 2010, at 6:11, carter at qosient.com wrote:
>>>>> 
>>>>>> Hey Javier,
>>>>>> Does this happen on any file? Particularly large files?
>>>>>> I'll look at this tonight, want to make sure I have all the information.
>>>>>> 
>>>>>> Carter 
>>>>>> Sent from my Verizon Wireless BlackBerry
>>>>>> 
>>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>>> Date: Sun, 15 Aug 2010 12:51:26 -0500
>>>>>> To: Carter Bullard<carter at qosient.com>
>>>>>> Cc: argus-info at lists.andrew.cmu.edu<argus-info at lists.andrew.cmu.edu>
>>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>>> 
>>>>>> Hi Carter,
>>>>>> 
>>>>>> I tested the development version and it shows the same behavior as 3.0.2
>>>>>> 
>>>>>> Best,
>>>>>> 
>>>>>> Javier Almillategui
>>>>>> 
>>>>>> Cell:(703)309-2060
>>>>>> E-mail: jalmillategui at me.com
>>>>>> 
>>>>>> On Aug 13, 2010, at 7:54, Carter Bullard <carter at qosient.com> wrote:
>>>>>> 
>>>>>>> Hey Javier,
>>>>>>> The odd release numbers are development versions, so there will be a few compile warnings, so no worries.
>>>>>>> However, the buffer overflow warnings you highlighted seem to be bogus.  The first one is in a system
>>>>>>> include file, which of course could happen, but in this case isn't, and the one in rasqlinsert.c is obviously
>>>>>>> an error.  We're using snprintf and the number of bytes copied (BUFSIZ, which should be 1K) is 1/4 the size
>>>>>>> of the array (4K).  These buffers are the full pathname of the archive variable, which we're not using at this
>>>>>>> time.
>>>>>>> 
>>>>>>> Go ahead and run the programs to see if all is better.
>>>>>>> 
>>>>>>> Carter
>>>>>>> 
>>>>>>> 
>>>>>>> On Aug 13, 2010, at 6:52 AM, Javier Almillategui wrote:
>>>>>>> 
>>>>>>>> Hi Carter,
>>>>>>>> 
>>>>>>>> I have compiled version 3.0.3 on my system and got some interesting warnings regarding buffer overflows:
>>>>>>>> 
>>>>>>>> making in ./common
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_main.c
>>>>>>>> rm -f ../lib/argus_parse.a; ar qc ../lib/argus_parse.a argus_main.o
>>>>>>>> ranlib ../lib/argus_parse.a
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_code.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_filter.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_util.c
>>>>>>>> ./argus_util.c: In function ‘ArgusParseResourceFile’:
>>>>>>>> ./argus_util.c:1680: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>> ./argus_util.c: In function ‘ArgusLog’:
>>>>>>>> ./argus_util.c:18124: warning: format not a string literal and no format arguments
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_auth.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_parser.c
>>>>>>>> flex -Pargus_ -t scanner.l > $$.scanner.c; mv $$.scanner.c scanner.c
>>>>>>>> bison -y -p argus_ -d grammar.y
>>>>>>>> mv y.tab.c grammar.c
>>>>>>>> mv y.tab.h tokdefs.h
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./scanner.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -Dyylval=argus_lval -c grammar.c
>>>>>>>> sed -e 's/.*/char version[] = "&";/' ./../VERSION > version.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./version.c
>>>>>>>> rm -f ../lib/argus_common.a; ar qc ../lib/argus_common.a argus_code.o argus_filter.o argus_util.o argus_auth.o argus_parser.o scanner.o grammar.o version.o
>>>>>>>> ranlib ../lib/argus_common.a
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_client.c
>>>>>>>> ./argus_client.c: In function ‘ArgusParseAggregator’:
>>>>>>>> ./argus_client.c:11758: warning: ignoring return value of ‘strtol’, declared with attribute warn_unused_result
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_label.c
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_grep.c
>>>>>>>> rm -f ../lib/argus_client.a; ar qc ../lib/argus_client.a argus_client.o argus_label.o argus_grep.o
>>>>>>>> ranlib ../lib/argus_client.a
>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_event.c
>>>>>>>> rm -f ../lib/argus_event.a; ar qc ../lib/argus_event.a argus_event.o
>>>>>>>> ranlib ../lib/argus_event.a
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>> making in ./include
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>> make[1]: Nothing to be done for `all'.
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>> making in ./clients
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ra.c
>>>>>>>> gcc -O3 -o ../bin/ra ra.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racount.c
>>>>>>>> gcc -O3 -o ../bin/racount racount.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasort.c
>>>>>>>> gcc -O3 -o ../bin/rasort rasort.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasplit.c
>>>>>>>> gcc -O3 -o ../bin/rasplit rasplit.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rabins.c
>>>>>>>> gcc -O3 -o ../bin/rabins rabins.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racluster.c
>>>>>>>> gcc -O3 -o ../bin/racluster racluster.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raconvert.c
>>>>>>>> gcc -O3 -o ../bin/raconvert raconvert.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastrip.c
>>>>>>>> gcc -O3 -o ../bin/rastrip rastrip.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rahisto.c
>>>>>>>> gcc -O3 -o ../bin/rahisto rahisto.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ralabel.c
>>>>>>>> gcc -O3 -o ../bin/ralabel ralabel.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapolicy.c
>>>>>>>> gcc -O3 -o ../bin/rapolicy rapolicy.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ranonymize.c
>>>>>>>> gcc -O3 -o ../bin/ranonymize ranonymize.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapath.c
>>>>>>>> gcc -O3 -o ../bin/rapath rapath.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastream.c
>>>>>>>> gcc -O3 -o ../bin/rastream rastream.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratree.c
>>>>>>>> gcc -O3 -o ../bin/ratree ratree.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rafilteraddr.c
>>>>>>>> gcc -O3 -o ../bin/rafilteraddr rafilteraddr.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratimerange.c
>>>>>>>> gcc -O3 -o ../bin/ratimerange ratimerange.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' raports.pl > ../bin/raports
>>>>>>>> chmod +x ../bin/raports
>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' rahosts.pl > ../bin/rahosts
>>>>>>>> chmod +x ../bin/rahosts
>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' radark.pl > ../bin/radark
>>>>>>>> chmod +x ../bin/radark
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rauserdata.c
>>>>>>>> gcc -O3 -o ../bin/rauserdata rauserdata.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raservices.c
>>>>>>>> gcc -O3 -o ../bin/raservices raservices.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raevent.c
>>>>>>>> gcc -O3 -o ../bin/raevent raevent.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ragrep.c
>>>>>>>> gcc -O3 -o ../bin/ragrep ragrep.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratemplate.c
>>>>>>>> gcc -O3 -o ../bin/ratemplate ratemplate.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>> making in ./ragraph
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>> sed 's+ at BINDIR@+/usr/local/bin+' ragraph.pl | \
>>>>>>>> 	sed 's+ at PERLBIN@+/usr/bin/perl+' > ../bin/ragraph
>>>>>>>> chmod +x ../bin/ragraph
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>> making in ./ratop
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratop.c
>>>>>>>> gcc -O3 -o ../bin/ratop ratop.o  ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP -lncurses 
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>> making in ./radium
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radium.c
>>>>>>>> ./radium.c: In function ‘RadiumParseResourceFile’:
>>>>>>>> ./radium.c:818: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>> gcc -O3 -o radium radium.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lwrap -lnsl -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>> making in ./radump
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radump.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-aodv.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-arp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-beep.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bfd.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bgp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bootp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-domain.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-krb.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ldp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-l2tp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-lmp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-msdp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ntp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pim.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pptp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-snmp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-syslog.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rip.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-telnet.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-tftp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isoclns.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ether.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-timed.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-radius.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rx.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-smb.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-stp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isakmp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-igmp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-dvmrp.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./smbutil.c
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./gmpls.c
>>>>>>>> gcc -O3 -o ../bin/radump radump.o print-aodv.o print-arp.o print-beep.o print-bfd.o print-bgp.o print-bootp.o print-domain.o print-krb.o print-ldp.o print-l2tp.o print-lmp.o print-msdp.o print-ntp.o print-pim.o print-pptp.o print-snmp.o print-syslog.o print-rip.o print-telnet.o print-tftp.o print-isoclns.o print-ether.o print-timed.o print-radius.o print-rx.o print-smb.o print-stp.o print-isakmp.o print-igmp.o print-dvmrp.o smbutil.o gmpls.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>> making in ./ramysql
>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasql.c
>>>>>>>> In file included from /usr/include/string.h:640,
>>>>>>>>                  from ./rasql.c:42:
>>>>>>>> In function ‘bzero’:
>>>>>>>> /usr/include/bits/string3.h:100: warning: call to__builtin___memset_chk will always overflow destination buffer
>>>>>>>> gcc -O3 -o ../bin/rasql rasql.o -L/usr/lib/mysql -lmysqlclient_r  ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqlinsert.c
>>>>>>>> In file included from /usr/include/stdio.h:910,
>>>>>>>>                  from ../include/argus_util.h:195,
>>>>>>>>                  from ./rasqlinsert.c:53:
>>>>>>>> In function ‘snprintf’,
>>>>>>>>     inlined from ‘ArgusProcessCommand’ at ./rasqlinsert.c:2531:
>>>>>>>> /usr/include/bits/stdio2.h:65: warning: call to__builtin___snprintf_chk will always overflow destination buffer
>>>>>>>> gcc -O3 -o ../bin/rasqlinsert rasqlinsert.o -L/usr/lib/mysql -lmysqlclient_r  ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP -lncurses 
>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqltimeindex.c
>>>>>>>> gcc -O3 -o ../bin/rasqltimeindex rasqltimeindex.o -L/usr/lib/mysql -lmysqlclient_r  ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz   -lpthread  -L/usr/lib -lGeoIP
>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>> 
>>>>>>>> I will look into this later and see what alternatives there are...  I will also test the compiled binaries later today.
>>>>>>>> 
>>>>>>>> best,
>>>>>>>> 
>>>>>>>> Javier Almillategui
>>>>>>>> 
>>>>>>>> On Aug 12, 2010, at 5:35 PM, Carter Bullard wrote:
>>>>>>>> 
>>>>>>>>> Hey Javier,
>>>>>>>>> The current client version to use that may fix this bug is argus-clients-3.0.3.17, which you can get here:
>>>>>>>>>   http://qosient.com/argus/dev/argus-clients-3.0.3.17.tar.gz
>>>>>>>>> 
>>>>>>>>> I believe that there was a memory leak that we fixed.  If you try the new code, could you please send
>>>>>>>>> some email if the problem was fixed?  That would be most helpful.  If its not fixed, send more email.
>>>>>>>>> 
>>>>>>>>> Carter
>>>>>>>>> 
>>>>>>>>> On Aug 12, 2010, at 4:56 PM, Javier Almillategui wrote:
>>>>>>>>> 
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> I have noticed a strange behavior in rasqlinsert (using argus client 3.0.2 compiled from source) in ubuntu 10.04 x86_64
>>>>>>>>>> 
>>>>>>>>>> after running the rasqlinsert to load a database using the following options:
>>>>>>>>>> 
>>>>>>>>>> rasqlinsert -w mysql://xxx:xxx@localhost/anondb/xxxtap_%Y_%m_%d -r xxx_tap_20100521_prepared.argus -s +2seq -s +3ltime -s +4dur -s +loss -s +1srcid -s -record -m none -M time 1d
>>>>>>>>>> 
>>>>>>>>>> the memory consumption of rasqlinsert continues to increase to the point of using a significant part of my systems memory.
>>>>>>>>>> 
>>>>>>>>>> it went from consuming ~16 MB to consuming 712 MB (at this point).
>>>>>>>>>> 
>>>>>>>>>> is there any known memory leak in rasqlinsert??
>>>>>>>>>> 
>>>>>>>>>> best,
>>>>>>>>>> 
>>>>>>>>>> Javier Almillategui
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>> 
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York  10022
>>> 
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>> 
>>> 
>>> 
>> 
> 
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York  10022
> 
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100928/4b741378/attachment.html>