rasqlinsert memory consumption
Javier Almillategui
jalmilla at gmu.edu
Wed Sep 29 14:14:45 EDT 2010
we are seeing traffic from a B class address space (eg: 172.16.0.0/16).
best,
Javier
On Sep 29, 2010, at 1:31 PM, Carter Bullard wrote:
> Hey Javier,
> So going through the complete set of memory uses for rasqlinsert(), there are a number of
> caches that we manage that could be impacting your memory use. Of particular interest are
> the IP address printing routines, that cache the actual string that you use to construct the
> SQL statement. We cache them, thinking that most addresses are recurring. I can add
> some logic to limit the memory use of this module, and age the entries if needed.
>
> Do you have an idea of how many IP addresses you see in in a day? Many sites are seeing
> around 1M , but if the IP addresses are an issue, then I'd think you would be seeing
> over 200M or so?
>
> Carter
>
> On Sep 28, 2010, at 2:27 PM, Javier Almillategui wrote:
>
>> Hi Carter,
>>
>> I would like to tell that it is 4G, but I don't note any specific memory consumption problem for files under 10G (which I rarely use for db loads) I usually use files that are 50G+ in size (about a week worth of flow data).
>>
>> I will test argus with a smaller files and let you know if there is a difference between 4G files and 10G.
>>
>> best,
>>
>> Javier
>>
>> On Sep 28, 2010, at 1:37 PM, Carter Bullard wrote:
>>
>>> Hey Javier,
>>> So I have been running rasqlinsert() on some bigger files under valgrind, a memory checking system,
>>> and can't find any memory issues that stand out. Since you are just poking records into the database
>>> without processing them ("-m none"), there really shouldn't be any memory use, so I'm thinking that
>>> we're using, but not leaking memory, for some reason.
>>>
>>> The culprits could be rasqlinsert(), the argus-client libraries or the mysql client libraries.
>>> Just wanted you to know that I'm still pondering your issue and hopefully will have something
>>> definitive to report later this week.
>>>
>>> You said that you didn't see appreciable memory use until a certain threshold of file size?
>>> Any since of what that size maybe? isn't 4G by any chance?
>>>
>>> Carter
>>>
>>> On Sep 28, 2010, at 11:16 AM, Javier Almillategui wrote:
>>>
>>>> Hi Carter,
>>>>
>>>> just let me know when you have a version ready and I will test it on my system
>>>>
>>>> best,
>>>>
>>>> Javier
>>>>
>>>> On Sep 27, 2010, at 7:05 PM, Carter Bullard wrote:
>>>>
>>>>> Hey Javier,
>>>>> I've been pondering your problem, and I'm making some changes to the way rasqlinsert() processes files.
>>>>> I've been working under the assumption that files are different from reading from a continuous stream of
>>>>> argus records, but your files are so large, that I shouldn't make that assumption.
>>>>>
>>>>> I should have something to test later today/tomorrow.
>>>>> Sorry for the huge delay.
>>>>>
>>>>> Carter
>>>>>
>>>>> On Aug 17, 2010, at 11:35 AM, Javier Almillategui wrote:
>>>>>
>>>>>> I'm using a single file.
>>>>>>
>>>>>> Javier Almillategui
>>>>>> Center for Secure Information Systems
>>>>>> George Mason University
>>>>>> Mobile: (703)309-2060
>>>>>> Email: jalmilla at gmu.edu
>>>>>>
>>>>>> On Aug 17, 2010, at 9:50, carter at qosient.com wrote:
>>>>>>
>>>>>>> Hey Javier,
>>>>>>> OK, and that is for a single file or multiple files in a run?
>>>>>>> Carter
>>>>>>> Sent from my Verizon Wireless BlackBerry
>>>>>>>
>>>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>>>> Date: Tue, 17 Aug 2010 08:41:32 -0500
>>>>>>> To: carter at qosient.com<carter at qosient.com>
>>>>>>> Cc: Argus<argus-info at lists.andrew.cmu.edu>
>>>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>>>>
>>>>>>> Hi Carter,
>>>>>>>
>>>>>>> I have been using argus files of 100+ GB in size. With smaller files, the memory consumption is not noted, because it doesn't run long enough.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Javier Almillategui
>>>>>>>
>>>>>>> Cell:(703)309-2060
>>>>>>> E-mail: jalmillategui at me.com
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>> On Aug 17, 2010, at 6:11, carter at qosient.com wrote:
>>>>>>>
>>>>>>>> Hey Javier,
>>>>>>>> Does this happen on any file? Particularly large files?
>>>>>>>> I'll look at this tonight, want to make sure I have all the information.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>> Sent from my Verizon Wireless BlackBerry
>>>>>>>>
>>>>>>>> From: Javier Almillategui <jalmilla at gmu.edu>
>>>>>>>> Date: Sun, 15 Aug 2010 12:51:26 -0500
>>>>>>>> To: Carter Bullard<carter at qosient.com>
>>>>>>>> Cc: argus-info at lists.andrew.cmu.edu<argus-info at lists.andrew.cmu.edu>
>>>>>>>> Subject: Re: [ARGUS] rasqlinsert memory consumption
>>>>>>>>
>>>>>>>> Hi Carter,
>>>>>>>>
>>>>>>>> I tested the development version and it shows the same behavior as 3.0.2
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Javier Almillategui
>>>>>>>>
>>>>>>>> Cell:(703)309-2060
>>>>>>>> E-mail: jalmillategui at me.com
>>>>>>>>
>>>>>>>> On Aug 13, 2010, at 7:54, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Javier,
>>>>>>>>> The odd release numbers are development versions, so there will be a few compile warnings, so no worries.
>>>>>>>>> However, the buffer overflow warnings you highlighted seem to be bogus. The first one is in a system
>>>>>>>>> include file, which of course could happen, but in this case isn't, and the one in rasqlinsert.c is obviously
>>>>>>>>> an error. We're using snprintf and the number of bytes copied (BUFSIZ, which should be 1K) is 1/4 the size
>>>>>>>>> of the array (4K). These buffers are the full pathname of the archive variable, which we're not using at this
>>>>>>>>> time.
>>>>>>>>>
>>>>>>>>> Go ahead and run the programs to see if all is better.
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Aug 13, 2010, at 6:52 AM, Javier Almillategui wrote:
>>>>>>>>>
>>>>>>>>>> Hi Carter,
>>>>>>>>>>
>>>>>>>>>> I have compiled version 3.0.3 on my system and got some interesting warnings regarding buffer overflows:
>>>>>>>>>>
>>>>>>>>>> making in ./common
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_main.c
>>>>>>>>>> rm -f ../lib/argus_parse.a; ar qc ../lib/argus_parse.a argus_main.o
>>>>>>>>>> ranlib ../lib/argus_parse.a
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_code.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_filter.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_util.c
>>>>>>>>>> ./argus_util.c: In function ‘ArgusParseResourceFile’:
>>>>>>>>>> ./argus_util.c:1680: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>>>> ./argus_util.c: In function ‘ArgusLog’:
>>>>>>>>>> ./argus_util.c:18124: warning: format not a string literal and no format arguments
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_auth.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_parser.c
>>>>>>>>>> flex -Pargus_ -t scanner.l > $$.scanner.c; mv $$.scanner.c scanner.c
>>>>>>>>>> bison -y -p argus_ -d grammar.y
>>>>>>>>>> mv y.tab.c grammar.c
>>>>>>>>>> mv y.tab.h tokdefs.h
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./scanner.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -Dyylval=argus_lval -c grammar.c
>>>>>>>>>> sed -e 's/.*/char version[] = "&";/' ./../VERSION > version.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./version.c
>>>>>>>>>> rm -f ../lib/argus_common.a; ar qc ../lib/argus_common.a argus_code.o argus_filter.o argus_util.o argus_auth.o argus_parser.o scanner.o grammar.o version.o
>>>>>>>>>> ranlib ../lib/argus_common.a
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_client.c
>>>>>>>>>> ./argus_client.c: In function ‘ArgusParseAggregator’:
>>>>>>>>>> ./argus_client.c:11758: warning: ignoring return value of ‘strtol’, declared with attribute warn_unused_result
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_label.c
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_grep.c
>>>>>>>>>> rm -f ../lib/argus_client.a; ar qc ../lib/argus_client.a argus_client.o argus_label.o argus_grep.o
>>>>>>>>>> ranlib ../lib/argus_client.a
>>>>>>>>>> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_event.c
>>>>>>>>>> rm -f ../lib/argus_event.a; ar qc ../lib/argus_event.a argus_event.o
>>>>>>>>>> ranlib ../lib/argus_event.a
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
>>>>>>>>>> making in ./include
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>>>> make[1]: Nothing to be done for `all'.
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
>>>>>>>>>> making in ./clients
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ra.c
>>>>>>>>>> gcc -O3 -o ../bin/ra ra.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racount.c
>>>>>>>>>> gcc -O3 -o ../bin/racount racount.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasort.c
>>>>>>>>>> gcc -O3 -o ../bin/rasort rasort.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasplit.c
>>>>>>>>>> gcc -O3 -o ../bin/rasplit rasplit.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rabins.c
>>>>>>>>>> gcc -O3 -o ../bin/rabins rabins.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racluster.c
>>>>>>>>>> gcc -O3 -o ../bin/racluster racluster.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raconvert.c
>>>>>>>>>> gcc -O3 -o ../bin/raconvert raconvert.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastrip.c
>>>>>>>>>> gcc -O3 -o ../bin/rastrip rastrip.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rahisto.c
>>>>>>>>>> gcc -O3 -o ../bin/rahisto rahisto.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ralabel.c
>>>>>>>>>> gcc -O3 -o ../bin/ralabel ralabel.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapolicy.c
>>>>>>>>>> gcc -O3 -o ../bin/rapolicy rapolicy.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ranonymize.c
>>>>>>>>>> gcc -O3 -o ../bin/ranonymize ranonymize.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapath.c
>>>>>>>>>> gcc -O3 -o ../bin/rapath rapath.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastream.c
>>>>>>>>>> gcc -O3 -o ../bin/rastream rastream.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratree.c
>>>>>>>>>> gcc -O3 -o ../bin/ratree ratree.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rafilteraddr.c
>>>>>>>>>> gcc -O3 -o ../bin/rafilteraddr rafilteraddr.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratimerange.c
>>>>>>>>>> gcc -O3 -o ../bin/ratimerange ratimerange.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' raports.pl > ../bin/raports
>>>>>>>>>> chmod +x ../bin/raports
>>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' rahosts.pl > ../bin/rahosts
>>>>>>>>>> chmod +x ../bin/rahosts
>>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' radark.pl > ../bin/radark
>>>>>>>>>> chmod +x ../bin/radark
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rauserdata.c
>>>>>>>>>> gcc -O3 -o ../bin/rauserdata rauserdata.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raservices.c
>>>>>>>>>> gcc -O3 -o ../bin/raservices raservices.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raevent.c
>>>>>>>>>> gcc -O3 -o ../bin/raevent raevent.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ragrep.c
>>>>>>>>>> gcc -O3 -o ../bin/ragrep ragrep.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratemplate.c
>>>>>>>>>> gcc -O3 -o ../bin/ratemplate ratemplate.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
>>>>>>>>>> making in ./ragraph
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>>>> sed 's+ at BINDIR@+/usr/local/bin+' ragraph.pl | \
>>>>>>>>>> sed 's+ at PERLBIN@+/usr/bin/perl+' > ../bin/ragraph
>>>>>>>>>> chmod +x ../bin/ragraph
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
>>>>>>>>>> making in ./ratop
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratop.c
>>>>>>>>>> gcc -O3 -o ../bin/ratop ratop.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
>>>>>>>>>> making in ./radium
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radium.c
>>>>>>>>>> ./radium.c: In function ‘RadiumParseResourceFile’:
>>>>>>>>>> ./radium.c:818: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
>>>>>>>>>> gcc -O3 -o radium radium.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lwrap -lnsl -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
>>>>>>>>>> making in ./radump
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radump.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-aodv.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-arp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-beep.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bfd.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bgp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bootp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-domain.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-krb.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ldp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-l2tp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-lmp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-msdp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ntp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pim.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pptp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-snmp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-syslog.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rip.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-telnet.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-tftp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isoclns.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ether.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-timed.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-radius.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rx.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-smb.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-stp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isakmp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-igmp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-dvmrp.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./smbutil.c
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./gmpls.c
>>>>>>>>>> gcc -O3 -o ../bin/radump radump.o print-aodv.o print-arp.o print-beep.o print-bfd.o print-bgp.o print-bootp.o print-domain.o print-krb.o print-ldp.o print-l2tp.o print-lmp.o print-msdp.o print-ntp.o print-pim.o print-pptp.o print-snmp.o print-syslog.o print-rip.o print-telnet.o print-tftp.o print-isoclns.o print-ether.o print-timed.o print-radius.o print-rx.o print-smb.o print-stp.o print-isakmp.o print-igmp.o print-dvmrp.o smbutil.o gmpls.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
>>>>>>>>>> making in ./ramysql
>>>>>>>>>> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasql.c
>>>>>>>>>> In file included from /usr/include/string.h:640,
>>>>>>>>>> from ./rasql.c:42:
>>>>>>>>>> In function ‘bzero’:
>>>>>>>>>> /usr/include/bits/string3.h:100: warning: call to__builtin___memset_chk will always overflow destination buffer
>>>>>>>>>> gcc -O3 -o ../bin/rasql rasql.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqlinsert.c
>>>>>>>>>> In file included from /usr/include/stdio.h:910,
>>>>>>>>>> from ../include/argus_util.h:195,
>>>>>>>>>> from ./rasqlinsert.c:53:
>>>>>>>>>> In function ‘snprintf’,
>>>>>>>>>> inlined from ‘ArgusProcessCommand’ at ./rasqlinsert.c:2531:
>>>>>>>>>> /usr/include/bits/stdio2.h:65: warning: call to__builtin___snprintf_chk will always overflow destination buffer
>>>>>>>>>> gcc -O3 -o ../bin/rasqlinsert rasqlinsert.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
>>>>>>>>>> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqltimeindex.c
>>>>>>>>>> gcc -O3 -o ../bin/rasqltimeindex rasqltimeindex.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
>>>>>>>>>> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>>>>>>>>>>
>>>>>>>>>> I will look into this later and see what alternatives there are... I will also test the compiled binaries later today.
>>>>>>>>>>
>>>>>>>>>> best,
>>>>>>>>>>
>>>>>>>>>> Javier Almillategui
>>>>>>>>>>
>>>>>>>>>> On Aug 12, 2010, at 5:35 PM, Carter Bullard wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Javier,
>>>>>>>>>>> The current client version to use that may fix this bug is argus-clients-3.0.3.17, which you can get here:
>>>>>>>>>>> http://qosient.com/argus/dev/argus-clients-3.0.3.17.tar.gz
>>>>>>>>>>>
>>>>>>>>>>> I believe that there was a memory leak that we fixed. If you try the new code, could you please send
>>>>>>>>>>> some email if the problem was fixed? That would be most helpful. If its not fixed, send more email.
>>>>>>>>>>>
>>>>>>>>>>> Carter
>>>>>>>>>>>
>>>>>>>>>>> On Aug 12, 2010, at 4:56 PM, Javier Almillategui wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> I have noticed a strange behavior in rasqlinsert (using argus client 3.0.2 compiled from source) in ubuntu 10.04 x86_64
>>>>>>>>>>>>
>>>>>>>>>>>> after running the rasqlinsert to load a database using the following options:
>>>>>>>>>>>>
>>>>>>>>>>>> rasqlinsert -w mysql://xxx:xxx@localhost/anondb/xxxtap_%Y_%m_%d -r xxx_tap_20100521_prepared.argus -s +2seq -s +3ltime -s +4dur -s +loss -s +1srcid -s -record -m none -M time 1d
>>>>>>>>>>>>
>>>>>>>>>>>> the memory consumption of rasqlinsert continues to increase to the point of using a significant part of my systems memory.
>>>>>>>>>>>>
>>>>>>>>>>>> it went from consuming ~16 MB to consuming 712 MB (at this point).
>>>>>>>>>>>>
>>>>>>>>>>>> is there any known memory leak in rasqlinsert??
>>>>>>>>>>>>
>>>>>>>>>>>> best,
>>>>>>>>>>>>
>>>>>>>>>>>> Javier Almillategui
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E 57th Street Suite 12D
>>>>> New York, New York 10022
>>>>>
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York 10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100929/ba2eb64b/attachment.html>
More information about the argus
mailing list