rasqlinsert memory consumption
Carter Bullard
carter at qosient.com
Fri Aug 13 08:54:32 EDT 2010
Hey Javier,
The odd release numbers are development versions, so there will be a few compile warnings, so no worries.
However, the buffer overflow warnings you highlighted seem to be bogus. The first one is in a system
include file, which of course could happen, but in this case isn't, and the one in rasqlinsert.c is obviously
an error. We're using snprintf and the number of bytes copied (BUFSIZ, which should be 1K) is 1/4 the size
of the array (4K). These buffers are the full pathname of the archive variable, which we're not using at this
time.
Go ahead and run the programs to see if all is better.
Carter
On Aug 13, 2010, at 6:52 AM, Javier Almillategui wrote:
> Hi Carter,
>
> I have compiled version 3.0.3 on my system and got some interesting warnings regarding buffer overflows:
>
> making in ./common
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_main.c
> rm -f ../lib/argus_parse.a; ar qc ../lib/argus_parse.a argus_main.o
> ranlib ../lib/argus_parse.a
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_code.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_filter.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_util.c
> ./argus_util.c: In function ‘ArgusParseResourceFile’:
> ./argus_util.c:1680: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
> ./argus_util.c: In function ‘ArgusLog’:
> ./argus_util.c:18124: warning: format not a string literal and no format arguments
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_auth.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_parser.c
> flex -Pargus_ -t scanner.l > $$.scanner.c; mv $$.scanner.c scanner.c
> bison -y -p argus_ -d grammar.y
> mv y.tab.c grammar.c
> mv y.tab.h tokdefs.h
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./scanner.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -Dyylval=argus_lval -c grammar.c
> sed -e 's/.*/char version[] = "&";/' ./../VERSION > version.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./version.c
> rm -f ../lib/argus_common.a; ar qc ../lib/argus_common.a argus_code.o argus_filter.o argus_util.o argus_auth.o argus_parser.o scanner.o grammar.o version.o
> ranlib ../lib/argus_common.a
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_client.c
> ./argus_client.c: In function ‘ArgusParseAggregator’:
> ./argus_client.c:11758: warning: ignoring return value of ‘strtol’, declared with attribute warn_unused_result
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_label.c
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_grep.c
> rm -f ../lib/argus_client.a; ar qc ../lib/argus_client.a argus_client.o argus_label.o argus_grep.o
> ranlib ../lib/argus_client.a
> gcc -O3 -I. -I../include -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -DARGUS_SYSLOG -c ./argus_event.c
> rm -f ../lib/argus_event.a; ar qc ../lib/argus_event.a argus_event.o
> ranlib ../lib/argus_event.a
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/common'
> making in ./include
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
> make[1]: Nothing to be done for `all'.
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/include'
> making in ./clients
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ra.c
> gcc -O3 -o ../bin/ra ra.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racount.c
> gcc -O3 -o ../bin/racount racount.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasort.c
> gcc -O3 -o ../bin/rasort rasort.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rasplit.c
> gcc -O3 -o ../bin/rasplit rasplit.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rabins.c
> gcc -O3 -o ../bin/rabins rabins.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./racluster.c
> gcc -O3 -o ../bin/racluster racluster.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raconvert.c
> gcc -O3 -o ../bin/raconvert raconvert.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastrip.c
> gcc -O3 -o ../bin/rastrip rastrip.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rahisto.c
> gcc -O3 -o ../bin/rahisto rahisto.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ralabel.c
> gcc -O3 -o ../bin/ralabel ralabel.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapolicy.c
> gcc -O3 -o ../bin/rapolicy rapolicy.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ranonymize.c
> gcc -O3 -o ../bin/ranonymize ranonymize.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rapath.c
> gcc -O3 -o ../bin/rapath rapath.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rastream.c
> gcc -O3 -o ../bin/rastream rastream.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratree.c
> gcc -O3 -o ../bin/ratree ratree.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rafilteraddr.c
> gcc -O3 -o ../bin/rafilteraddr rafilteraddr.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratimerange.c
> gcc -O3 -o ../bin/ratimerange ratimerange.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> sed 's+ at PERLBIN@+/usr/bin/perl+' raports.pl > ../bin/raports
> chmod +x ../bin/raports
> sed 's+ at PERLBIN@+/usr/bin/perl+' rahosts.pl > ../bin/rahosts
> chmod +x ../bin/rahosts
> sed 's+ at PERLBIN@+/usr/bin/perl+' radark.pl > ../bin/radark
> chmod +x ../bin/radark
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./rauserdata.c
> gcc -O3 -o ../bin/rauserdata rauserdata.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raservices.c
> gcc -O3 -o ../bin/raservices raservices.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./raevent.c
> gcc -O3 -o ../bin/raevent raevent.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ragrep.c
> gcc -O3 -o ../bin/ragrep ragrep.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratemplate.c
> gcc -O3 -o ../bin/ratemplate ratemplate.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/clients'
> making in ./ragraph
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
> sed 's+ at BINDIR@+/usr/local/bin+' ragraph.pl | \
> sed 's+ at PERLBIN@+/usr/bin/perl+' > ../bin/ragraph
> chmod +x ../bin/ragraph
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ragraph'
> making in ./ratop
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./ratop.c
> gcc -O3 -o ../bin/ratop ratop.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ratop'
> making in ./radium
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radium.c
> ./radium.c: In function ‘RadiumParseResourceFile’:
> ./radium.c:818: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
> gcc -O3 -o radium radium.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lwrap -lnsl -lm -lz -lpthread -L/usr/lib -lGeoIP
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radium'
> making in ./radump
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./radump.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-aodv.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-arp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-beep.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bfd.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bgp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-bootp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-domain.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-krb.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ldp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-l2tp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-lmp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-msdp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ntp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pim.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-pptp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-snmp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-syslog.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rip.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-telnet.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-tftp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isoclns.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-ether.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-timed.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-radius.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-rx.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-smb.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-stp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-isakmp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-igmp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./print-dvmrp.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./smbutil.c
> gcc -O3 -I. -I../include -I../common -I/usr/include -DHAVE_CONFIG_H -c ./gmpls.c
> gcc -O3 -o ../bin/radump radump.o print-aodv.o print-arp.o print-beep.o print-bfd.o print-bgp.o print-bootp.o print-domain.o print-krb.o print-ldp.o print-l2tp.o print-lmp.o print-msdp.o print-ntp.o print-pim.o print-pptp.o print-snmp.o print-syslog.o print-rip.o print-telnet.o print-tftp.o print-isoclns.o print-ether.o print-timed.o print-radius.o print-rx.o print-smb.o print-stp.o print-isakmp.o print-igmp.o print-dvmrp.o smbutil.o gmpls.o ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/radump'
> making in ./ramysql
> make[1]: Entering directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasql.c
> In file included from /usr/include/string.h:640,
> from ./rasql.c:42:
> In function ‘bzero’:
> /usr/include/bits/string3.h:100: warning: call to __builtin___memset_chk will always overflow destination buffer
> gcc -O3 -o ../bin/rasql rasql.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqlinsert.c
> In file included from /usr/include/stdio.h:910,
> from ../include/argus_util.h:195,
> from ./rasqlinsert.c:53:
> In function ‘snprintf’,
> inlined from ‘ArgusProcessCommand’ at ./rasqlinsert.c:2531:
> /usr/include/bits/stdio2.h:65: warning: call to __builtin___snprintf_chk will always overflow destination buffer
> gcc -O3 -o ../bin/rasqlinsert rasqlinsert.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP -lncurses
> gcc -O3 -I. -I../include -I../common -I/usr/include -I/usr/include/mysql -DHAVE_CONFIG_H -c ./rasqltimeindex.c
> gcc -O3 -o ../bin/rasqltimeindex rasqltimeindex.o -L/usr/lib/mysql -lmysqlclient_r ../lib/argus_parse.a ../lib/argus_common.a ../lib/argus_client.a -lm -lz -lpthread -L/usr/lib -lGeoIP
> make[1]: Leaving directory `/home/jalmillategui/Downloads/argus-clients-3.0.3.17/ramysql'
>
> I will look into this later and see what alternatives there are... I will also test the compiled binaries later today.
>
> best,
>
> Javier Almillategui
>
> On Aug 12, 2010, at 5:35 PM, Carter Bullard wrote:
>
>> Hey Javier,
>> The current client version to use that may fix this bug is argus-clients-3.0.3.17, which you can get here:
>> http://qosient.com/argus/dev/argus-clients-3.0.3.17.tar.gz
>>
>> I believe that there was a memory leak that we fixed. If you try the new code, could you please send
>> some email if the problem was fixed? That would be most helpful. If its not fixed, send more email.
>>
>> Carter
>>
>> On Aug 12, 2010, at 4:56 PM, Javier Almillategui wrote:
>>
>>> Hi All,
>>>
>>> I have noticed a strange behavior in rasqlinsert (using argus client 3.0.2 compiled from source) in ubuntu 10.04 x86_64
>>>
>>> after running the rasqlinsert to load a database using the following options:
>>>
>>> rasqlinsert -w mysql://xxx:xxx@localhost/anondb/xxxtap_%Y_%m_%d -r xxx_tap_20100521_prepared.argus -s +2seq -s +3ltime -s +4dur -s +loss -s +1srcid -s -record -m none -M time 1d
>>>
>>> the memory consumption of rasqlinsert continues to increase to the point of using a significant part of my systems memory.
>>>
>>> it went from consuming ~16 MB to consuming 712 MB (at this point).
>>>
>>> is there any known memory leak in rasqlinsert??
>>>
>>> best,
>>>
>>> Javier Almillategui
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100813/8da49b0b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3681 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100813/8da49b0b/attachment.bin>
More information about the argus
mailing list