new clients rc.62 on the server - description of rastream()
Terry Burton
tez at terryburton.co.uk
Wed Oct 31 14:47:52 EDT 2007
On Oct 31, 2007 2:25 PM, Terry Burton <tez at terryburton.co.uk> wrote:
> On 10/31/07, Carter Bullard <carter at qosient.com> wrote:
> > Sorry for the inconvenience!! I haven't seen this, but I'll try to reproduce. A few questions.
> > Platform (intel/ppc/sparc), available memory, estimated record load and how many
> > probes?
<...snip...>
> The platform is i686 GNU/Linux using pretty standard Debian Etch.
> ~40,000 records/min aggregated from three probes (2 netflow + 1 SPAN)
> using radium. Host has 2GB RAM + 1.5GB swap. Attached is last night's
> memory plot for the host. Guess what time I switched on rastream ;-)
>
> You may want to wait for the result of my testing before putting too
> much effort into this yourself.
Unfortunately I didn't find the time today to have a deep look into
the problem, however I did get a chance to run the rastream process
through valgrind for about 15 mins (output below).
Tomorrow is another day... :-)
Warm regards,
Terry
----
root at mink:~# valgrind --leak-check=full
~tez/argus-clients-3.0.0.rc.62/bin/rastream -X -S localhost:569 -M
time 5m -B 10s -f /bin/true -w
/srv/argus/archive/%Y-%m-%d/\$srcid-%H:%M:%S.arg
==13514== Memcheck, a memory error detector.
==13514== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==13514== Using LibVEX rev 1658, a library for dynamic binary translation.
==13514== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==13514== Using valgrind-3.2.1-Debian, a dynamic binary
instrumentation framework.
==13514== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==13514== For more details, rerun with: -v
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010C4E: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010C5D: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010C6C: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010DDC: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010DDC: (within /lib/ld-2.3.6.so)
==13514== by 0x4004B78: (within /lib/ld-2.3.6.so)
==13514== by 0x4006792: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514==
==13514== Invalid read of size 4
==13514== at 0x4010DE9: (within /lib/ld-2.3.6.so)
==13514== by 0x4004B78: (within /lib/ld-2.3.6.so)
==13514== by 0x4006792: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== Address 0x41CB404 is 36 bytes inside a block of size 37 alloc'd
==13514== at 0x401D38B: malloc (vg_replace_malloc.c:149)
==13514== by 0x4006B83: (within /lib/ld-2.3.6.so)
==13514== by 0x416458F: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4008ED5: (within /lib/ld-2.3.6.so)
==13514== by 0x41649E4: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4008B2E: (within /lib/ld-2.3.6.so)
==13514== by 0x41649E4: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x4163FFE: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4115E8A: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4116A38: getaddrinfo (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x80834BA: ArgusAddHostList (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804EF62: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== Thread 2:
==13514== Conditional jump or move depends on uninitialised value(s)
==13514== at 0x4010E21: (within /lib/ld-2.3.6.so)
==13514== by 0x4164292: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x416681C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x400B44E: (within /lib/ld-2.3.6.so)
==13514== by 0x416687D: __libc_dlopen_mode (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FEC3: __nss_lookup_function (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x413FFBF: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4141A75: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x41412A8: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4146822: getservent_r (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4140BD5: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x4146717: getservent (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514==
==13514== Thread 1:
==13514== Syscall param write(buf) points to uninitialised byte(s)
==13514== at 0x4000792: (within /lib/ld-2.3.6.so)
==13514== by 0x40C0F94: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40C122E: _IO_do_write (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40C1B2D: _IO_file_overflow (in
/lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40C112A: _IO_file_xsputn (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40B7F7E: fwrite (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x8078AE6: ArgusWriteNewLogfile (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804B324: RaSendArgusRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804C748: ArgusClientTimeout (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E80: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== Address 0x40211EA is not stack'd, malloc'd or (recently) free'd
==13514==
==13514== Syscall param write(buf) points to uninitialised byte(s)
==13514== at 0x4000792: (within /lib/ld-2.3.6.so)
==13514== by 0x40C0F94: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40C122E: _IO_do_write (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40C1A0B: _IO_file_sync (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x40B6E61: fflush (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x804C919: ArgusClientTimeout (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E80: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== Address 0x402101E is not stack'd, malloc'd or (recently) free'd
==13514==
==13514== ERROR SUMMARY: 5491 errors from 11 contexts (suppressed: 15 from 1)
==13514== malloc/free: in use at exit: 131,786,684 bytes in 452,475 blocks.
==13514== malloc/free: 1,868,435 allocs, 1,415,960 frees, 364,251,624
bytes allocated.
==13514== For counts of detected errors, rerun with: -v
==13514== searching for pointers to 452,475 not-freed blocks.
==13514== checked 139,170,140 bytes.
==13514==
==13514==
==13514== 14 (8 direct, 6 indirect) bytes in 1 blocks are definitely
lost in loss record 2 of 48
==13514== at 0x401C6CA: calloc (vg_replace_malloc.c:279)
==13514== by 0x809327B: ArgusNewAggregator (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804C3DC: ArgusSetCurrentWfile (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804C7AC: ArgusClientTimeout (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E80: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514==
==13514== 68 bytes in 1 blocks are possibly lost in loss record 9 of 48
==13514== at 0x401C6CA: calloc (vg_replace_malloc.c:279)
==13514== by 0x400E188: (within /lib/ld-2.3.6.so)
==13514== by 0x400E24B: _dl_allocate_tls (in /lib/ld-2.3.6.so)
==13514== by 0x4053846: pthread_create@@GLIBC_2.1 (in
/lib/tls/i686/cmov/libpthread-2.3.6.so)
==13514== by 0x8050F06: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514==
==13514== 408 (312 direct, 96 indirect) bytes in 2 blocks are
definitely lost in loss record 18 of 48
==13514== at 0x401C6CA: calloc (vg_replace_malloc.c:279)
==13514== by 0x8086109: setArgusWfile (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804F351: ArgusParseArgs (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x80504B0: ArgusMainInit (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x80507FF: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514==
==13514== 322 bytes in 6 blocks are definitely lost in loss record 19 of 48
==13514== at 0x401D38B: malloc (vg_replace_malloc.c:149)
==13514== by 0x40CCE4F: strdup (in /lib/tls/i686/cmov/libc-2.3.6.so)
==13514== by 0x804A92A: ArgusRunScript (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804BC5C: ArgusSetCurrentWfile (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804C7AC: ArgusClientTimeout (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E80: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514==
==13514== 58,904 bytes in 74 blocks are possibly lost in loss record 38 of 48
==13514== at 0x401C6CA: calloc (vg_replace_malloc.c:279)
==13514== by 0x809BBAD: ArgusCopyRecordStruct (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x809C384: ArgusAlignRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804CB62: RaProcessThisRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804D30E: RaProcessRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x808237C: ArgusHandleDatum (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x80950EA: ArgusReadStreamSocket (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E57: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514==
==13514== 1,840,352 (1,838,760 direct, 1,592 indirect) bytes in 2,310
blocks are definitely lost in loss record 45 of 48
==13514== at 0x401C6CA: calloc (vg_replace_malloc.c:279)
==13514== by 0x809BBAD: ArgusCopyRecordStruct (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x809C384: ArgusAlignRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804CB62: RaProcessThisRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x804D30E: RaProcessRecord (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x808237C: ArgusHandleDatum (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x80950EA: ArgusReadStreamSocket (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8095E57: ArgusReadStream (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514== by 0x8050D4B: main (in
/home/tb33/argus-clients-3.0.0.rc.62/bin/rastream)
==13514==
==13514== LEAK SUMMARY:
==13514== definitely lost: 1,839,402 bytes in 2,319 blocks.
==13514== indirectly lost: 1,694 bytes in 5 blocks.
==13514== possibly lost: 58,972 bytes in 75 blocks.
==13514== still reachable: 129,886,616 bytes in 450,076 blocks.
==13514== suppressed: 0 bytes in 0 blocks.
==13514== Reachable blocks (those to which a pointer was found) are not shown.
==13514== To see them, rerun with: --show-reachable=yes
More information about the argus
mailing list