new clients rc.62 on the server - description of rastream()

[Terry Burton]

On 10/31/07, Carter Bullard <carter at qosient.com> wrote:
> Sorry for the inconvenience!!  I haven't seen this, but I'll try to reproduce.  A few questions.
> Platform (intel/ppc/sparc), available memory, estimated record load and how many
> probes?

Hi Carter,

The platform is i686 GNU/Linux using pretty standard Debian Etch.
~40,000 records/min aggregated from three probes (2 netflow + 1 SPAN)
using radium. Host has 2GB RAM + 1.5GB swap. Attached is last night's
memory plot for the host. Guess what time I switched on rastream ;-)

You may want to wait for the result of my testing before putting too
much effort into this yourself.


Thanks again,

Tez


>
> Thanks for the email!!!
>
> Carter
>
> Carter Bullard
> QoSient LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
> -----Original Message-----
> From: "Terry Burton" <tez at terryburton.co.uk>
>
> Date: Wed, 31 Oct 2007 12:52:56
> To:Argus <argus-info at lists.andrew.cmu.edu>
> Cc:"Carter Bullard" <carter at qosient.com>
> Subject: Re: [ARGUS] new clients rc.62 on the server - description of rastream()
>
>
> On 10/30/07, Carter Bullard <carter at qosient.com> wrote:
> > rastream() is now ready to use in production situations.  It is the
> > replacement
> > for argusarchive, and is designed to make archive generation much
> > easier.
> > It is a designed as a persistent raspilt() that can process archive
> > files as they
> > are closed.  The example shell script that is included in the
> > distribution
> <...snip...>
>
> Hi,
>
> rastream is an excellent tool and I have switched over to using this
> for our logging and post-processing, rather than using the "-w ...."
> option to argus/radium.
>
> It does appear to leak memory though as the process fell over after
> about 8 hours leaving the following in syslog:
>
> Oct 31 08:25:15 mink rastream[24204]: 08:25:15.963144 ArgusRunScript
> (/srv/argus/archive/2007-10-31/xxx.yyy.7.1-08:20:00.arg) fork() error
> Cannot allocate memory
> Oct 31 08:25:16 mink rastream[24204]: 08:25:16.169338 ArgusRunScript
> (/srv/argus/archive/2007-10-31/xxx.yyy.7.1-00:10:00.arg) fork() error
> Cannot allocate memory
>
> i686 GNU/Linux, invoked as rastream -X -S localhost:569 -M time 5m -B
> 10s -f /bin/true -w
> /srv/argus/archive/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>
> I will investigate this further, hopefully later today.
>
> A suggestion:
>
> Would it be possible to present the template variables, ($srcid, %Y,
> %M, et.al.) to the "-f script", perhaps via environment variables?
> Currently the post-processing script has to reparse these out of its
> "-r parameter".
>
>
> Thanks again,
>
> Tez
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graph_image.php.png
Type: image/png
Size: 32672 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20071031/e78f56c4/attachment.png>