argus memory usage

[Carter Bullard]

Hey Peter,
I still think you may be starving your ArgusOutput thread, which of course
would cause the problem where you don't get any records, and no file
recreation on moving the file.

So the memory strategy is "the ArgusModel allocates buffers that are to 
written
out of the argus, (flow status records) and passes them to the ArgusOutput
processor, who writes them to all the clients that want a copy, and then
the ArgusOutput process deletes them".   So two threads having to share
a message queue, and one is allocating, and one is deleteing.

For efficiency, we are keeping a pool of buffers, so that argus can respond
to quick surges in flows, and so you won't necessarily see the ArgusOutput
delete the records until later, so it holding onto them is not bad, but 500K
records is probably too much.

What I'm going to do tonight is remove the .threads tag in the distribution,
and then change the buffer pool manager to  be a little dumber, to see if
that doesn't help.

If you could run an argus without any clients, to see if the bug is in the
ArgusModeler vs the ArgusOutputProcessor.  If it is running fine, then
run with only one client, and then with two (remember this is how we
got here with the memory problems :o)

That would help a great deal.

Carter







Carter

Peter Van Epp wrote:
> 	I turned on logging of ArgusCalloc Malloc and Free and then wrote 
> a perl script to parse the output and match up allocs and frees. We seem to have
> a lot less frees than I think we should have (which would explain the leak :-)):
>
> across about this time period:
>
> hcids:/scratch # head debug.log
> argus[7006]: 18 Aug 07 19:20:41.170516 ArgusCalloc (1, 1088) returning 0x1017f010
> argus[7006]: 18 Aug 07 19:20:41.170732 ArgusCalloc (1, 64) returning 0x1017f500
> ...
> hcids:/scratch # tail debug.log
> ...
> argus[7006]: 18 Aug 07 19:41:38.954361 ArgusMallocListRecord (672) returning 0x10c15b28
> argus[7006]: 18 Aug 07 19:41:38.954400 ArgusCalloc (130, 4) returning 0x2be15b40
>
> we see many more m/callocs than frees:
>
> 0x1c0bcae0 via m
> 0x1f3d44b0 via m
> Counters: Calloc: 337335 Malloc 238607 Free 18240
>
> (it is also keeping state but didn't see any double frees which is good)
> Around 500,000 records allocated and only 18,000 freed again which seems too
> small to me.
>
> Peter Van Epp / Operations and Technical Support 
> Simon Fraser University, Burnaby, B.C. Canada
>
>