Archive management question/feature request

[Carter Bullard]

Hey Jesse,
Sorry for the delayed response.
Having a daily script that goes through and tosses directories of a days worth
of records is pretty straight forward, which can support a fixed time retention
strategy.  And to modify that to toss days until total filesystem size is "<"
the critical number is not hard either. Better implemented as a shell or perl
or whatever scripts, rather than C programs, maybe ….

We have tools that process based on size of files (rasplit), and we have
database methods that allow us to keep very tight track of time (rasqltimeindex),
so that you can deal with the insertion and deletion of data within
the archive, while maintaining some temporal properties.  Putting this
together to provide some archive management would be a good project.

We'll need just a bit better definition that what you have, such
as configuration methods, error recovery and some automation, but
if you want to put some detail to configuration, I can provide
some development cycles.

Carter


On Sep 11, 2013, at 9:56 AM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Hi all,
> 
> So rasplit/rastream are great for organizing a disk-based archive of argus data. I'm wondering: how difficult it would be to include some features to manage the size of the archive?
> 
> The scenario I'm imagining is that when rasplit/rastream starts to write a new file, it also checks how much disk space is used under it's 'base' path. If size > limit, remove files until space == target_limit. Another potentially useful model would be to set a limit on the number of files created, so you could in a time-based mode ensure you keep at least X days of flows. 
> 
> I do this with a bash script currently, but was thinking that this might be a useful thing to be included in the clients. Other collection tools such as tcpdump and nprobe support this kind of archive management, so it would seem like there are some examples out there that code could potentially even be borrowed from.
> 
> What are the groups thoughts? Good idea? Bad? Would you like to see this? My own C/C++ skills are so ancient they are nearly forgotten, but I would like to help see this feature implemented.
> 
> Cheers,
> 
> Jesse
> 
> -- 
> Jesse Bowling
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130926/1c68cddd/attachment.bin>