argusarchive script

[Peter Van Epp]

	Pretty much correct. With the default directory layout, and a one 
hour cron job its desirable that the filename be the start time of the 
record so that when midnight comes and the logs cycle data, for the new day
is in the new directory. If the record spans midnight it is less clear what
should be done.  Dave's (thanks Dave!) script points out another thing I hadn't 
thought of / run in to yet (because I currently don't cycle logs hourly so 
the time changes exists inside a single argus record) which is that the 
current script will overwrite an hours worth of data when daylight saving
time occurs back by an hour. I'd classify that as a bug in the current 
script :-).  We also need to remember sort collation order when doing the 
alternate file naming so the default "sort the directory" still gets all the
files in the correct time sequence (Dave may have already thought of that when
adding a .1 to the end of the new file ...).
	I came across this extracting data from midnight to midnight to 
compare 1.8.1 output with 2.0.5 output from the same data stream to make 
sure my scripts are being changed correctly. With the default setup I get
23:00 to 23:00 (by selecting everything in the directory for the day with 
Russell's argus:archive script). I fixed that for now by creative file moving
:-) and the note to do something more permenant about it in future and thus
this discussion.

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

> 
> Hey Peter,
> I think I hear you saying that the right way to do
> it is to split the file along hard time boundaries.
> That way if the file spans 12 midnight or noon, you
> would like to generate the before and after file.
> 
> But short of this, we could read the first record
> to see what the start time is in the file and use
> that as guide for the file timestamp.
> 
> Carter
> 
> Carter Bullard
> QoSient, LLC
> 300 E. 56th Street, Suite 18K
> New York, New York  10022
> 
> carter at qosient.com
> Phone +1 212 588-9133
> Fax   +1 212 588-9134
> http://qosient.com
> 
> > -----Original Message-----
> > From: owner-argus-info at lists.andrew.cmu.edu 
> > [mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of 
> > David J Brumley
> > Sent: Friday, May 10, 2002 1:04 PM
> > To: Peter Van Epp
> > Cc: argus-info at lists.andrew.cmu.edu
> > Subject: Re: argusarchive script
> > 
> > 
> > 
> > I've included my rotate script, which basically takes current 
> > time - 1/2 hour to come up with last hours date. Seems to 
> > work. It does take into consideration the daylight savings 
> > time when calculating the hour, though how to deal with it is 
> > implementation dependent :)
> > 
> > -david
> > 
> > 
> > > 	Now I've started converting to 2.x (you don't want to 
> > rush these 
> > > things after all :-)), I've hit my first rub. The 
> > argusarchive script 
> > > file naming scheme has a short coming (at least as far as I'm 
> > > concerned) in that the first and last files of the day are in the 
> > > wrong place (i.e in "yesterday" or "tomorrow". At present the file 
> > > argus.2002.04.29.00.00.00.gz (for instance) really belongs in 
> > > archive/2002/04/28/argus.2002.04.28.23.00.00.gz not 
> > > archive/2002/04/29/argus.2002.04.29.00.00.00.gz (where it is) since 
> > > the data in it starts at 04/28 23:00 to 04/28 23:59. While 
> > this could 
> > > be compensated for in the extraction perl script, I think the right 
> > > answer is probably to change the archive script to "do the right 
> > > thing". The question is how best to do that. The first thing that 
> > > occurs to me is generate the file name when the switch happens (as 
> > > now) but save it in a file in the file system for the next 
> > switch and 
> > > use the file name saved at the last switch (or startup which is a 
> > > wrinkle that needs thinking about) as the archive file 
> > name. Essentially I want the file name to be the date at the 
> > start of the capture period not the end (because then data 
> > for a day is in a single directory as it should be).
> > > 	Anyone have any clever suggestions about how to easily 
> > achieve this
> > > without storing file names (with the attendent messyness of 
> > needing to consider
> > > setting an appropriate filename on startup) or should we 
> > just store the 
> > > filename and be done with it?
> > > 
> > > Peter Van Epp / Operations and Technical Support
> > > Simon Fraser University, Burnaby, B.C. Canada
> > 
> > -- 
> > David Brumley
> > 650.723.2445
> > 
> 
> 
>