rasqlinsert reading a sparse single source file

[Carter Bullard]

Hey Dave,
rasqlinsert() has a lazy cache strategy, to control the impact on mysql() for inserting,
updating and deleting rows into the tables.  rasqlinsert() is actually ratop(), where
mysql() replaces ratop()s screen.

Depending on the command line options, such as "-M cache",  rasqlinsert() can
realize that a new table will need to be generated, as soon as a record is read in, but
schedule the table insert or update for that record for later.  Could be that we'll get 1000
records that need to be aggregated together, so wait to do that before pushing
data into mysql.  The lazy cache is managed in real time, not record timestamp time.

Could be that there is a problem with flushing the lazy cache on EOF, as I suspect
that rasqlinsert() will have processed all the data, and not even attempt a mysql
insert, until EOF, since you have so few records to process.

Also, remember, that there are command line options for rasqlinsert() that can
cause it to delete records from the database, if there are idle timeout conditions,
just like ratop().  So lets make sure that you're not turning those on.

So "-D7" should tell you what sql commands are being scheduled, and it should also
tell you which ones are actually being committed to the database.  That should be 
revealing.

Oh yes, data is always the best way to fix a problem !!!  And how it is that you're
calling rasqlinsert().

Thanks !!!!

Carter

On Oct 5, 2013, at 3:35 PM, "David Edelman" <dedelman at iname.com> wrote:

> If I use ra –R * -w /tmp/somefile – host 1.2.3.4 to extract the records for a single host over a few year period I end up with a file that I can read with ra. In my case, there are about 900 flow records. There are only 277 dates where there is flow activity and my database tables are built based on year_month_day.
> 
> 
> If I use rasqlinsert –r /tmp/somefile I end up with all the correct table built but most (not all) are empty. It doesn’t look like an aggregation problem and rasqlinsert knew that it had to create tables for each of the 277 dates.
> 
> Any thoughts? Any good –D parameters to enable? Carter, I can provide the source file privately if that will help.
> 
> 
> --Dave
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/960959a2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/960959a2/attachment.bin>