rasqlinsert reading a sparse single source file

Sun Oct 6 20:40:15 EDT 2013

I rebuilt with .devel and .debug and there is no question that all of the
tables are being created and only at the very end of the processing are any
data being written to any of the tables. It looks like not much is being
written:

rasqlinsert -D 7  -M time 1d -r /tmp/5-161-164-145.argus.gz -w
mysql://argus@localhost/argus/argus6_%Y_%m_%d -m srcid matrix proto -s ltime
dur srcid saddr daddr proto bytes

The last bits of the debug output and the only part of the output with a SQL
INSERT command looks like this, I suspect a dead thread, so gdb is my next
step.

rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusGenerateRecord (0x7f554401c840, 0) len 204
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusScheduleSQLQuery (0x5431f010, 0x42eb300, 0x4401c840, INSERT INTO
argus.argus6_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,record)
VALUES
("1380648451.046","0.000","10.25.236.7","5.161.164.145","169.173.35.180","ud
p","4500",...), 32) done
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461 ArgusCalloc (1,
32) returning 0x4c010d70
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusPushBackList (0x42ec380, 0x7f554c010d70, 1) returning 1
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusGenerateRecord (0x7f554401c4a0, 0) len 220
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusScheduleSQLQuery (0x5431f010, 0x42eb300, 0x4401c4a0, INSERT INTO
argus.argus6_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,record)
VALUES
("1380653153.836","21.105","169.185.96.6","5.161.164.145","169.185.208.76","
tcp","344",...), 32) done
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461 ArgusCalloc (1,
32) returning 0x4c0110f0
rasqlinsert[21496.00776a51557f0000]: 2013-10-07-00:29:21.461
ArgusPushBackList (0x42ec380, 0x7f554c0110f0, 1) returning 1
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:21.461 ArgusSQLQuery
(INSERT INTO argus.argus6_2013_10_01
(ltime,dur,srcid,saddr,daddr,proto,bytes,record) VALUES
("1380648451.046","0.000","10.25.236.7","5.161.164.145","169.173.35.180","ud
p","4500",...))
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:21.461 ArgusFree
(0x7f554c010d70)
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:21.461 ArgusSQLQuery
(INSERT INTO argus.argus6_2013_10_01
(ltime,dur,srcid,saddr,daddr,proto,bytes,record) VALUES
("1380653153.836","21.105","169.185.96.6","5.161.164.145","169.185.208.76","
tcp","344",...))
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:21.461 ArgusFree
(0x7f554c0110f0)
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:21.461
ArgusProcessSQLQueryList: residual buffer SQL Query len 938
rasqlinsert[21496.0067ea50557f0000]: 2013-10-07-00:29:21.470
ArgusProcessQueue (0x42ec120) returning 0
rasqlinsert[21496.0067ea50557f0000]: 2013-10-07-00:29:21.470
ArgusClientTimeout()
rasqlinsert[21496.00678b52557f0000]: 2013-10-07-00:29:24.490
ArgusMySQLProcess() done!
rasqlinsert[21496.40473c54557f0

From: Carter Bullard [mailto:carter at qosient.com] 
Sent: Sunday, October 06, 2013 12:07 PM
To: David Edelman
Cc: Argus
Subject: Re: rasqlinsert reading a sparse single source file

Hey Dave,
rasqlinsert() has a lazy cache strategy, to control the impact on mysql()
for inserting,
updating and deleting rows into the tables.  rasqlinsert() is actually
ratop(), where
mysql() replaces ratop()s screen.

Depending on the command line options, such as "-M cache",  rasqlinsert()
can
realize that a new table will need to be generated, as soon as a record is
read in, but
schedule the table insert or update for that record for later.  Could be
that we'll get 1000
records that need to be aggregated together, so wait to do that before
pushing
data into mysql.  The lazy cache is managed in real time, not record
timestamp time.

Could be that there is a problem with flushing the lazy cache on EOF, as I
suspect
that rasqlinsert() will have processed all the data, and not even attempt a
mysql
insert, until EOF, since you have so few records to process.

Also, remember, that there are command line options for rasqlinsert() that
can
cause it to delete records from the database, if there are idle timeout
conditions,
just like ratop().  So lets make sure that you're not turning those on.

So "-D7" should tell you what sql commands are being scheduled, and it
should also
tell you which ones are actually being committed to the database.  That
should be 
revealing.

Oh yes, data is always the best way to fix a problem !!!  And how it is that
you're
calling rasqlinsert().

Thanks !!!!

Carter

On Oct 5, 2013, at 3:35 PM, "David Edelman" <dedelman at iname.com
<mailto:dedelman at iname.com> > wrote:

If I use ra -R * -w /tmp/somefile - host 1.2.3.4 to extract the records for
a single host over a few year period I end up with a file that I can read
with ra. In my case, there are about 900 flow records. There are only 277
dates where there is flow activity and my database tables are built based on
year_month_day. 

If I use rasqlinsert -r /tmp/somefile I end up with all the correct table
built but most (not all) are empty. It doesn't look like an aggregation
problem and rasqlinsert knew that it had to create tables for each of the
277 dates.
Any thoughts? Any good -D parameters to enable? Carter, I can provide the
source file privately if that will help.

--Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/10b92cdd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6283 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131006/10b92cdd/attachment.bin>