new argus-clients-3.0.7.28 on the server
Carter Bullard
carter at qosient.com
Mon May 19 22:57:29 EDT 2014
Hey Dave,
ArgusMySQLSelectProcess is using ArgusParser->RaParseDone as the
indicator to exit. ArgusProcessData() is setting this right before
it exits, indicating that we’ve finished parsing data.
Problem is that we’ve scheduled the inserts, from the parsing engine,
but we haven’t necessarily gotten them into the various queues.
Inserts are first put into the Select queue, to see if they are
in the database, and if they aren’t we do the Insert.
We should exit when both ArgusCloseDown and ArgusParser->RaParseDone
go high, not just when ArgusParser->RaParseDone goes high.
This is a slight change, so need to ring it out.
I can make that change tomorrow.
Carter
On May 19, 2014, at 10:44 PM, David Edelman <dedelman at iname.com> wrote:
> Very interesting with -D3 I see all of the tables being created (the source file is sparse, there are days with no activity at all.) Then there are two inserts scheduled and executed (I do find those records in the correct table). Then it looks like ArgusMySQLSelectProcess() closes and it's all over.
>
> By the way, ltime, dur, and bytes are actually set to 0 in the database:
>
>
> mysql> select ltime, dur, srcid, saddr, daddr, proto, bytes, sco, dco from xtyst_2013_10_01;
>
> +----------+----------+--------------+---------------+----------------+-------+-------+------+------+
>
> | ltime | dur | srcid | saddr | daddr | proto | bytes | sco | dco |
>
> +----------+----------+--------------+---------------+----------------+-------+-------+------+------+
>
> | 0.000000 | 0.000000 | 10.25.236.7 | 5.161.164.145 | 169.173.35.180 | udp | 0 | IR | US |
>
> | 0.000000 | 0.000000 | 169.185.96.6 | 5.161.164.145 | 169.185.208.76 | tcp | 0 | IR | US |
>
> +----------+----------+--------------+---------------+----------------+-------+-------+------+------+
>
> 2 rows in set (0.00 sec)
>
>
> Is there a good D value to show what's happening on the various threads?
>
>
> Many instance of RaProcessSPlitOptions and ArgusCreateSQLSaveTable omitted
>
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.684 RaProcessSplitOptions(xtyst_2013_09_27, 4096, 0x4ad00630): returns 0
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.695 ArgusCreateSQLSaveTable (xtyst_2013_09_27) returning
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.695 RaProcessSplitOptions(xtyst_2013_09_30, 4096, 0x4ad00630): returns 0
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.707 ArgusCreateSQLSaveTable (xtyst_2013_09_30) returning
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.707 RaProcessSplitOptions(xtyst_2013_10_01, 4096, 0x4ad00630): returns 0
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.718 ArgusCreateSQLSaveTable (xtyst_2013_10_01) returning
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.719 ArgusCloseInput(0x4ad00010) closing
>
> rasqlinsert[8835.00c77f42617f0000]: 2014-05-20-02:36:30.719 ArgusCloseInput(0x4ad00010) done
>
> rasqlinsert[8835.00d7ff42617f0000]: 2014-05-20-02:36:30.872 ArgusScheduleSQLQuery (0x4adf2010, 0x3708330, 0x3c0039f0, INSERT INTO argus.xtyst_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,sco,dco,record) VALUES ("0.000","0.000","10.25.236.7","5.161.164.145","169.173.35.180","udp","0","IR","US",...), 32) done
>
> rasqlinsert[8835.00d7ff42617f0000]: 2014-05-20-02:36:30.872 ArgusScheduleSQLQuery (0x4adf2010, 0x3708330, 0x3c004190, INSERT INTO argus.xtyst_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,sco,dco,record) VALUES ("0.000","0.000","169.185.96.6","5.161.164.145","169.185.208.76","tcp","0","IR","US",...), 32) done
>
> rasqlinsert[8835.00973849617f0000]: 2014-05-20-02:36:30.873 ArgusSQLQuery (INSERT INTO argus.xtyst_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,sco,dco,record) VALUES ("0.000","0.000","10.25.236.7","5.161.164.145","169.173.35.180","udp","0","IR","US",...))
>
> rasqlinsert[8835.00973849617f0000]: 2014-05-20-02:36:30.874 ArgusSQLQuery (INSERT INTO argus.xtyst_2013_10_01 (ltime,dur,srcid,saddr,daddr,proto,bytes,sco,dco,record) VALUES ("0.000","0.000","169.185.96.6","5.161.164.145","169.185.208.76","tcp","0","IR","US",...))
>
> rasqlinsert[8835.00973849617f0000]: 2014-05-20-02:36:30.874 ArgusMySQLInsertProcess: residual buffer Count 2 SQL Query len 947
>
> rasqlinsert[8835.00f7ff43617f0000]: 2014-05-20-02:36:30.874 ArgusMySQLSelectProcess() done!
>
> rasqlinsert[8835.00e77f43617f0000]: 2014-05-20-02:36:30.874 ArgusMySQLUpdateProcess() done!
>
> rasqlinsert[8835.00973849617f0000]: 2014-05-20-02:36:31.217 ArgusMySQLInsertProcess() done!
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.217 ArgusWindowClose () returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.217 RaParseComplete(caught signal 0)
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.217 ArgusShutDown (0)
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.217 ArgusWindowClose () returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.217 RaParseComplete(caught signal 0)
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.218 ArgusDeleteModeList () returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.218 ArgusDeleteFileList () returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.218 ArgusDeleteList (0x3707050, 4) returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.218 ArgusDeleteList (0x37070f0, 4) returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.218 ArgusDeleteLabeler (0x7f614adf2010, 0x3707d10) returning
>
> rasqlinsert[8835.8078e94a617f0000]: 2014-05-20-02:36:31.219 ArgusDeleteAggregator(0x7f614adf2010, 0x3708330) returned
>
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Monday, May 19, 2014 10:05 PM
> To: David Edelman
> Cc: Argus
> Subject: Re: [ARGUS] new argus-clients-3.0.7.28 on the server
>
> All should be good, regarding sco/dco and saddr/daddr swapping, at least that is the desire ;O)
>
> If you run with -D2 or -D3, you should see the sql statements that rasqlinsert() would be
>
> trying to generate.
>
> I was working on a problem where the process that generates the statements terminates
>
> without processing all the records on the queue, you may be running into that.
>
> Run with -D3, that should show if there is thread termination without writing anything.
>
> Carter
>
> On May 19, 2014, at 10:01 PM, David Edelman <dedelman at iname.com> wrote:
>
> > The ratop problem seems to be fixed. I'm still checking how sco and dco are
>
> > handles when matrix is used in an aggregation. I still have a problem with
>
> > rasqlinsert with the attached file. It seems to create the appropriate
>
> > tables, but they are all empty.
>
> >
>
> > rasqlinsert -M time 1d -M cache -r 5-161-164-145.argus.gz -w
>
> > mysql://argus@localhost/argus/xtest_%Y_%m_%d -m srcid matrix proto -s ltime
>
> > dur srcid saddr daddr proto bytes sco dco
>
> >
>
> > It's no better when I run it without -M cache
>
> >
>
> > --Dave
>
> >
>
> > -----Original Message-----
>
> > From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>
> > [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On
>
> > Behalf Of Carter Bullard
>
> > Sent: Sunday, May 18, 2014 9:03 PM
>
> > To: Argus
>
> > Subject: [ARGUS] new argus-clients-3.0.7.28 on the server
>
> >
>
> > Gentle people,
>
> > This new version is another step closer to release.
>
> > Fixes for ratop() seg fault, and various memory fixes.
>
> > Fixes for aggregators and matrix aggregation, especially
>
> > mysql table fetching of previously aggregated data.
>
> >
>
> > This can be fetched at the normal places:
>
> > http://qosient.com/argus/dev/argus-client-latest.tar.gz
>
> >
>
> > Thanks !!!
>
> > Carter
>
> >
>
> > On May 15, 2014, at 5:18 PM, Carter Bullard <carter at qosient.com> wrote:
>
> >
>
> >> Gentle people,
>
> >> new client software up loaded to fix the bugs found in
>
> >> racluster() aggregation when specifying idle and status
>
> >> timers for specific matched rules.
>
> >>
>
> >> http://qosient.com/argus/dev/argus-clients-latest.tar.gz
>
> >>
>
> >> Also, in the process, I found a bug in the Patricia tree
>
> >> algorithms, which could affect rafilteraddr() and some
>
> >> of the metadata programs. If you are doing address based
>
> >> labeling, you'll need to test out this new release.
>
> >>
>
> >> Thanks !!!!!
>
> >> Carter
>
> >
>
> > <5-161-164-145.argus.gz>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140519/b90d232a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140519/b90d232a/attachment.bin>
More information about the argus
mailing list