argus-clients-3.0.2.tar.gz with mysql support
carter at qosient.com
carter at qosient.com
Wed Mar 4 20:40:07 EST 2009
Hey Pablo,
Right before I sent the code out, I added some code to explicitly close any open files, in the routine RaParseComplete. I suspect that that code is being run by another thread, while the output thread is flushing its queue.
If you comment out that section of code, it will probably work?
Carter
Sent from my Verizon Wireless BlackBerry
-----Original Message-----
From: "Pablo J. Rebollo-Sosa" <Pablo.Rebollo at ece.uprm.edu>
Date: Wed, 04 Mar 2009 19:42:58
To: <carter at qosient.com>
Cc: Argus<argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] argus-clients-3.0.2.tar.gz with mysql support
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
isCarter,
I ran the the tests with racluster and the outputs are the same.
root at argus:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w
- - | racount
racount records total_pkts src_pkts dst_pkts
total_bytes src_bytes dst_bytes
sum 2334 336945 193790 143155
276763462 239874647 36888815
root at argus:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w
- - | ra -w - | racount
racount records total_pkts src_pkts dst_pkts
total_bytes src_bytes dst_bytes
sum 2334 336945 193790 143155
276763462 239874647 36888815
I still thinking that there is a problem. For example I tried the
following.
root at nsm:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w
- - | rasort -m bytes -w - | ra -N 5
StartTime Flgs Proto SrcAddr Sport Dir
DstAddr Dport TotPkts TotBytes State
17:00:02.870130 e ip 74.125.0.163 <->
0.0.0.0 25377 27162034 CON
17:00:43.147390 e ip 68.142.122.70 <->
0.0.0.0 23169 21108846 CON
17:00:02.635304 e ip 63.251.219.114 <->
0.0.0.0 21325 19133350 CON
17:00:02.075566 e ip 66.90.64.10 <->
0.0.0.0 19430 18250118 CON
17:01:13.549650 e ip 74.125.0.214 <->
0.0.0.0 15158 14123330 CON
- From the previous output the maximum total bytes is 27162034. But if the
"- net 136.145.34.0/24" filter is applied I get a bigger value for total
bytes (686322024).
root at nsm:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w
- - - net 136.145.34.0/24 | rasort -m bytes -w - | ra -N 5
StartTime Flgs Proto SrcAddr Sport Dir
DstAddr Dport TotPkts TotBytes State
17:00:00.584726 e ip 136.145.34.11 <->
0.0.0.0 669192 686322024 CON
17:00:00.584726 e ip 96.7.19.16 <->
0.0.0.0 669185 686321500 CON
17:00:00.099711 eU F ip 136.145.34.80 <->
0.0.0.0 71461 50620443 CON
17:00:43.147390 e ip 68.142.122.70 <->
0.0.0.0 23140 21093742 CON
17:00:01.903740 e ip 136.145.34.81 <->
0.0.0.0 23002 21043963 CON
Any clues?
Best regards,
Pablo J. Rebollo
carter at qosient.com wrote:
> Use racount instead of wc, but I suspect a problem.
> So instead of " | wc -l"
> Try " -w - | racount"
>
> Racluster maybe leaving a few records in its output when it closes terminates. I'll take a look today!!
>
> Carter
>
> Sent from my Verizon Wireless BlackBerry
>
> -----Original Message-----
> From: "Pablo J. Rebollo-Sosa" <Pablo.Rebollo at ece.uprm.edu>
>
> Date: Tue, 03 Mar 2009 17:52:56
> To: Carter Bullard<carter at qosient.com>
> Cc: Argus<argus-info at lists.andrew.cmu.edu>
> Subject: Re: [ARGUS] argus-clients-3.0.2.tar.gz with mysql support
>
>
> Dear Carter,
>
> I'm testing the new clients and noticed odd results with ra. When using
> racluster with a specific file I get certain amount of lines
>
> server# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr | wc -l
> 5555
>
> The problem is when using racluster with -w option with ra. When
> running the command I get fewer amount of lines.
>
> racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w - | ra -r -
> | wc -l
> 2334
>
> Any suggestions?
>
> Best regards,
>
> Pablo J. Rebollo
>
> Carter Bullard wrote:
>> Gentle people,
>> First pass at the new argus-clients distribution is on the dev server.
>> ftp:/qosient.com/dev/argus-3.0/argus-clients-3.0.2.tar.gz
>
>> First pass because there will be modifications before its released,
>> as the user data analysis programs still need a little tweak.
>
>> This version addresses many problems, particularly those
>> relating to backward compatibility to argus-2.x streams.
>> I have not had a chance to directly test the changes on
>> some of the bugs on the list but I suspect that this version
>> should fix those backward compatibility bugs.
>
>> If you try the code, and it doesn't have your issue fixed,
>> please, please, please, send email, so that I can get those
>> issues dealt with.
>
>> I am pleased to say that the database programs, rasqlinsert()
>> and rasql() are mostly ready to go. I don't have a manpage yet,
>> so hopefully the "-h" option will give you guidance.
>
>> I will be sending out sometime this week detail on the use of
>> rasqlinsert(), the format of the database url that is needed to
>> access database data, and the concepts of rasql() and why
>> its needed.
>
>> If you want to give rasqlinsert a run, like loading tables from
>> files, try these types of commands:
>
>> rasqlinsert -r file -w mysql://user@host/db/table -m none
>
>> This will load the table 'db.table' with the records, and the
>> fields will be those that you would expect to be printed if
>> you had run ra against the file. To modify the schema, just
>> use the "-s field" command.
>
>> The "-m none" removes any keys that rasqlinsert() may have
>> wanted to use based on your .rarc file, so MySQL won't
>> complain about DUPLICATE inserts into the table.
>
>> If you then run these programs:
>
>> rasql -r mysql://user@host/db/table
>
>> or
>
>> rasqlinsert -r mysql://user@host/db/table
>
>> rasqlinsert() will look like ratop(), but its data will come from
>> the MySQL tables.
>
>> rasqlinsert pokes the actual binary record into the database,
>> along with ascii representations of the attributes. This is
>> so programs like rasql() can get argus records, rather
>> than ascii text out of the database. If you want to get rid
>> of the binary BLOBs, use "-s -record". rasql(), when reading
>> this type of table, will just return, without any data.
>
>> A set of programs I use a lot are:
>
>> rabins -S localhost -M time 30s -B 5s -w - | \
>> rasqlinsert -r - -w mysql://user@host/ratop/flowTable -m none
>
>> This reads data from a live stream, holds it for 30s, aggregating
>> common records together, and then pokes it into the database
>> table. This table will grow forever with argus records, but you can
>> see how something very simple like this can be the base of
>> a large flow system.
>
>> Hope all is most excellent, and thanks for all the help!!!!
>
>> Carter
>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>
>
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkmvEf4ACgkQxjU5UYZ6K6dQoACeNpfxh3+lTKklNVz3YDc8fxoN
wdAAnipgJNsG9E31PEjX0766lTjlNnPI
=w/oi
-----END PGP SIGNATURE-----
More information about the argus
mailing list