argus-clients-3.0.2.tar.gz with mysql support
Carter Bullard
carter at qosient.com
Fri Mar 6 09:43:07 EST 2009
Hey Pablo,
Sorry for the delayed response.
Try this patch, to see if you get better behavior.
==== //depot/argus/clients/clients/racluster.c#39 - /home/carter/argus/
clients/clients/racluster.c ====
150c150
< if (!(ArgusParser->RaParseCompleting)) {
---
> if (!(ArgusParser->RaParseCompleting++)) {
234d233
< }
269a269
> }
Carter
On Mar 4, 2009, at 6:42 PM, Pablo J. Rebollo-Sosa wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> isCarter,
>
> I ran the the tests with racluster and the outputs are the same.
>
> root at argus:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m
> saddr -w
> - - | racount
> racount records total_pkts src_pkts dst_pkts
> total_bytes src_bytes dst_bytes
> sum 2334 336945 193790 143155
> 276763462 239874647 36888815
>
> root at argus:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m
> saddr -w
> - - | ra -w - | racount
> racount records total_pkts src_pkts dst_pkts
> total_bytes src_bytes dst_bytes
> sum 2334 336945 193790 143155
> 276763462 239874647 36888815
>
> I still thinking that there is a problem. For example I tried the
> following.
>
> root at nsm:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr
> -w
> - - | rasort -m bytes -w - | ra -N 5
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 17:00:02.870130 e ip 74.125.0.163 <->
> 0.0.0.0 25377 27162034 CON
> 17:00:43.147390 e ip 68.142.122.70 <->
> 0.0.0.0 23169 21108846 CON
> 17:00:02.635304 e ip 63.251.219.114 <->
> 0.0.0.0 21325 19133350 CON
> 17:00:02.075566 e ip 66.90.64.10 <->
> 0.0.0.0 19430 18250118 CON
> 17:01:13.549650 e ip 74.125.0.214 <->
> 0.0.0.0 15158 14123330 CON
>
> - From the previous output the maximum total bytes is 27162034. But
> if the
> "- net 136.145.34.0/24" filter is applied I get a bigger value for
> total
> bytes (686322024).
>
> root at nsm:~# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr
> -w
> - - - net 136.145.34.0/24 | rasort -m bytes -w - | ra -N 5
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 17:00:00.584726 e ip 136.145.34.11 <->
> 0.0.0.0 669192 686322024 CON
> 17:00:00.584726 e ip 96.7.19.16 <->
> 0.0.0.0 669185 686321500 CON
> 17:00:00.099711 eU F ip 136.145.34.80 <->
> 0.0.0.0 71461 50620443 CON
> 17:00:43.147390 e ip 68.142.122.70 <->
> 0.0.0.0 23140 21093742 CON
> 17:00:01.903740 e ip 136.145.34.81 <->
> 0.0.0.0 23002 21043963 CON
>
> Any clues?
>
> Best regards,
>
> Pablo J. Rebollo
>
> carter at qosient.com wrote:
>> Use racount instead of wc, but I suspect a problem.
>> So instead of " | wc -l"
>> Try " -w - | racount"
>>
>> Racluster maybe leaving a few records in its output when it closes
>> terminates. I'll take a look today!!
>>
>> Carter
>>
>> Sent from my Verizon Wireless BlackBerry
>>
>> -----Original Message-----
>> From: "Pablo J. Rebollo-Sosa" <Pablo.Rebollo at ece.uprm.edu>
>>
>> Date: Tue, 03 Mar 2009 17:52:56
>> To: Carter Bullard<carter at qosient.com>
>> Cc: Argus<argus-info at lists.andrew.cmu.edu>
>> Subject: Re: [ARGUS] argus-clients-3.0.2.tar.gz with mysql support
>>
>>
>> Dear Carter,
>>
>> I'm testing the new clients and noticed odd results with ra. When
>> using
>> racluster with a specific file I get certain amount of lines
>>
>> server# racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr |
>> wc -l
>> 5555
>>
>> The problem is when using racluster with -w option with ra. When
>> running the command I get fewer amount of lines.
>>
>> racluster -r argus.2009.03.03.17.00.00 -M rmon -m saddr -w - | ra -
>> r -
>> | wc -l
>> 2334
>>
>> Any suggestions?
>>
>> Best regards,
>>
>> Pablo J. Rebollo
>>
>> Carter Bullard wrote:
>>> Gentle people,
>>> First pass at the new argus-clients distribution is on the dev
>>> server.
>>> ftp:/qosient.com/dev/argus-3.0/argus-clients-3.0.2.tar.gz
>>
>>> First pass because there will be modifications before its released,
>>> as the user data analysis programs still need a little tweak.
>>
>>> This version addresses many problems, particularly those
>>> relating to backward compatibility to argus-2.x streams.
>>> I have not had a chance to directly test the changes on
>>> some of the bugs on the list but I suspect that this version
>>> should fix those backward compatibility bugs.
>>
>>> If you try the code, and it doesn't have your issue fixed,
>>> please, please, please, send email, so that I can get those
>>> issues dealt with.
>>
>>> I am pleased to say that the database programs, rasqlinsert()
>>> and rasql() are mostly ready to go. I don't have a manpage yet,
>>> so hopefully the "-h" option will give you guidance.
>>
>>> I will be sending out sometime this week detail on the use of
>>> rasqlinsert(), the format of the database url that is needed to
>>> access database data, and the concepts of rasql() and why
>>> its needed.
>>
>>> If you want to give rasqlinsert a run, like loading tables from
>>> files, try these types of commands:
>>
>>> rasqlinsert -r file -w mysql://user@host/db/table -m none
>>
>>> This will load the table 'db.table' with the records, and the
>>> fields will be those that you would expect to be printed if
>>> you had run ra against the file. To modify the schema, just
>>> use the "-s field" command.
>>
>>> The "-m none" removes any keys that rasqlinsert() may have
>>> wanted to use based on your .rarc file, so MySQL won't
>>> complain about DUPLICATE inserts into the table.
>>
>>> If you then run these programs:
>>
>>> rasql -r mysql://user@host/db/table
>>
>>> or
>>
>>> rasqlinsert -r mysql://user@host/db/table
>>
>>> rasqlinsert() will look like ratop(), but its data will come from
>>> the MySQL tables.
>>
>>> rasqlinsert pokes the actual binary record into the database,
>>> along with ascii representations of the attributes. This is
>>> so programs like rasql() can get argus records, rather
>>> than ascii text out of the database. If you want to get rid
>>> of the binary BLOBs, use "-s -record". rasql(), when reading
>>> this type of table, will just return, without any data.
>>
>>> A set of programs I use a lot are:
>>
>>> rabins -S localhost -M time 30s -B 5s -w - | \
>>> rasqlinsert -r - -w mysql://user@host/ratop/flowTable -m none
>>
>>> This reads data from a live stream, holds it for 30s, aggregating
>>> common records together, and then pokes it into the database
>>> table. This table will grow forever with argus records, but you can
>>> see how something very simple like this can be the base of
>>> a large flow system.
>>
>>> Hope all is most excellent, and thanks for all the help!!!!
>>
>>> Carter
>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York 10022
>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>
>>
>>
>>
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkmvEf4ACgkQxjU5UYZ6K6dQoACeNpfxh3+lTKklNVz3YDc8fxoN
> wdAAnipgJNsG9E31PEjX0766lTjlNnPI
> =w/oi
> -----END PGP SIGNATURE-----
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
More information about the argus
mailing list