mysql database changes - new patch
Carter Bullard
carter at qosient.com
Mon Jun 25 10:10:52 EDT 2012
Time is currently defined as a double, so that it can get the fractional part,
so you may not need to change that representation, unless double is not
a good performer?
Please keep sending your preferences for data formats to the list so I can make
appropriate changes. The Stanford guys started this effort by asking for addresses
to be in decimal, and with your list, we should be able to do a good job on
preliminary database work. They are using some very high performance
commercial databases for their work, and seem rather happy, but that doesn't
cover Stanford's 10G monitoring efforts.
I will put in the database schema creation support for these formats next, but
that will be in argus-clients-3.0.7.1, which i should put up later this week.
You will find that the db will not be able to keep up with stuffing primitive data
into the database, as the network and argus can generate more data than the
db can accept. But maintaining derived data, like an ethernet/ip address table
with rasqlinsert() is no problem, or an ethernet matrix table is a no brainer.
I'd be interested in any results that you discover !!!!!
Carter
On Jun 25, 2012, at 10:00 AM, CS Lee wrote:
> hi Carter,
>
> After applying latest patch from you, here's the result -
>
> mysql> SELECT stime, saddr, sport, daddr, dport, proto, state FROM tbl_test;
> +-------------------+------------+-------+------------+-------+-------+-------+
> | stime | saddr | sport | daddr | dport | proto | state |
> +-------------------+------------+-------+------------+-------+-------+-------+
> | 1340623505.000000 | 3232235779 | 8 | 3512053643 | 5923 | icmp | ECO |
> +-------------------+------------+-------+------------+-------+-------+-------+
>
> I will run more testing on other data later. My other plan is to change stime to be integer as well so that it will be very efficient for partition strategy. String data type can't be used for mysql partition. Currently my tweaking and fine tuning is gearing toward high speed mysql db for argus and see if my strategy works better than default schema.
>
> Thank you!
>
> On Mon, Jun 25, 2012 at 9:48 PM, CS Lee <geek00l at gmail.com> wrote:
> hi Carter,
>
> I tested the previous patch and it works, so I'm testing this one now.
>
>
> On Mon, Jun 25, 2012 at 9:27 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey CS Lee,
> My bad, I needed to change a few lines, as the conditional shouldn't define the scope !!!!!
> Here is a new patch that should work a bit better:
>
> ==== //depot/argus-3.0.6/clients/common/argus_util.c#3 - /Volumes/Users/carter/argus/release/argus-3.0.6/clients/common/argus_util.c ====
> 7200c7200
> < char upbuf[32], *upstr = upbuf;
> ---
> > char upbuf[32], *upstr = upbuf, *format = NULL;
> 7202,7205c7202,7203
> < switch (proto) {
> < case IPPROTO_ICMP:
> < sprintf(upstr, "0x%4.4x", port);
> < break;
> ---
> > if (parser->RaPrintAlgorithmList[parser->RaPrintIndex] != NULL)
> > format = parser->RaPrintAlgorithmList[parser->RaPrintIndex]->format;
> 7207,7209c7205,7209
> < default:
> < sprintf(upstr, "%d", port);
> < break;
> ---
> > if ((format == NULL) || (strlen(format) == 0)) {
> > switch (proto) {
> > case IPPROTO_ICMP: format = "0x%4.4x"; break;
> > default: format = "%d"; break;
> > }
> 7211a7212,7213
> > snprintf (upstr, 128, format, port);
> >
>
> Carter
>
> On Jun 25, 2012, at 9:16 AM, Carter Bullard wrote:
>
>> Hey CS Lee,
>> Hmmmmm, well that is not how I thought it would go. This patch will enable the print format
>> extension for all port values. So, " sport::%d dport::%d" should work for icmp as well.
>>
>> I'm adding this to the main thread of code, if it works for you, so it will be in any new
>> clients tarfiles that we release. And it will be in argus-clients-3.0.7.1.
>>
>> Carter
>>
>> ==== //depot/argus-3.0.6/clients/common/argus_util.c#3 - /Volumes/Users/carter/argus/release/argus-3.0.6/clients/common/argus_util.c ====
>> 7202,7205c7202,7203
>> < switch (proto) {
>> < case IPPROTO_ICMP:
>> < sprintf(upstr, "0x%4.4x", port);
>> < break;
>> ---
>> > if (parser->RaPrintAlgorithmList[parser->RaPrintIndex] != NULL) {
>> > char *format = parser->RaPrintAlgorithmList[parser->RaPrintIndex]->format;
>> 7207,7209c7205,7211
>> < default:
>> < sprintf(upstr, "%d", port);
>> < break;
>> ---
>> > if ((format == NULL) || (strlen(format) == 0)) {
>> > switch (proto) {
>> > case IPPROTO_ICMP: format = "0x%4.4x"; break;
>> > default: format = "%d"; break;
>> > }
>> > }
>> > snprintf (upstr, 128, format, port);
>>
>> Carter
>>
>> On Jun 25, 2012, at 12:27 AM, CS Lee wrote:
>>
>>> hi Carter,
>>>
>>> While the changes work correctly for tcp and udp, it seems the hex value doesn't get to convert to decimal for icmp in sport and dport.
>>>
>>> Default
>>> mysql> SELECT stime, saddr, sport, daddr, dport, spkts, dpkts, proto, state FROM tbl_argus;
>>> +-------------------+-------------+--------+----------------+--------+-------+-------+-------+-------+
>>> | stime | saddr | sport | daddr | dport | spkts | dpkts | proto | state |
>>> +-------------------+-------------+--------+----------------+--------+-------+-------+-------+-------+
>>> | 1340623505.000000 | 192.168.1.3 | 0x0008 | 209.85.175.139 | 0x1723 | 8 | 1 | icmp | ECO |
>>> +-------------------+-------------+--------+----------------+--------+-------+-------+-------+-------+
>>>
>>> After drop the table, and apply the patch
>>>
>>> mysql> SELECT stime, saddr, sport, daddr, dport, spkts, dpkts, proto, state FROM tbl_argus;
>>> +-------------------+------------+-------+------------+-------+-------+-------+-------+-------+
>>> | stime | saddr | sport | daddr | dport | spkts | dpkts | proto | state |
>>> +-------------------+------------+-------+------------+-------+-------+-------+-------+-------+
>>> | 1340623505.000000 | 3232235779 | 0 | 3512053643 | 0 | 8 | 1 | icmp | ECO |
>>> +-------------------+------------+-------+------------+-------+-------+-------+-------+-------+
>>> 1 row in set (0.00 sec)
>>>
>>> If hex conversion is right, then 0x0008 will be 8 and 0x1723 will be 5923 in decimal. Here's my testing table -
>>>
>>> mysql> desc tbl_argus;
>>> +-----------+-----------------------+------+-----+---------+----------------+
>>> | Field | Type | Null | Key | Default | Extra |
>>> +-----------+-----------------------+------+-----+---------+----------------+
>>> | autoid | int(11) | NO | PRI | NULL | auto_increment |
>>> | stime | double(18,6) unsigned | NO | | NULL | |
>>> | ltime | double(18,6) unsigned | NO | | NULL | |
>>> | dur | double(18,6) | NO | | NULL | |
>>> | srcid | varchar(64) | YES | | NULL | |
>>> | flgs | varchar(32) | YES | | NULL | |
>>> | proto | varchar(16) | NO | | NULL | |
>>> | saddr | varchar(64) | NO | | NULL | |
>>> | sport | smallint(5) unsigned | YES | | NULL | |
>>> | dir | varchar(3) | YES | | NULL | |
>>> | daddr | varchar(64) | NO | | NULL | |
>>> | dport | smallint(5) unsigned | YES | | NULL | |
>>> | pkts | bigint(20) | YES | | NULL | |
>>> | bytes | bigint(20) | YES | | NULL | |
>>> | appbytes | bigint(20) | YES | | NULL | |
>>> | state | varchar(32) | YES | | NULL | |
>>> | spkts | bigint(20) | YES | | NULL | |
>>> | dpkts | bigint(20) | YES | | NULL | |
>>> | sbytes | bigint(20) | YES | | NULL | |
>>> | dbytes | bigint(20) | YES | | NULL | |
>>> | sappbytes | bigint(20) | YES | | NULL | |
>>> | dappbytes | bigint(20) | YES | | NULL | |
>>> | smaxsz | smallint(5) unsigned | YES | | NULL | |
>>> | dmaxsz | smallint(5) unsigned | YES | | NULL | |
>>> | sminsz | smallint(5) unsigned | YES | | NULL | |
>>> | dminsz | smallint(5) unsigned | YES | | NULL | |
>>> | smac | varchar(24) | YES | | NULL | |
>>> | dmac | varchar(24) | YES | | NULL | |
>>> | sco | varchar(2) | YES | | NULL | |
>>> | dco | varchar(2) | YES | | NULL | |
>>> | sas | int(10) unsigned | YES | | NULL | |
>>> | das | int(10) unsigned | YES | | NULL | |
>>> | record | blob | YES | | NULL | |
>>> +-----------+-----------------------+------+-----+---------+----------------+
>>> 33 rows in set (0.01 sec)
>>>
>>> --
>>> Best Regards,
>>>
>>> CS Lee<geek00L[at]gmail.com>
>>>
>>> http://geek00l.blogspot.com
>>> http://defcraft.net
>>
>
>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
> http://defcraft.net
>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
> http://defcraft.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120625/b58edb61/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120625/b58edb61/attachment.bin>
More information about the argus
mailing list