TCP port 0 or *?
Carter Bullard
carter at qosient.com
Mon Aug 18 16:03:15 EDT 2014
already in the code on the server, which hopefully will be official soon.
Carter
On Aug 18, 2014, at 3:40 PM, John T. Myers <myersj0 at gmail.com> wrote:
> Thanks! Will you roll this into 3.0.8 eventually?
>
> On Aug 18, 2014, at 2:44 PM, Carter Bullard <carter at qosient.com> wrote:
>
>> Oooops, pretty bad patch for performance, try this instead.
>> Carter
>>
>> diff -c rasqlinsert.c.orig rasqlinsert.c
>> *** rasqlinsert.c.orig Mon Aug 18 14:33:59 2014
>> --- rasqlinsert.c Mon Aug 18 14:34:30 2014
>> ***************
>> *** 7517,7522 ****
>> --- 7517,7525 ----
>> ArgusParser->RaPrintAlgorithm = ArgusParser->RaPrintAlgorithmList[i];
>>
>> + if ((ArgusAutoId == 0) && !strncmp(ArgusParser->RaPrintAlgorithm->field, "autoid", 6))
>> + ArgusAutoId = 1;
>> +
>> for (x = 0; x < ARGUS_MAX_PRINT_ALG; x++) {
>> if (!strcmp(ArgusParser->RaPrintAlgorithm->field, RaPrintAlgorithmTable[x].field)) {
>> if (ind++ > 0)
>> sprintf (&sbuf[strlen(sbuf)], ",”);
>>
>> On Aug 18, 2014, at 2:39 PM, Carter Bullard <carter at qosient.com> wrote:
>>
>>> Hey John,
>>> You were right on the money !!!! Try this patch, and all should be good !!!
>>> (seems I struck out today, but at least we have a fix ;O)
>>>
>>> We moved all that code around to get insert and update performance up,
>>> and missed that. Thanks !!!!!
>>>
>>> Carter
>>>
>>> diff -c rasqlinsert.c.orig rasqlinsert.c
>>> *** rasqlinsert.c.orig Mon Aug 18 14:33:59 2014
>>> --- rasqlinsert.c Mon Aug 18 14:34:30 2014
>>> ***************
>>> *** 7517,7522 ****
>>> --- 7517,7525 ----
>>> ArgusParser->RaPrintAlgorithm = ArgusParser->RaPrintAlgorithmList[i];
>>>
>>> for (x = 0; x < ARGUS_MAX_PRINT_ALG; x++) {
>>> + if ((ArgusAutoId == 0) && !strncmp(ArgusParser->RaPrintAlgorithm->field, "autoid", 6))
>>> + ArgusAutoId = 1;
>>> +
>>> if (!strcmp(ArgusParser->RaPrintAlgorithm->field, RaPrintAlgorithmTable[x].field)) {
>>> if (ind++ > 0)
>>> sprintf (&sbuf[strlen(sbuf)], ",”);
>>>
>>>
>>>
>>> On Aug 18, 2014, at 2:22 PM, Carter Bullard <carter at qosient.com> wrote:
>>>
>>>> Hey John,
>>>> Oh the code base for the clients is really different between 3.0.6 and 3.0.8, although
>>>> most of the functionality is similar.
>>>>
>>>> So, I can recreate the problem here, and I’m working on it. I think you’re close in
>>>> your interpretation, but I’m thinking that the -m none is just tossing the kbuf away.
>>>> Let me look at this for a second and I’ll get back at you.
>>>>
>>>> Carter
>>>>
>>>> On Aug 18, 2014, at 2:19 PM, John T. Myers <myersj0 at gmail.com> wrote:
>>>>
>>>>> I’m running 5.5.38-0ubuntu0.14.04.1
>>>>>
>>>>> I just re-compiled 3.0.6 with debugging. The output from 3.0.6 includes the primary key(autoid) statement right before the record blob.
>>>>>
>>>>> It looks like there are some differences in the debug statements before table creation even begins (different code paths?)
>>>>>
>>>>> In 3.0.8 it looks like you set ArgusAutoId in 3 different places: InsertProcess, SelectProcess, and UpdateProcess — but none of the DEBUG statements for those 3 functions are being printed in the output before the table creation begins so that leads me to maybe think ArgusAutoId is never being properly set to 1 so kbuf is never set to be filled in CreateSQLSaveTable?
>>>>>
>>>>> Sorry this is my first time looking at the code and it looks like things are being done pretty differently between the two versions.
>>>>>
>>>>> Here is the output from 3.0.6:
>>>>> rasqlinsert[20899.4007d89e037f0000]: 13:50:17.234999 ArgusAddHostList (0x9ec3c010, localhost:1776, 1, 6) returning 1
>>>>> rasqlinsert[20900.4007d89e037f0000]: 13:50:17.235428 Argusfinish_parse (0x19c6738)
>>>>> rasqlinsert[20899.4007d89e037f0000]: 13:50:17.236686 ArgusFilterCompile () done 0
>>>>> rasqlinsert[20899.4007d89e037f0000]: 13:50:17.271443 RaTopNewProcess(0x9ec3c010) returns 0x19d9180
>>>>> rasqlinsert[20899.4007d89e037f0000]: 13:50:17.351102 generating table CREATE table tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32),srcid varchar(64), primary key (autoid), record blob) ENGINE=MyISAM
>>>>> rasqlinsert[20899.4007d89e037f0000]: 13:50:17.356651 ArgusCreateSQLSaveTable (tablename) returning
>>>>>
>>>>> Here is the output from 3.0.8:
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.886120 ArgusAddHostList (0xf1b03010, localhost:1776, 1, 6) returning 1
>>>>> rasqlinsert[16859.4077c4f1a07f0000]: 13:37:41.887107 Argusfinish_parse (0x3c47800)
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.888752 ArgusFilterCompile () done 0
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.888942 RaCursesNewProcess(0xf1b03010) returns 0x3c47740
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.889045 RaCursesNewProcess(0xf1b03010) returns 0x3c47890
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.889076 RaCursesNewProcess(0xf1b03010) returns 0x3c479e0
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.938463 RaMySQLInit: connect 192.168.87.230 user 0
>>>>> rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.948226 ArgusCreateSQLSaveTable: CREATE table dbname.tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32),srcid varchar(64), record blob) ENGINE=MyISAM
>>>>> rasqlinsert[16858]: 13:37:41.948953 mysql_real_query error Incorrect table definition; there can be only one auto column and it must be defined as a key
>>>>>
>>>>>
>>>>> On August 18, 2014 at 1:43:14 PM, Carter Bullard (carter at qosient.com) wrote:
>>>>>
>>>>>> Hey John,
>>>>>> That hasn’t been a requirement of mysql in the past, that the autoid be a key.
>>>>>> The idea is that its just an auto incrementing integer that is updated on insertion.
>>>>>>
>>>>>> Are you running a newest version ?? Oldest version of mysql ???
>>>>>> Carter
>>>>>>
>>>>>> On Aug 18, 2014, at 1:14 PM, John T. Myers <myersj0 at gmail.com> wrote:
>>>>>>
>>>>>> I am only building a table with one autoid in it.
>>>>>>
>>>>>> The table does not exist, on purpose, to make sure that rasqlinsert creates the table, right now the database is empty.
>>>>>>
>>>>>> Here is my full command:
>>>>>>
>>>>>> rasqlinsert -D 3 -S localhost:1776 ip -w mysql://user:pass@192.168.87.230/dbname/tablename -s autoid stime flgs proto saddr sport dir daddr dport pkts bytes state record -m none
>>>>>>
>>>>>> Here is the resulting create statement:
>>>>>>
>>>>>> rasqlinsert[8311.40b72b2a097f0000]: 13:03:12.454167 ArgusCreateSQLSaveTable: CREATE table dbname.tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32), record blob) ENGINE=MyISAM
>>>>>>
>>>>>> It seems that the 'primary key(autoid)’ requirement is not making into the CREATE statement...
>>>>>>
>>>>>>
>>>>>>
>>>>>> On August 18, 2014 at 12:55:13 PM, Carter Bullard (carter at qosient.com) wrote:
>>>>>>
>>>>>> So grab the code again, as you should notice in the other mailing list threads,you’re tickling a bug that is only recently snuck into the code on the server.
>>>>>> What version of mysql are you using ????So what kind of table are you trying to build that has 2 auto ids in it.Are you trying to write into an existing table, but with a different schema ???
>>>>>> So printing the autoid into the database does cause us to add autoid to theschema, but should require any addition to the key, especially since you aresaying no keys. Haven’t seen this problem ever ….
>>>>>> So if you drop any existing table that it may be colliding with, does it get any better ??If you compile with the debug support turned on, you can get rasqlinsert() toprint out the CREATE statement, which may show what the problem is.
>>>>>> In your client distribution directory: % touch .debug % ./configure; make clean; make % rasqlinsert -D 3 …..
>>>>>> using the options that you were using before.What does the CREATE statement look like ???
>>>>>> Carter
>>>>>>
>>>>>> On Aug 18, 2014, at 12:38 PM, John T. Myers <myersj0 at gmail.com> wrote:
>>>>>> I compiled 3.0.8 on 3.13.0-24-generic on Linux Mint:ra -S localhost:1776 - ip and not fragonlySegmentation fault
>>>>>> Also, rasqlinsert behavior seems to be different. When using 3.0.6 adding ‘autoid’ to my list of columns worked fine and defining no keys was not an issue..."-s autoid stime flgs proto saddr sport dir daddr dport pkts bytes state srcid record -d -m none” —> works fine on 3.0.6
>>>>>> Now that produces the following error with 3.0.8:rasqlinsert[3766]: 12:33:32.078827 mysql_real_query error Incorrect table definition; there can be only one auto column and it must be defined as a key
>>>>>> When trying to define a key via ‘-m autoid’ that still produces the same error.
>>>>>>
>>>>>> On August 18, 2014 at 10:35:40 AM, Carter Bullard (carter at qosient.com) wrote:Oh my, try using argus-3.0.8 and argus-clients-3.0.8 fromthe development server. Its about to be released, only waitingfor me to find the time to finish the last pass on manages.
>>>>>> http://qosient.com/argus/dev/argus-latest.tar.gz
>>>>>> http://qosient.com/argus/dev/argus-clients-latest.tar.gz
>>>>>>
>>>>>>
>>>>>> Carter
>>>>>> On Aug 18, 2014, at 9:13 AM, John T. Myers <myersj0 at gmail.com> wrote:
>>>>>> I am using version 3.0.6
>>>>>> Here are my results:ra -S 192.168.87.211:1776 - local not fragonlyra[25818]: 09:10:39.890034 not fragonly filter syntax error
>>>>>> ra -S 192.168.87.211:1776 - ip and not fragStill shows all packets with a ‘f’ flg set, for example, some of my output from ra: 09:12:52.933445 e f tcp 192.168.87.202.* -> 192.168.87.230.* 1 1514 INT 09:12:52.936886 e tcp 192.168.87.202.50773 -> 192.168.87.230.mysql 16 1430 FIN 09:12:52.942989 e i tcp 192.168.87.202.50774 -> 192.168.87.230.mysql 15 1243 FIN 09:12:52.948681 e i tcp 192.168.87.202.50775 -> 192.168.87.230.mysql 21 2576 FIN 09:12:52.987558 e tcp 192.168.87.211.36295 <?> 192.168.87.230.mysql 88 33192 CON 09:12:53.375496 e tcp 192.168.87.202.50776 -> 192.168.87.230.mysql 16 2045 FIN 09:12:53.382321 e tcp 192.168.87.202.50777 -> 192.168.87.230.mysql 22 4287 FIN 09:12:53.852959 e f tcp 192.168.87.202.* -> 192.168.87.230.* 1 1514 INTOn August 18, 2014 at 9:07:37 AM, Carter Bullard (carter at qosient.com) wrote:Hey John,I'll look at the filter. Does the filter "local not fragonly" work ?
>>>>>> The remote argus source may not have the filter syntax support,so the "local" does the filtering in the receiver.
>>>>>> Try the filter "ip and not frag" ????
>>>>>> Sorry not near any code right this second.What version are you using ????
>>>>>> Carter
>>>>>>
>>>>>> On Aug 18, 2014, at 9:02 AM, "John T. Myers" <myersj0 at gmail.com> wrote:
>>>>>>
>>>>>> Also, in testing those filters ‘fragonly’ does not appear to work.
>>>>>> ra -S 192.168.87.211:1776 - fragonlyra[25298]: 08:48:25.562475 remote Filter errorAnd when I run : ra -S 192.168.87.211:1776 - ip and fragI get no output at all so I’m not sure if that primitive is actually filtering partial fragments out or just looking for ‘F’ flgs to filter on.
>>>>>> John
>>>>>>
>>>>>> On August 18, 2014 at 8:40:06 AM, John T. Myers (myersj0 at gmail.com) wrote:Carter,
>>>>>> I’m not seeing any ‘F’ in the flgs field, but I am seeing some ‘g’aps. This collect is just from a VM where there is very little traffic. All of the partial ‘f’ragments are being caused by a MySQL connection (between MySQL Workbench and a DB that rasqlinsert is feeding into). We’re talking < 100 flows per second so I’m wondering why it’s missing the first fragmented packet.
>>>>>>
>>>>>> On August 18, 2014 at 8:12:24 AM, Carter Bullard (carter at qosient.com) wrote:Hey John,These are partial fragments flows, where argus didn't see the first fragment packet. The fragments don't have either a TCP or UDP header so there aren't any port numbers to generate the 5-tuple flow record. Argus tracks fragments, but it needs to see the first one to put them all together. You should be able to filter them out with "not fragonly" or "not frag".
>>>>>> So why are you getting so many partial fragments, ie why don't you see the first one ?? Load to high so you're droppi g lots of packets?? Load balancers and your seeing only one link of the balance?? PF_RING ??? Gigamons ?? Do you see 5-tuple flows that have fragments ( 'F' in the flgs field) ?
>>>>>> Carter
>>>>>>
>>>>>> On Aug 17, 2014, at 9:20 PM, "John T. Myers" <myersj0 at gmail.com> wrote:
>>>>>>
>>>>>> I am trying to use Argus to log non-aggregated/processed flow into MySQL, but rasqlinsert sets the port numbers to 0 (or * when just using ra) whenever the “f” partial fragmentation flag is set. Is there a way to just ignore that through a filter ... because it fills up more database rows than any other flows that are being collected.
>>>>>> This happens with a simple: ra -S someipaddress:port ip
>>>>>> Thanks!
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140818/ded7bfba/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140818/ded7bfba/attachment.bin>
More information about the argus
mailing list