TCP port 0 or *?

Mon Aug 18 14:38:46 EDT 2014

Ok thanks! 

Removing the ‘-m none’ appends 'primary key (srcid,proto,saddr,sport,daddr,dport)’ into kbuf so it seems to be skipping over the 'if (ArgusAutoId)’ condition to add the ‘autoid’ string right after ‘primary key(‘ gets loaded into kbuf.

On August 18, 2014 at 2:22:11 PM, Carter Bullard (carter at qosient.com) wrote:
Hey John,
Oh the code base for the clients is really different between 3.0.6 and 3.0.8, although
most of the functionality is similar.

So, I can recreate the problem here, and I’m working on it.  I think you’re close in
your interpretation, but I’m thinking that the -m none is just tossing the kbuf away.
Let me look at this for a second and I’ll get back at you.

Carter

On Aug 18, 2014, at 2:19 PM, John T. Myers <myersj0 at gmail.com> wrote:

I’m running 5.5.38-0ubuntu0.14.04.1

I just re-compiled 3.0.6 with debugging. The output from 3.0.6 includes the primary key(autoid) statement right before the record blob.

It looks like there are some differences in the debug statements before table creation even begins (different code paths?)

In 3.0.8 it looks like you set ArgusAutoId in 3 different places: InsertProcess, SelectProcess, and UpdateProcess — but none of the DEBUG statements for those 3 functions are being printed in the output before the table creation begins so that leads me to maybe think ArgusAutoId is never being properly set to 1 so kbuf is never set to be filled in CreateSQLSaveTable?

Sorry this is my first time looking at the code and it looks like things are being done pretty differently between the two versions.

Here is the output from 3.0.6:
rasqlinsert[20899.4007d89e037f0000]: 13:50:17.234999 ArgusAddHostList (0x9ec3c010, localhost:1776, 1, 6) returning 1
rasqlinsert[20900.4007d89e037f0000]: 13:50:17.235428 Argusfinish_parse (0x19c6738)
rasqlinsert[20899.4007d89e037f0000]: 13:50:17.236686 ArgusFilterCompile () done 0
rasqlinsert[20899.4007d89e037f0000]: 13:50:17.271443 RaTopNewProcess(0x9ec3c010) returns 0x19d9180
rasqlinsert[20899.4007d89e037f0000]: 13:50:17.351102 generating table CREATE table tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32),srcid varchar(64), primary key (autoid), record blob) ENGINE=MyISAM
rasqlinsert[20899.4007d89e037f0000]: 13:50:17.356651 ArgusCreateSQLSaveTable (tablename) returning

Here is the output from 3.0.8:
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.886120 ArgusAddHostList (0xf1b03010, localhost:1776, 1, 6) returning 1
rasqlinsert[16859.4077c4f1a07f0000]: 13:37:41.887107 Argusfinish_parse (0x3c47800)
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.888752 ArgusFilterCompile () done 0
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.888942 RaCursesNewProcess(0xf1b03010) returns 0x3c47740
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.889045 RaCursesNewProcess(0xf1b03010) returns 0x3c47890
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.889076 RaCursesNewProcess(0xf1b03010) returns 0x3c479e0
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.938463 RaMySQLInit: connect 192.168.87.230 user 0
rasqlinsert[16858.4077c4f1a07f0000]: 13:37:41.948226 ArgusCreateSQLSaveTable: CREATE table dbname.tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32),srcid varchar(64), record blob) ENGINE=MyISAM
rasqlinsert[16858]: 13:37:41.948953 mysql_real_query error Incorrect table definition; there can be only one auto column and it must be defined as a key

On August 18, 2014 at 1:43:14 PM, Carter Bullard (carter at qosient.com) wrote:

Hey John,
That hasn’t been a requirement of mysql in the past, that the autoid be a key.
The idea is that its just an auto incrementing integer that is updated on insertion.

Are you running a newest version ??  Oldest version of mysql ???
Carter

On Aug 18, 2014, at 1:14 PM, John T. Myers <myersj0 at gmail.com> wrote:

I am only building a table with one autoid in it.

The table does not exist, on purpose, to make sure that rasqlinsert creates the table, right now the database is empty.

Here is my full command:

rasqlinsert -D 3 -S localhost:1776 ip -w mysql://user:pass@192.168.87.230/dbname/tablename -s autoid stime flgs proto saddr sport dir daddr dport pkts bytes state record -m none

Here is the resulting create statement:

rasqlinsert[8311.40b72b2a097f0000]: 13:03:12.454167 ArgusCreateSQLSaveTable: CREATE table dbname.tablename (autoid int not null auto_increment,stime double(18,6) unsigned not null,flgs varchar(32),proto varchar(16) not null,saddr varchar(64) not null,sport varchar(10) not null,dir varchar(3),daddr varchar(64) not null,dport varchar(10) not null,pkts bigint,bytes bigint,state varchar(32), record blob) ENGINE=MyISAM

It seems that the 'primary key(autoid)’ requirement is not making into the CREATE statement...

On August 18, 2014 at 12:55:13 PM, Carter Bullard (carter at qosient.com) wrote:

So grab the code again, as you should notice in the other mailing list threads,
you’re tickling a bug that is only recently snuck into the code on the server.

What version of mysql are you using ????
So what kind of table are you trying to build that has 2 auto ids in it.
Are you trying to write into an existing table, but with a different schema ???

So printing the autoid into the database does cause us to add autoid to the
schema, but should require any addition to the key, especially since you are
saying no keys.  Haven’t seen this problem ever ….

So if you drop any existing table that it may be colliding with, does it get any better ??
If you compile with the debug support turned on, you can get rasqlinsert() to
print out the CREATE statement, which may show what the problem is.

In your client distribution directory:
   % touch .debug
   % ./configure; make clean; make
   % rasqlinsert -D 3 …..

using the options that you were using before.
What does the CREATE statement look like ???

Carter

On Aug 18, 2014, at 12:38 PM, John T. Myers <myersj0 at gmail.com> wrote:

I compiled 3.0.8 on 3.13.0-24-generic on Linux Mint:
ra -S localhost:1776 - ip and not fragonly
Segmentation fault

Also, rasqlinsert behavior seems to be different. When using 3.0.6 adding ‘autoid’ to my list of columns worked fine and defining no keys was not an issue...
"-s autoid stime flgs proto saddr sport dir daddr dport pkts bytes state srcid record -d -m none” —> works fine on 3.0.6

Now that produces the following error with 3.0.8:
rasqlinsert[3766]: 12:33:32.078827 mysql_real_query error Incorrect table definition; there can be only one auto column and it must be defined as a key

When trying to define a key via ‘-m autoid’ that still produces the same error.

On August 18, 2014 at 10:35:40 AM, Carter Bullard (carter at qosient.com) wrote:
Oh my, try using argus-3.0.8 and argus-clients-3.0.8 from
the development server.  Its about to be released, only waiting
for me to find the time to finish the last pass on manages.

   http://qosient.com/argus/dev/argus-latest.tar.gz
   http://qosient.com/argus/dev/argus-clients-latest.tar.gz

Carter

On Aug 18, 2014, at 9:13 AM, John T. Myers <myersj0 at gmail.com> wrote:

I am using version 3.0.6

Here are my results:
ra -S 192.168.87.211:1776 - local not fragonly
ra[25818]: 09:10:39.890034 not fragonly filter syntax error

ra -S 192.168.87.211:1776 - ip and not frag
Still shows all packets with a ‘f’ flg set, for example, some of my output from ra:
   09:12:52.933445  e    f      tcp     192.168.87.202.*         ->     192.168.87.230.*             1       1514   INT
   09:12:52.936886  e           tcp     192.168.87.202.50773     ->     192.168.87.230.mysql        16       1430   FIN
   09:12:52.942989  e i         tcp     192.168.87.202.50774     ->     192.168.87.230.mysql        15       1243   FIN
   09:12:52.948681  e i         tcp     192.168.87.202.50775     ->     192.168.87.230.mysql        21       2576   FIN
   09:12:52.987558  e           tcp     192.168.87.211.36295    <?>     192.168.87.230.mysql        88      33192   CON
   09:12:53.375496  e           tcp     192.168.87.202.50776     ->     192.168.87.230.mysql        16       2045   FIN
   09:12:53.382321  e           tcp     192.168.87.202.50777     ->     192.168.87.230.mysql        22       4287   FIN
   09:12:53.852959  e    f      tcp     192.168.87.202.*         ->     192.168.87.230.*             1       1514   INT
On August 18, 2014 at 9:07:37 AM, Carter Bullard (carter at qosient.com) wrote:

Hey John,
I'll look at the filter.  Does the filter "local not fragonly" work ?

The remote argus source may not have the filter syntax support,
so the "local" does the filtering in the receiver.

Try the filter "ip and not frag" ????

Sorry not near any code right this second.
What version are you using ????

Carter

On Aug 18, 2014, at 9:02 AM, "John T. Myers" <myersj0 at gmail.com> wrote:

Also, in testing those filters ‘fragonly’ does not appear to work.

ra -S 192.168.87.211:1776 - fragonly
ra[25298]: 08:48:25.562475 remote Filter error
And when I run : 

ra -S 192.168.87.211:1776 - ip and frag

I get no output at all so I’m not sure if that primitive is actually filtering partial fragments out or just looking for ‘F’ flgs to filter on.

John

On August 18, 2014 at 8:40:06 AM, John T. Myers (myersj0 at gmail.com) wrote:

Carter,

I’m not seeing any ‘F’ in the flgs field, but I am seeing some ‘g’aps. This collect is just from a VM where there is very little traffic. All of the partial ‘f’ragments are being caused by a MySQL connection (between MySQL Workbench and a DB that rasqlinsert is feeding into). We’re talking < 100 flows per second so I’m wondering why it’s missing the first fragmented packet.

On August 18, 2014 at 8:12:24 AM, Carter Bullard (carter at qosient.com) wrote:

Hey John,
These are partial fragments flows, where argus didn't see the first fragment packet.  The fragments don't have either a TCP or UDP header so there aren't any port numbers to generate the 5-tuple flow record.  Argus tracks fragments, but it needs to see the first one to put them all together.   You should be able to filter them out with "not fragonly" or "not frag".  

So why are you getting so many partial fragments, ie why don't you see the first one ??  Load to high so you're droppi g lots of packets??  Load balancers and your seeing only one link of the balance??  PF_RING ???  Gigamons ??  Do you see 5-tuple flows that have fragments ( 'F' in the flgs field) ?

Carter

On Aug 17, 2014, at 9:20 PM, "John T. Myers" <myersj0 at gmail.com> wrote:

I am trying to use Argus to log non-aggregated/processed flow into MySQL, but rasqlinsert sets the port numbers to 0 (or * when just using ra) whenever the “f” partial fragmentation flag is set. Is there a way to just ignore that through a filter ... because it fills up more database rows than any other flows that are being collected.

This happens with a simple: ra -S someipaddress:port ip 

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140818/48d9d15f/attachment.html>