RA_CIDR_ADDRESS_FORMAT="yes" and rasqlinsert -S [radium source] may be a problem
David Edelman
dedelman at iname.com
Fri May 10 13:47:37 EDT 2013
Carter that does accurately summarize the situation.
--Dave
Dave Edelman
On May 10, 2013, at 10:55, Carter Bullard <carter at qosient.com> wrote:
> Hey Dave,
> Wow, now that is a long email !!!!
>
> OK, the " RA_CIDR_ADDRESS_FORMAT=yes " variable should only come
> into play when there is aggregation.
>
> So in your data flow machine that you have generated:
>
> argus -> radium --> rastream() -----> archive
> |
> +------> rasqlinsert() ---> DB
>
> only rasqlinsert() is an aggregator ( -m srcid matrix proto). Nothing in the
> data coming from argus() or radium() will result in a CIDR IP address.
> A CIDR address happens when the ARGUS_FLOW_DSR contains values
> in the IP address mask fields, which will be zero, coming from argus(),
> radium() and rastream() in your setup.
>
> While rasqlinsert() aggregation has the potential to generate CIDR
> addresses, given the aggregation key that rasqlinsert() is using
> (-m srcid matrix proto ), there shouldn't be any CIDR addresses any
> where in your information system.
>
> Now, based on what you're showing, everything seems to be OK. The data
> in the archive is good, and the mysql database values for the src and dst
> addresses all look good, in that there aren't any CIDR addresses showing
> up. Because rasqlinsert() is like any other ra* program, if there was a real
> CIDR address in the record when it is inserted into the DB,
>
> What seems to be errant, are the argus records that are in the DB.
>
> rasqlinsert() populates the database with attributes using strings that
> it generates, using the same routines that ra* programs use to write to the
> terminal ( ArgusPrintRecord() ). But, behind the scene, rasqlinsert() also
> writes into the database, the binary argus record that it has aggregated.
> This is the data that rasql() will fetch from the database and use to
> print its output.
>
> So, when mysql() reads the address data from the table, no CIDR address
> strings. But when rasql() reads the binary argus data from the table,
> we get CIDRs. I suspect that we're getting some data corruption, such
> that the " smask " and " dmask " fields end up with values in them, which
> makes rasql() print the addresses as CIDR values.
>
> If this is the case, then when rasql() reads the data from the table with the
> " RA_CIDR_ADDRESS_FORMAT= no ", then the data should be good.
>
> Does this describe what your seeing ? If so, there are a few things we
> can do to fix…...
>
> Carter
>
>
>
> On May 9, 2013, at 11:14 PM, "Dave Edelman" <dedelman at iname.com> wrote:
>
>> I have a single instance of argus that has been running for years and
>> creating hourly files of the flow data. On a daily basis, I copy a day's
>> worth of the flow files to a second system where I run rqasqlinsert in
>> various flavors to create several different tables.
>>
>> I finally decided to use radium and rastream to do this the right way (but I
>> didn't stop the local file creation, just to be safe.)
>>
>> The details follow but it looks like there is a toxic interaction between
>> RA_CIDR_ADDRESS_FORMAT="yes" in ~/.rarc and rasqlinsert using -S from a
>> radium instance in client version 3.0.7.8 and possibly earlier.
>>
>>
>> --------- Danger beyond this point are the gory details
>> --------------------------------
>>
>> The original method worked very well:
>> A typical MySQL table queried for all 10.1.1.10 activity gave reasonable
>> results (I learned not to select the record blob :-) )
>>
>> mysql> select ltime, dur, saddr, daddr, proto, bytes from matrix_2013_04_09
>> where saddr = '10.1.1.10' or daddr = '10.1.1.10';
>> +-------------------+--------------+------------+-----------------+-------+-
>> ------------+
>> | ltime | dur | saddr | daddr | proto |
>> bytes |
>> +-------------------+--------------+------------+-----------------+-------+-
>> ------------+
>> | 1365551889.364000 | 86287.953000 | 10.1.1.10 | 10.1.1.60 | tcp |
>> 843282 |
>> | 1365551948.803000 | 86347.195000 | 10.1.1.10 | 10.1.1.46 | tcp |
>> 783536 |
>> | 1365551993.490000 | 86391.336000 | 10.1.1.10 | 10.1.1.45 | tcp |
>> 16050742461 |
>> | 1365551986.978000 | 86381.195000 | 10.1.1.10 | 10.1.1.50 | tcp |
>> 3962458654 |
>> | 1365551957.703000 | 86340.031000 | 10.1.1.10 | 255.255.255.255 | udp |
>> 1346514 |
>> | 1365551992.462000 | 86374.555000 | 10.1.1.10 | 10.1.1.50 | udp |
>> 1667967 |
>> | 1365551992.462000 | 86369.734000 | 10.1.1.10 | 10.1.1.50 | icmp |
>> 1426656 |
>> | 1365551938.461000 | 86280.000000 | 10.1.1.45 | 10.1.1.10 | arp |
>> 157312 |
>> | 1365551964.909000 | 86302.922000 | 10.1.1.50 | 10.1.1.10 | arp |
>> 152960 |
>> | 1365551953.810000 | 86274.445000 | 10.1.1.46 | 10.1.1.10 | arp |
>> 107904 |
>> | 1365551889.353000 | 86206.047000 | 10.1.1.60 | 10.1.1.10 | arp |
>> 108544 |
>> | 1365551868.423000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | udp |
>> 30672 |
>> | 1365551873.421000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | arp |
>> 36352 |
>> | 1365551639.982000 | 85736.523000 | 10.1.1.10 | 10.1.1.127 | udp |
>> 61560 |
>> | 1365551920.751000 | 85978.414000 | 10.1.1.10 | 10.1.1.50 | arp |
>> 49408 |
>> | 1365551560.220000 | 77317.164000 | 10.1.1.10 | 10.1.1.101 | udp |
>> 115032 |
>> | 1365551560.220000 | 77310.164000 | 10.1.1.10 | 10.1.1.101 | arp |
>> 29696 |
>> | 1365550944.567000 | 76398.391000 | 10.1.1.10 | 224.0.0.251 | udp |
>> 3703 |
>> | 1365529023.282000 | 54396.770000 | 10.1.1.10 | 10.1.1.101 | tcp |
>> 1032139 |
>> | 1365475483.795000 | 786.242000 | 10.1.1.101 | 10.1.1.10 | arp |
>> 896 |
>> | 1365501012.119000 | 24844.955000 | 10.1.1.10 | 10.1.1.45 | udp |
>> 2130 |
>> | 1365501012.119000 | 24844.953000 | 10.1.1.10 | 10.1.1.45 | icmp |
>> 2718 |
>> | 1365501007.827000 | 24840.254000 | 10.1.1.10 | 10.1.1.126 | arp |
>> 384 |
>> | 1365501012.354000 | 24844.787000 | 10.1.1.10 | 167.206.245.130 | udp |
>> 5766 |
>> | 1365501008.507000 | 24840.812000 | 10.1.1.10 | 113.37.91.61 | icmp |
>> 612 |
>> | 1365501012.895000 | 24844.477000 | 10.1.1.10 | 113.37.91.61 | tcp |
>> 37293 |
>> | 1365502197.518000 | 24840.180000 | 10.1.1.126 | 10.1.1.10 | arp |
>> 384 |
>> +-------------------+--------------+------------+-----------------+-------+-
>> ------------+
>> 27 rows in set (0.00 sec)
>>
>> rasql gave happy results:
>>
>> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_04_09 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1365551889.364 86287.953 69.113.13.218 10.1.1.60
>> 10.1.1.10 tcp 843282
>> 1365551948.803 86347.195 69.113.13.218 10.1.1.46
>> 10.1.1.10 tcp 783536
>> 1365551993.490 86391.336 69.113.13.218 10.1.1.45
>> 10.1.1.10 tcp 160507424*
>> 1365551986.978 86381.195 69.113.13.218 10.1.1.50
>> 10.1.1.10 tcp 3962458654
>> 1365551957.703 86340.031 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1346514
>> 1365551992.462 86374.555 69.113.13.218 10.1.1.10
>> 10.1.1.50 udp 1667967
>> 1365551992.462 86369.734 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1426656
>> 1365551938.461 86280.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 157312
>> 1365551964.909 86302.922 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 152960
>> 1365551953.810 86274.445 69.113.13.218 10.1.1.46
>> 10.1.1.10 arp 107904
>> 1365551889.353 86206.047 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 108544
>> 1365551868.423 86055.141 69.113.13.218 10.1.1.10
>> 10.1.1.12 udp 30672
>> 1365551873.421 86055.141 69.113.13.218 10.1.1.10
>> 10.1.1.12 arp 36352
>> 1365551639.982 85736.523 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1365551920.751 85978.414 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 49408
>> 1365551560.220 77317.164 69.113.13.218 10.1.1.10
>> 10.1.1.101 udp 115032
>> 1365551560.220 77310.164 69.113.13.218 10.1.1.10
>> 10.1.1.101 arp 29696
>> 1365550944.567 76398.391 69.113.13.218 10.1.1.10
>> 224.0.0.251 udp 3703
>> 1365529023.282 54396.770 69.113.13.218 10.1.1.101
>> 10.1.1.10 tcp 1032139
>> 1365475483.795 786.242 69.113.13.218 10.1.1.101
>> 10.1.1.10 arp 896
>> 1365501012.119 24844.955 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1365501012.119 24844.953 69.113.13.218 10.1.1.45
>> 10.1.1.10 icmp 2718
>> 1365501007.827 24840.254 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1365501012.354 24844.787 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1365501008.507 24840.812 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1365501012.895 24844.477 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 37293
>> 1365502197.518 24840.180 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>>
>> And a confirmation from the original flow files checked out well
>>
>> racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
>> daddr proto bytes - host 10.1.1.10
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1365551992.462 86374.555 69.113.13.218 10.1.1.10
>> 10.1.1.50 udp 1667967
>> 1365551560.220 77317.164 69.113.13.218 10.1.1.10
>> 10.1.1.101 udp 115032
>> 1365551957.703 86340.031 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1346514
>> 1365501012.354 24844.787 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1365551868.423 86055.141 69.113.13.218 10.1.1.10
>> 10.1.1.12 udp 30672
>> 1365551639.982 85736.523 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1365501012.119 24844.955 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1365550944.567 76398.391 69.113.13.218 10.1.1.10
>> 224.0.0.251 udp 3703
>> 1365551986.978 86389.391 69.113.13.218 10.1.1.10
>> 10.1.1.50 tcp 3962486779
>> 1365529023.282 54396.770 69.113.13.218 10.1.1.10
>> 10.1.1.101 tcp 1032139
>> 1365501012.895 24844.477 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 37293
>> 1365551993.490 86391.336 69.113.13.218 10.1.1.10
>> 10.1.1.45 tcp 160507424*
>> 1365551948.803 86347.195 69.113.13.218 10.1.1.10
>> 10.1.1.46 tcp 783536
>> 1365551889.364 86287.953 69.113.13.218 10.1.1.10
>> 10.1.1.60 tcp 843282
>> 1365501012.119 24844.953 69.113.13.218 10.1.1.10
>> 10.1.1.45 icmp 2718
>> 1365551992.462 86369.734 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1426656
>> 1365501008.507 24840.812 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1365551873.421 86055.141 69.113.13.218 10.1.1.10
>> 10.1.1.12 arp 36352
>> 1365551920.751 85978.414 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 49408
>> 1365551560.220 77310.164 69.113.13.218 10.1.1.10
>> 10.1.1.101 arp 29696
>> 1365501007.827 24840.254 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1365551938.461 86340.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 157440
>> 1365551953.810 86274.445 69.113.13.218 10.1.1.46
>> 10.1.1.10 arp 107904
>> 1365551964.909 86302.922 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 152960
>> 1365551889.353 86206.047 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 108544
>> 1365475483.795 786.242 69.113.13.218 10.1.1.101
>> 10.1.1.10 arp 896
>> 1365502197.518 24840.180 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>>
>>
>>
>> Then I set these three running on the machine with the database
>> (argus-clients-3.0.7.8)
>> /usr/local/bin/radium -f /usr/local/argus/SNKradium.conf -d
>> /usr/local/bin/rastream -S localhost:9603 -f /usr/local/argus/SNKstream.sh
>> -M time 1h -B 15 -w /data/argus/%Y/%m/%d/argus.%Y.%m.%d.%H -d
>> /usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w
>> mysql://argus@localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime
>> dur srcid saddr daddr proto bytes -d
>>
>>
>> # cat /usr/local/argus/SNKradium.conf
>> RADIUM_DAEMON=no
>> RADIUM_CLASSIFIER_FILE=/usr/local/argus/SNKlabel.conf
>> RADIUM_ACCESS_PORT=9603
>> RADIUM_ARGUS_SERVER=rodnel-new:561
>>
>> The SNKstream.sh file doesn't do anything but gzip the file.
>>
>> Now I get these results:
>> The MySQL table is a bit unusual but not absolutely awful:
>>
>> mysql> select ltime,dur,srcid,saddr, daddr, proto, bytes from
>> matrix_2013_05_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10';
>> +-------------------+--------------+---------------+------------+-----------
>> ------+-------+------------+
>> | ltime | dur | srcid | saddr | daddr
>> | proto | bytes |
>> +-------------------+--------------+---------------+------------+-----------
>> ------+-------+------------+
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
>> | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
>> | icmp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.50 | 10.1.1.10
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
>> | tcp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
>> 255.255.255.255 | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.60
>> | tcp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
>> | tcp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.60 | 10.1.1.10
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.45 | 10.1.1.10
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.127
>> | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
>> | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
>> | icmp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.126
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
>> 167.206.245.130 | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
>> 113.37.91.61 | icmp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
>> 113.37.91.61 | tcp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.126 | 10.1.1.10
>> | arp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.71
>> | udp | 4609938798 |
>> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.71 | 10.1.1.10
>> | arp | 4609938798 |
>> +-------------------+--------------+---------------+------------+-----------
>> ------+-------+------------+
>> 20 rows in set (0.00 sec)
>>
>> The files created by rastream look correct:
>>
>> racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
>> daddr proto bytes - host 10.1.1.10
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368143927.352 86340.555 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1346514
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
>> 10.1.1.50 udp 1674059
>> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1368110304.637 0.001 69.113.13.218 10.1.1.10
>> 10.1.1.71 udp 188
>> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 35811
>> 1368143944.782 86342.789 69.113.13.218 10.1.1.10
>> 10.1.1.45 tcp 173423905*
>> 1368143942.334 86354.383 69.113.13.218 10.1.1.10
>> 10.1.1.50 tcp 3945119719
>> 1368143931.073 86329.680 69.113.13.218 10.1.1.10
>> 10.1.1.60 tcp 429295
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
>> 10.1.1.45 icmp 2718
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1439680
>> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 60032
>> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 160128
>> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 158848
>> 1368143875.921 86209.273 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 109440
>> 1368110304.635 0.000 69.113.13.218 10.1.1.71
>> 10.1.1.10 arp 128
>> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>>
>> Then we come to the output of rasql which for some reason informs me way too
>> many times that something on my network (10.1.1.0/25) sent a bunch of
>> traffic to somewhere NB: this is the one place where I see CIDR notation and
>> that might be is a clue.
>>
>> rasql -u -r mysql://argus@localhost/argus/matrix_2013_05_09 -M sql=" saddr =
>> '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
>> 0.0.0.0/4 ip 4609938798
>>
>> So I copied one day's files from the system running argus to a clean
>> directory on the MySQL machine and ran the rasqlinsert incantation that I
>> always used to use:
>>
>> rasqlinsert -M time 1d -r * -w
>> mysql://argus@localhost/argus/testMatrix_%Y_%m_%d -m srcid matrix proto -s
>> ltime dur srcid saddr daddr proto bytes
>>
>> and got results like the ones I used to get:
>>
>> rasql -u -r mysql://argus@localhost/argus/testMatrix_2013_05_09 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368143991.233 86389.844 69.113.13.218 10.1.1.60
>> 10.1.1.10 tcp 429589
>> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
>> 10.1.1.10 tcp 173086923*
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
>> 10.1.1.10 udp 1674059
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1439680
>> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 158848
>> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
>> 10.1.1.10 tcp 3945115585
>> 1368143987.364 86340.555 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1346514
>> 1368143996.241 86329.594 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 109568
>> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 160128
>> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 60032
>> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
>> 10.1.1.10 icmp 2718
>> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 35811
>> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>> 1368110304.635 0.000 69.113.13.218 10.1.1.71
>> 10.1.1.10 arp 128
>> 1368110304.637 0.001 69.113.13.218 10.1.1.71
>> 10.1.1.10 udp 188
>>
>> Then I used the files created by rastream to do the same thing (remember
>> that these files came from the same radium feed as fed the rasqlinsert that
>> wasn't so good)
>> cd /data/argus/2013/05/09
>> rasqlinsert -M time 1d -r * -w
>> mysql://argus@localhost/argus/test2Matrix_%Y_%m_%d -m srcid matrix proto -s
>> ltime dur srcid saddr daddr proto bytes
>>
>> and got the results that I expected:
>>
>> rasql -u -r mysql://argus@localhost/argus/test2Matrix_2013_05_09 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368143931.073 86329.680 69.113.13.218 10.1.1.60
>> 10.1.1.10 tcp 429295
>> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
>> 10.1.1.10 tcp 173423905*
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
>> 10.1.1.10 udp 1674059
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1439680
>> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 158848
>> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
>> 10.1.1.10 tcp 3945115585
>> 1368143927.352 86280.539 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1345782
>> 1368143875.921 86209.273 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 109440
>> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 160128
>> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 60032
>> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
>> 10.1.1.10 icmp 2718
>> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 35811
>> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>> 1368110304.635 0.000 69.113.13.218 10.1.1.71
>> 10.1.1.10 arp 128
>> 1368110304.637 0.001 69.113.13.218 10.1.1.71
>> 10.1.1.10 udp 188
>>
>> Just in case the -M cache is making a difference, I included it in a test
>> and it didn't break anything:
>>
>> rasqlinsert -M time 1d -r * -M cache -w
>> mysql://argus@localhost/argus/test3Matrix_%Y_%m_%d -m srcid matrix proto -s
>> ltime dur srcid saddr daddr proto bytes
>> rasql -u -r mysql://argus@localhost/argus/test3Matrix_2013_05_09 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368143991.233 86389.844 69.113.13.218 10.1.1.60
>> 10.1.1.10 tcp 429589
>> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
>> 10.1.1.10 tcp 173086923*
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
>> 10.1.1.10 udp 1674059
>> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 1439680
>> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 158848
>> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
>> 10.1.1.10 tcp 3945115585
>> 1368143987.364 86340.555 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 1346514
>> 1368143996.241 86329.594 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 109568
>> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 160128
>> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 60032
>> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 61560
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
>> 10.1.1.45 udp 2130
>> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
>> 10.1.1.10 icmp 2718
>> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
>> 10.1.1.126 arp 384
>> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
>> 167.206.245.130 udp 5766
>> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
>> 113.37.91.61 icmp 612
>> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
>> 113.37.91.61 tcp 35811
>> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
>> 10.1.1.10 arp 384
>> 1368110304.635 0.000 69.113.13.218 10.1.1.71
>> 10.1.1.10 arp 128
>> 1368110304.637 0.001 69.113.13.218 10.1.1.71
>> 10.1.1.10 udp 188
>>
>> I kill CIDR notation in my ~/.rarc file to see what happens (I dropped the
>> current table and restarted the clients) and it is looking much better
>>
>> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_10 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368153616.837 60.160 69.113.13.218 10.1.1.60
>> 10.1.1.10 tcp 588
>> 1368154260.248 700.975 69.113.13.218 10.1.1.50
>> 10.1.1.10 udp 13899
>> 1368154260.248 700.975 69.113.13.218 10.1.1.10
>> 10.1.1.50 icmp 10688
>> 1368153868.734 307.102 69.113.13.218 10.1.1.50
>> 10.1.1.10 tcp 425755
>> 1368154283.605 721.920 69.113.13.218 10.1.1.60
>> 10.1.1.10 arp 1408
>> 1368154247.544 675.886 69.113.13.218 10.1.1.10
>> 255.255.255.255 udp 12078
>> 1368153964.784 360.043 69.113.13.218 10.1.1.45
>> 10.1.1.10 tcp 10108
>> 1368154172.306 566.982 69.113.13.218 10.1.1.50
>> 10.1.1.10 arp 1280
>> 1368154209.756 600.000 69.113.13.218 10.1.1.45
>> 10.1.1.10 arp 1408
>> 1368153742.552 0.000 69.113.13.218 10.1.1.10
>> 10.1.1.127 udp 513
>> 1368153908.043 67.891 69.113.13.218 10.1.1.10
>> 10.1.1.50 arp 256
>>
>>
>> The fix is not retroactive, NB: the testMatrix, test2Matrix, and test3Matrix
>> tables were all generated by rasqlinsert with the .rarc containing
>> RA_CIDR_ADDRESS_FORMAT="yes" and they were fine so it looks like an
>> interaction between CIDR notation and rasqlinsert -S from a radium source
>>
>> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_09 -M sql="
>> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
>> LastTime Dur SrcId SrcAddr
>> DstAddr Proto TotBytes
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
>> 0.0.0.0 ip 4609938798
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130510/c7caeb0c/attachment.html>
More information about the argus
mailing list