RA_CIDR_ADDRESS_FORMAT="yes" and rasqlinsert -S [radium source] may be a problem
Carter Bullard
carter at qosient.com
Fri May 10 10:55:29 EDT 2013
Hey Dave,
Wow, now that is a long email !!!!
OK, the " RA_CIDR_ADDRESS_FORMAT=yes " variable should only come
into play when there is aggregation.
So in your data flow machine that you have generated:
argus -> radium --> rastream() -----> archive
|
+------> rasqlinsert() ---> DB
only rasqlinsert() is an aggregator ( -m srcid matrix proto). Nothing in the
data coming from argus() or radium() will result in a CIDR IP address.
A CIDR address happens when the ARGUS_FLOW_DSR contains values
in the IP address mask fields, which will be zero, coming from argus(),
radium() and rastream() in your setup.
While rasqlinsert() aggregation has the potential to generate CIDR
addresses, given the aggregation key that rasqlinsert() is using
(-m srcid matrix proto ), there shouldn't be any CIDR addresses any
where in your information system.
Now, based on what you're showing, everything seems to be OK. The data
in the archive is good, and the mysql database values for the src and dst
addresses all look good, in that there aren't any CIDR addresses showing
up. Because rasqlinsert() is like any other ra* program, if there was a real
CIDR address in the record when it is inserted into the DB,
What seems to be errant, are the argus records that are in the DB.
rasqlinsert() populates the database with attributes using strings that
it generates, using the same routines that ra* programs use to write to the
terminal ( ArgusPrintRecord() ). But, behind the scene, rasqlinsert() also
writes into the database, the binary argus record that it has aggregated.
This is the data that rasql() will fetch from the database and use to
print its output.
So, when mysql() reads the address data from the table, no CIDR address
strings. But when rasql() reads the binary argus data from the table,
we get CIDRs. I suspect that we're getting some data corruption, such
that the " smask " and " dmask " fields end up with values in them, which
makes rasql() print the addresses as CIDR values.
If this is the case, then when rasql() reads the data from the table with the
" RA_CIDR_ADDRESS_FORMAT= no ", then the data should be good.
Does this describe what your seeing ? If so, there are a few things we
can do to fix…...
Carter
On May 9, 2013, at 11:14 PM, "Dave Edelman" <dedelman at iname.com> wrote:
> I have a single instance of argus that has been running for years and
> creating hourly files of the flow data. On a daily basis, I copy a day's
> worth of the flow files to a second system where I run rqasqlinsert in
> various flavors to create several different tables.
>
> I finally decided to use radium and rastream to do this the right way (but I
> didn't stop the local file creation, just to be safe.)
>
> The details follow but it looks like there is a toxic interaction between
> RA_CIDR_ADDRESS_FORMAT="yes" in ~/.rarc and rasqlinsert using -S from a
> radium instance in client version 3.0.7.8 and possibly earlier.
>
>
> --------- Danger beyond this point are the gory details
> --------------------------------
>
> The original method worked very well:
> A typical MySQL table queried for all 10.1.1.10 activity gave reasonable
> results (I learned not to select the record blob :-) )
>
> mysql> select ltime, dur, saddr, daddr, proto, bytes from matrix_2013_04_09
> where saddr = '10.1.1.10' or daddr = '10.1.1.10';
> +-------------------+--------------+------------+-----------------+-------+-
> ------------+
> | ltime | dur | saddr | daddr | proto |
> bytes |
> +-------------------+--------------+------------+-----------------+-------+-
> ------------+
> | 1365551889.364000 | 86287.953000 | 10.1.1.10 | 10.1.1.60 | tcp |
> 843282 |
> | 1365551948.803000 | 86347.195000 | 10.1.1.10 | 10.1.1.46 | tcp |
> 783536 |
> | 1365551993.490000 | 86391.336000 | 10.1.1.10 | 10.1.1.45 | tcp |
> 16050742461 |
> | 1365551986.978000 | 86381.195000 | 10.1.1.10 | 10.1.1.50 | tcp |
> 3962458654 |
> | 1365551957.703000 | 86340.031000 | 10.1.1.10 | 255.255.255.255 | udp |
> 1346514 |
> | 1365551992.462000 | 86374.555000 | 10.1.1.10 | 10.1.1.50 | udp |
> 1667967 |
> | 1365551992.462000 | 86369.734000 | 10.1.1.10 | 10.1.1.50 | icmp |
> 1426656 |
> | 1365551938.461000 | 86280.000000 | 10.1.1.45 | 10.1.1.10 | arp |
> 157312 |
> | 1365551964.909000 | 86302.922000 | 10.1.1.50 | 10.1.1.10 | arp |
> 152960 |
> | 1365551953.810000 | 86274.445000 | 10.1.1.46 | 10.1.1.10 | arp |
> 107904 |
> | 1365551889.353000 | 86206.047000 | 10.1.1.60 | 10.1.1.10 | arp |
> 108544 |
> | 1365551868.423000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | udp |
> 30672 |
> | 1365551873.421000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | arp |
> 36352 |
> | 1365551639.982000 | 85736.523000 | 10.1.1.10 | 10.1.1.127 | udp |
> 61560 |
> | 1365551920.751000 | 85978.414000 | 10.1.1.10 | 10.1.1.50 | arp |
> 49408 |
> | 1365551560.220000 | 77317.164000 | 10.1.1.10 | 10.1.1.101 | udp |
> 115032 |
> | 1365551560.220000 | 77310.164000 | 10.1.1.10 | 10.1.1.101 | arp |
> 29696 |
> | 1365550944.567000 | 76398.391000 | 10.1.1.10 | 224.0.0.251 | udp |
> 3703 |
> | 1365529023.282000 | 54396.770000 | 10.1.1.10 | 10.1.1.101 | tcp |
> 1032139 |
> | 1365475483.795000 | 786.242000 | 10.1.1.101 | 10.1.1.10 | arp |
> 896 |
> | 1365501012.119000 | 24844.955000 | 10.1.1.10 | 10.1.1.45 | udp |
> 2130 |
> | 1365501012.119000 | 24844.953000 | 10.1.1.10 | 10.1.1.45 | icmp |
> 2718 |
> | 1365501007.827000 | 24840.254000 | 10.1.1.10 | 10.1.1.126 | arp |
> 384 |
> | 1365501012.354000 | 24844.787000 | 10.1.1.10 | 167.206.245.130 | udp |
> 5766 |
> | 1365501008.507000 | 24840.812000 | 10.1.1.10 | 113.37.91.61 | icmp |
> 612 |
> | 1365501012.895000 | 24844.477000 | 10.1.1.10 | 113.37.91.61 | tcp |
> 37293 |
> | 1365502197.518000 | 24840.180000 | 10.1.1.126 | 10.1.1.10 | arp |
> 384 |
> +-------------------+--------------+------------+-----------------+-------+-
> ------------+
> 27 rows in set (0.00 sec)
>
> rasql gave happy results:
>
> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_04_09 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1365551889.364 86287.953 69.113.13.218 10.1.1.60
> 10.1.1.10 tcp 843282
> 1365551948.803 86347.195 69.113.13.218 10.1.1.46
> 10.1.1.10 tcp 783536
> 1365551993.490 86391.336 69.113.13.218 10.1.1.45
> 10.1.1.10 tcp 160507424*
> 1365551986.978 86381.195 69.113.13.218 10.1.1.50
> 10.1.1.10 tcp 3962458654
> 1365551957.703 86340.031 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1346514
> 1365551992.462 86374.555 69.113.13.218 10.1.1.10
> 10.1.1.50 udp 1667967
> 1365551992.462 86369.734 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1426656
> 1365551938.461 86280.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 157312
> 1365551964.909 86302.922 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 152960
> 1365551953.810 86274.445 69.113.13.218 10.1.1.46
> 10.1.1.10 arp 107904
> 1365551889.353 86206.047 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 108544
> 1365551868.423 86055.141 69.113.13.218 10.1.1.10
> 10.1.1.12 udp 30672
> 1365551873.421 86055.141 69.113.13.218 10.1.1.10
> 10.1.1.12 arp 36352
> 1365551639.982 85736.523 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1365551920.751 85978.414 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 49408
> 1365551560.220 77317.164 69.113.13.218 10.1.1.10
> 10.1.1.101 udp 115032
> 1365551560.220 77310.164 69.113.13.218 10.1.1.10
> 10.1.1.101 arp 29696
> 1365550944.567 76398.391 69.113.13.218 10.1.1.10
> 224.0.0.251 udp 3703
> 1365529023.282 54396.770 69.113.13.218 10.1.1.101
> 10.1.1.10 tcp 1032139
> 1365475483.795 786.242 69.113.13.218 10.1.1.101
> 10.1.1.10 arp 896
> 1365501012.119 24844.955 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1365501012.119 24844.953 69.113.13.218 10.1.1.45
> 10.1.1.10 icmp 2718
> 1365501007.827 24840.254 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1365501012.354 24844.787 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1365501008.507 24840.812 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1365501012.895 24844.477 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 37293
> 1365502197.518 24840.180 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
>
> And a confirmation from the original flow files checked out well
>
> racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
> daddr proto bytes - host 10.1.1.10
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1365551992.462 86374.555 69.113.13.218 10.1.1.10
> 10.1.1.50 udp 1667967
> 1365551560.220 77317.164 69.113.13.218 10.1.1.10
> 10.1.1.101 udp 115032
> 1365551957.703 86340.031 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1346514
> 1365501012.354 24844.787 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1365551868.423 86055.141 69.113.13.218 10.1.1.10
> 10.1.1.12 udp 30672
> 1365551639.982 85736.523 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1365501012.119 24844.955 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1365550944.567 76398.391 69.113.13.218 10.1.1.10
> 224.0.0.251 udp 3703
> 1365551986.978 86389.391 69.113.13.218 10.1.1.10
> 10.1.1.50 tcp 3962486779
> 1365529023.282 54396.770 69.113.13.218 10.1.1.10
> 10.1.1.101 tcp 1032139
> 1365501012.895 24844.477 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 37293
> 1365551993.490 86391.336 69.113.13.218 10.1.1.10
> 10.1.1.45 tcp 160507424*
> 1365551948.803 86347.195 69.113.13.218 10.1.1.10
> 10.1.1.46 tcp 783536
> 1365551889.364 86287.953 69.113.13.218 10.1.1.10
> 10.1.1.60 tcp 843282
> 1365501012.119 24844.953 69.113.13.218 10.1.1.10
> 10.1.1.45 icmp 2718
> 1365551992.462 86369.734 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1426656
> 1365501008.507 24840.812 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1365551873.421 86055.141 69.113.13.218 10.1.1.10
> 10.1.1.12 arp 36352
> 1365551920.751 85978.414 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 49408
> 1365551560.220 77310.164 69.113.13.218 10.1.1.10
> 10.1.1.101 arp 29696
> 1365501007.827 24840.254 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1365551938.461 86340.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 157440
> 1365551953.810 86274.445 69.113.13.218 10.1.1.46
> 10.1.1.10 arp 107904
> 1365551964.909 86302.922 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 152960
> 1365551889.353 86206.047 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 108544
> 1365475483.795 786.242 69.113.13.218 10.1.1.101
> 10.1.1.10 arp 896
> 1365502197.518 24840.180 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
>
>
>
> Then I set these three running on the machine with the database
> (argus-clients-3.0.7.8)
> /usr/local/bin/radium -f /usr/local/argus/SNKradium.conf -d
> /usr/local/bin/rastream -S localhost:9603 -f /usr/local/argus/SNKstream.sh
> -M time 1h -B 15 -w /data/argus/%Y/%m/%d/argus.%Y.%m.%d.%H -d
> /usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w
> mysql://argus@localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime
> dur srcid saddr daddr proto bytes -d
>
>
> # cat /usr/local/argus/SNKradium.conf
> RADIUM_DAEMON=no
> RADIUM_CLASSIFIER_FILE=/usr/local/argus/SNKlabel.conf
> RADIUM_ACCESS_PORT=9603
> RADIUM_ARGUS_SERVER=rodnel-new:561
>
> The SNKstream.sh file doesn't do anything but gzip the file.
>
> Now I get these results:
> The MySQL table is a bit unusual but not absolutely awful:
>
> mysql> select ltime,dur,srcid,saddr, daddr, proto, bytes from
> matrix_2013_05_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10';
> +-------------------+--------------+---------------+------------+-----------
> ------+-------+------------+
> | ltime | dur | srcid | saddr | daddr
> | proto | bytes |
> +-------------------+--------------+---------------+------------+-----------
> ------+-------+------------+
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
> | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
> | icmp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.50 | 10.1.1.10
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
> | tcp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
> 255.255.255.255 | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.60
> | tcp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
> | tcp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.60 | 10.1.1.10
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.45 | 10.1.1.10
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.127
> | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
> | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
> | icmp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.126
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
> 167.206.245.130 | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
> 113.37.91.61 | icmp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
> 113.37.91.61 | tcp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.126 | 10.1.1.10
> | arp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.71
> | udp | 4609938798 |
> | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.71 | 10.1.1.10
> | arp | 4609938798 |
> +-------------------+--------------+---------------+------------+-----------
> ------+-------+------------+
> 20 rows in set (0.00 sec)
>
> The files created by rastream look correct:
>
> racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
> daddr proto bytes - host 10.1.1.10
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368143927.352 86340.555 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1346514
> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
> 10.1.1.50 udp 1674059
> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1368110304.637 0.001 69.113.13.218 10.1.1.10
> 10.1.1.71 udp 188
> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 35811
> 1368143944.782 86342.789 69.113.13.218 10.1.1.10
> 10.1.1.45 tcp 173423905*
> 1368143942.334 86354.383 69.113.13.218 10.1.1.10
> 10.1.1.50 tcp 3945119719
> 1368143931.073 86329.680 69.113.13.218 10.1.1.10
> 10.1.1.60 tcp 429295
> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
> 10.1.1.45 icmp 2718
> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1439680
> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 60032
> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 160128
> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 158848
> 1368143875.921 86209.273 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 109440
> 1368110304.635 0.000 69.113.13.218 10.1.1.71
> 10.1.1.10 arp 128
> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
>
> Then we come to the output of rasql which for some reason informs me way too
> many times that something on my network (10.1.1.0/25) sent a bunch of
> traffic to somewhere NB: this is the one place where I see CIDR notation and
> that might be is a clue.
>
> rasql -u -r mysql://argus@localhost/argus/matrix_2013_05_09 -M sql=" saddr =
> '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
> 0.0.0.0/4 ip 4609938798
>
> So I copied one day's files from the system running argus to a clean
> directory on the MySQL machine and ran the rasqlinsert incantation that I
> always used to use:
>
> rasqlinsert -M time 1d -r * -w
> mysql://argus@localhost/argus/testMatrix_%Y_%m_%d -m srcid matrix proto -s
> ltime dur srcid saddr daddr proto bytes
>
> and got results like the ones I used to get:
>
> rasql -u -r mysql://argus@localhost/argus/testMatrix_2013_05_09 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368143991.233 86389.844 69.113.13.218 10.1.1.60
> 10.1.1.10 tcp 429589
> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
> 10.1.1.10 tcp 173086923*
> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
> 10.1.1.10 udp 1674059
> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1439680
> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 158848
> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
> 10.1.1.10 tcp 3945115585
> 1368143987.364 86340.555 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1346514
> 1368143996.241 86329.594 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 109568
> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 160128
> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 60032
> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
> 10.1.1.10 icmp 2718
> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 35811
> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
> 1368110304.635 0.000 69.113.13.218 10.1.1.71
> 10.1.1.10 arp 128
> 1368110304.637 0.001 69.113.13.218 10.1.1.71
> 10.1.1.10 udp 188
>
> Then I used the files created by rastream to do the same thing (remember
> that these files came from the same radium feed as fed the rasqlinsert that
> wasn't so good)
> cd /data/argus/2013/05/09
> rasqlinsert -M time 1d -r * -w
> mysql://argus@localhost/argus/test2Matrix_%Y_%m_%d -m srcid matrix proto -s
> ltime dur srcid saddr daddr proto bytes
>
> and got the results that I expected:
>
> rasql -u -r mysql://argus@localhost/argus/test2Matrix_2013_05_09 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368143931.073 86329.680 69.113.13.218 10.1.1.60
> 10.1.1.10 tcp 429295
> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
> 10.1.1.10 tcp 173423905*
> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
> 10.1.1.10 udp 1674059
> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1439680
> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 158848
> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
> 10.1.1.10 tcp 3945115585
> 1368143927.352 86280.539 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1345782
> 1368143875.921 86209.273 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 109440
> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 160128
> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 60032
> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
> 10.1.1.10 icmp 2718
> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 35811
> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
> 1368110304.635 0.000 69.113.13.218 10.1.1.71
> 10.1.1.10 arp 128
> 1368110304.637 0.001 69.113.13.218 10.1.1.71
> 10.1.1.10 udp 188
>
> Just in case the -M cache is making a difference, I included it in a test
> and it didn't break anything:
>
> rasqlinsert -M time 1d -r * -M cache -w
> mysql://argus@localhost/argus/test3Matrix_%Y_%m_%d -m srcid matrix proto -s
> ltime dur srcid saddr daddr proto bytes
> rasql -u -r mysql://argus@localhost/argus/test3Matrix_2013_05_09 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368143991.233 86389.844 69.113.13.218 10.1.1.60
> 10.1.1.10 tcp 429589
> 1368143944.782 86342.789 69.113.13.218 10.1.1.45
> 10.1.1.10 tcp 173086923*
> 1368143957.087 86335.609 69.113.13.218 10.1.1.50
> 10.1.1.10 udp 1674059
> 1368143957.087 86335.609 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 1439680
> 1368143942.065 86316.039 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 158848
> 1368143942.334 86313.445 69.113.13.218 10.1.1.50
> 10.1.1.10 tcp 3945115585
> 1368143987.364 86340.555 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 1346514
> 1368143996.241 86329.594 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 109568
> 1368143949.756 86280.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 160128
> 1368143870.011 85965.906 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 60032
> 1368143651.470 85679.195 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 61560
> 1368106993.205 44648.055 69.113.13.218 10.1.1.10
> 10.1.1.45 udp 2130
> 1368106993.205 44648.055 69.113.13.218 10.1.1.45
> 10.1.1.10 icmp 2718
> 1368106986.534 44640.969 69.113.13.218 10.1.1.10
> 10.1.1.126 arp 384
> 1368106993.711 44648.406 69.113.13.218 10.1.1.10
> 167.206.245.130 udp 5766
> 1368106987.571 44641.789 69.113.13.218 10.1.1.10
> 113.37.91.61 icmp 612
> 1368106995.036 44648.059 69.113.13.218 10.1.1.10
> 113.37.91.61 tcp 35811
> 1368108176.467 44641.297 69.113.13.218 10.1.1.126
> 10.1.1.10 arp 384
> 1368110304.635 0.000 69.113.13.218 10.1.1.71
> 10.1.1.10 arp 128
> 1368110304.637 0.001 69.113.13.218 10.1.1.71
> 10.1.1.10 udp 188
>
> I kill CIDR notation in my ~/.rarc file to see what happens (I dropped the
> current table and restarted the clients) and it is looking much better
>
> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_10 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368153616.837 60.160 69.113.13.218 10.1.1.60
> 10.1.1.10 tcp 588
> 1368154260.248 700.975 69.113.13.218 10.1.1.50
> 10.1.1.10 udp 13899
> 1368154260.248 700.975 69.113.13.218 10.1.1.10
> 10.1.1.50 icmp 10688
> 1368153868.734 307.102 69.113.13.218 10.1.1.50
> 10.1.1.10 tcp 425755
> 1368154283.605 721.920 69.113.13.218 10.1.1.60
> 10.1.1.10 arp 1408
> 1368154247.544 675.886 69.113.13.218 10.1.1.10
> 255.255.255.255 udp 12078
> 1368153964.784 360.043 69.113.13.218 10.1.1.45
> 10.1.1.10 tcp 10108
> 1368154172.306 566.982 69.113.13.218 10.1.1.50
> 10.1.1.10 arp 1280
> 1368154209.756 600.000 69.113.13.218 10.1.1.45
> 10.1.1.10 arp 1408
> 1368153742.552 0.000 69.113.13.218 10.1.1.10
> 10.1.1.127 udp 513
> 1368153908.043 67.891 69.113.13.218 10.1.1.10
> 10.1.1.50 arp 256
>
>
> The fix is not retroactive, NB: the testMatrix, test2Matrix, and test3Matrix
> tables were all generated by rasqlinsert with the .rarc containing
> RA_CIDR_ADDRESS_FORMAT="yes" and they were fine so it looks like an
> interaction between CIDR notation and rasqlinsert -S from a radium source
>
> rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_09 -M sql="
> saddr = '10.1.1.10' or daddr = '10.1.1.10'"
> LastTime Dur SrcId SrcAddr
> DstAddr Proto TotBytes
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
> 1368144006.774 86377.883 69.113.13.218 10.1.1.0
> 0.0.0.0 ip 4609938798
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130510/a1e0d703/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130510/a1e0d703/attachment.bin>
More information about the argus
mailing list