RA_CIDR_ADDRESS_FORMAT="yes" and rasqlinsert -S [radium source] may be a problem
Dave Edelman
dedelman at iname.com
Sat May 11 00:05:01 EDT 2013
Carter,
I think that my response may have been too terse. Any record that was
inserted while RA_CIDR_ADDRESS_FORMAT=yes is bad when it is retrieved
regardless of the value of RA_CIDR_ADDRESS_FORMAT at the time that it is
retrieved. Any record that is inserted while RA_CIDR_ADDRESS_FORMAT=no is
fine regardless of the value of RA_CIDR_ADDRESS_FORMAT at the time of
retrieval.
It looks like the dmask and smask fields are in the flow DSR and since ra
-M dsrs=-flow is more or less a self-fulfilling prophecy, the obvious quick
fix isn't an option.
I glued together a quick client that reads the flow records, creates the
duplicate structure, prints out the values of dmask and smask and sets them
to 0 then sends the rewritten record to the next client in the chain:
rasql -w - | rafixit -w - | ra
The dmask and smask values are set but making them 0 doesn't fix the
problem since saddr and daddr are already munged up.
--Dave
From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, May 10, 2013 10:55 AM
To: Dave Edelman
Cc: Argus
Subject: Re: RA_CIDR_ADDRESS_FORMAT="yes" and rasqlinsert -S [radium source]
may be a problem
Hey Dave,
Wow, now that is a long email !!!!
OK, the " RA_CIDR_ADDRESS_FORMAT=yes " variable should only come
into play when there is aggregation.
So in your data flow machine that you have generated:
argus -> radium --> rastream() -----> archive
|
+------> rasqlinsert() ---> DB
only rasqlinsert() is an aggregator ( -m srcid matrix proto). Nothing in
the
data coming from argus() or radium() will result in a CIDR IP address.
A CIDR address happens when the ARGUS_FLOW_DSR contains values
in the IP address mask fields, which will be zero, coming from argus(),
radium() and rastream() in your setup.
While rasqlinsert() aggregation has the potential to generate CIDR
addresses, given the aggregation key that rasqlinsert() is using
(-m srcid matrix proto ), there shouldn't be any CIDR addresses any
where in your information system.
Now, based on what you're showing, everything seems to be OK. The data
in the archive is good, and the mysql database values for the src and dst
addresses all look good, in that there aren't any CIDR addresses showing
up. Because rasqlinsert() is like any other ra* program, if there was a
real
CIDR address in the record when it is inserted into the DB,
What seems to be errant, are the argus records that are in the DB.
rasqlinsert() populates the database with attributes using strings that
it generates, using the same routines that ra* programs use to write to the
terminal ( ArgusPrintRecord() ). But, behind the scene, rasqlinsert() also
writes into the database, the binary argus record that it has aggregated.
This is the data that rasql() will fetch from the database and use to
print its output.
So, when mysql() reads the address data from the table, no CIDR address
strings. But when rasql() reads the binary argus data from the table,
we get CIDRs. I suspect that we're getting some data corruption, such
that the " smask " and " dmask " fields end up with values in them, which
makes rasql() print the addresses as CIDR values.
If this is the case, then when rasql() reads the data from the table with
the
" RA_CIDR_ADDRESS_FORMAT= no ", then the data should be good.
Does this describe what your seeing ? If so, there are a few things we
can do to fix....
Carter
On May 9, 2013, at 11:14 PM, "Dave Edelman" <dedelman at iname.com> wrote:
I have a single instance of argus that has been running for years and
creating hourly files of the flow data. On a daily basis, I copy a day's
worth of the flow files to a second system where I run rqasqlinsert in
various flavors to create several different tables.
I finally decided to use radium and rastream to do this the right way (but I
didn't stop the local file creation, just to be safe.)
The details follow but it looks like there is a toxic interaction between
RA_CIDR_ADDRESS_FORMAT="yes" in ~/.rarc and rasqlinsert using -S from a
radium instance in client version 3.0.7.8 and possibly earlier.
--------- Danger beyond this point are the gory details
--------------------------------
The original method worked very well:
A typical MySQL table queried for all 10.1.1.10 activity gave reasonable
results (I learned not to select the record blob :-) )
mysql> select ltime, dur, saddr, daddr, proto, bytes from matrix_2013_04_09
where saddr = '10.1.1.10' or daddr = '10.1.1.10';
+-------------------+--------------+------------+-----------------+-------+-
------------+
| ltime | dur | saddr | daddr | proto |
bytes |
+-------------------+--------------+------------+-----------------+-------+-
------------+
| 1365551889.364000 | 86287.953000 | 10.1.1.10 | 10.1.1.60 | tcp |
843282 |
| 1365551948.803000 | 86347.195000 | 10.1.1.10 | 10.1.1.46 | tcp |
783536 |
| 1365551993.490000 | 86391.336000 | 10.1.1.10 | 10.1.1.45 | tcp |
16050742461 |
| 1365551986.978000 | 86381.195000 | 10.1.1.10 | 10.1.1.50 | tcp |
3962458654 |
| 1365551957.703000 | 86340.031000 | 10.1.1.10 | 255.255.255.255 | udp |
1346514 |
| 1365551992.462000 | 86374.555000 | 10.1.1.10 | 10.1.1.50 | udp |
1667967 |
| 1365551992.462000 | 86369.734000 | 10.1.1.10 | 10.1.1.50 | icmp |
1426656 |
| 1365551938.461000 | 86280.000000 | 10.1.1.45 | 10.1.1.10 | arp |
157312 |
| 1365551964.909000 | 86302.922000 | 10.1.1.50 | 10.1.1.10 | arp |
152960 |
| 1365551953.810000 | 86274.445000 | 10.1.1.46 | 10.1.1.10 | arp |
107904 |
| 1365551889.353000 | 86206.047000 | 10.1.1.60 | 10.1.1.10 | arp |
108544 |
| 1365551868.423000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | udp |
30672 |
| 1365551873.421000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | arp |
36352 |
| 1365551639.982000 | 85736.523000 | 10.1.1.10 | 10.1.1.127 | udp |
61560 |
| 1365551920.751000 | 85978.414000 | 10.1.1.10 | 10.1.1.50 | arp |
49408 |
| 1365551560.220000 | 77317.164000 | 10.1.1.10 | 10.1.1.101 | udp |
115032 |
| 1365551560.220000 | 77310.164000 | 10.1.1.10 | 10.1.1.101 | arp |
29696 |
| 1365550944.567000 | 76398.391000 | 10.1.1.10 | 224.0.0.251 | udp |
3703 |
| 1365529023.282000 | 54396.770000 | 10.1.1.10 | 10.1.1.101 | tcp |
1032139 |
| 1365475483.795000 | 786.242000 | 10.1.1.101 | 10.1.1.10 | arp |
896 |
| 1365501012.119000 | 24844.955000 | 10.1.1.10 | 10.1.1.45 | udp |
2130 |
| 1365501012.119000 | 24844.953000 | 10.1.1.10 | 10.1.1.45 | icmp |
2718 |
| 1365501007.827000 | 24840.254000 | 10.1.1.10 | 10.1.1.126 | arp |
384 |
| 1365501012.354000 | 24844.787000 | 10.1.1.10 | 167.206.245.130 | udp |
5766 |
| 1365501008.507000 | 24840.812000 | 10.1.1.10 | 113.37.91.61 | icmp |
612 |
| 1365501012.895000 | 24844.477000 | 10.1.1.10 | 113.37.91.61 | tcp |
37293 |
| 1365502197.518000 | 24840.180000 | 10.1.1.126 | 10.1.1.10 | arp |
384 |
+-------------------+--------------+------------+-----------------+-------+-
------------+
27 rows in set (0.00 sec)
rasql gave happy results:
rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_04_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1365551889.364 86287.953 69.113.13.218 10.1.1.60
10.1.1.10 tcp 843282
1365551948.803 86347.195 69.113.13.218 10.1.1.46
10.1.1.10 tcp 783536
1365551993.490 86391.336 69.113.13.218 10.1.1.45
10.1.1.10 tcp 160507424*
1365551986.978 86381.195 69.113.13.218 10.1.1.50
10.1.1.10 tcp 3962458654
1365551957.703 86340.031 69.113.13.218 10.1.1.10
255.255.255.255 udp 1346514
1365551992.462 86374.555 69.113.13.218 10.1.1.10
10.1.1.50 udp 1667967
1365551992.462 86369.734 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1426656
1365551938.461 86280.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 157312
1365551964.909 86302.922 69.113.13.218 10.1.1.50
10.1.1.10 arp 152960
1365551953.810 86274.445 69.113.13.218 10.1.1.46
10.1.1.10 arp 107904
1365551889.353 86206.047 69.113.13.218 10.1.1.60
10.1.1.10 arp 108544
1365551868.423 86055.141 69.113.13.218 10.1.1.10
10.1.1.12 udp 30672
1365551873.421 86055.141 69.113.13.218 10.1.1.10
10.1.1.12 arp 36352
1365551639.982 85736.523 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1365551920.751 85978.414 69.113.13.218 10.1.1.10
10.1.1.50 arp 49408
1365551560.220 77317.164 69.113.13.218 10.1.1.10
10.1.1.101 udp 115032
1365551560.220 77310.164 69.113.13.218 10.1.1.10
10.1.1.101 arp 29696
1365550944.567 76398.391 69.113.13.218 10.1.1.10
224.0.0.251 udp 3703
1365529023.282 54396.770 69.113.13.218 10.1.1.101
10.1.1.10 tcp 1032139
1365475483.795 786.242 69.113.13.218 10.1.1.101
10.1.1.10 arp 896
1365501012.119 24844.955 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1365501012.119 24844.953 69.113.13.218 10.1.1.45
10.1.1.10 icmp 2718
1365501007.827 24840.254 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1365501012.354 24844.787 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1365501008.507 24840.812 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1365501012.895 24844.477 69.113.13.218 10.1.1.10
113.37.91.61 tcp 37293
1365502197.518 24840.180 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
And a confirmation from the original flow files checked out well
racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
daddr proto bytes - host 10.1.1.10
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1365551992.462 86374.555 69.113.13.218 10.1.1.10
10.1.1.50 udp 1667967
1365551560.220 77317.164 69.113.13.218 10.1.1.10
10.1.1.101 udp 115032
1365551957.703 86340.031 69.113.13.218 10.1.1.10
255.255.255.255 udp 1346514
1365501012.354 24844.787 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1365551868.423 86055.141 69.113.13.218 10.1.1.10
10.1.1.12 udp 30672
1365551639.982 85736.523 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1365501012.119 24844.955 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1365550944.567 76398.391 69.113.13.218 10.1.1.10
224.0.0.251 udp 3703
1365551986.978 86389.391 69.113.13.218 10.1.1.10
10.1.1.50 tcp 3962486779
1365529023.282 54396.770 69.113.13.218 10.1.1.10
10.1.1.101 tcp 1032139
1365501012.895 24844.477 69.113.13.218 10.1.1.10
113.37.91.61 tcp 37293
1365551993.490 86391.336 69.113.13.218 10.1.1.10
10.1.1.45 tcp 160507424*
1365551948.803 86347.195 69.113.13.218 10.1.1.10
10.1.1.46 tcp 783536
1365551889.364 86287.953 69.113.13.218 10.1.1.10
10.1.1.60 tcp 843282
1365501012.119 24844.953 69.113.13.218 10.1.1.10
10.1.1.45 icmp 2718
1365551992.462 86369.734 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1426656
1365501008.507 24840.812 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1365551873.421 86055.141 69.113.13.218 10.1.1.10
10.1.1.12 arp 36352
1365551920.751 85978.414 69.113.13.218 10.1.1.10
10.1.1.50 arp 49408
1365551560.220 77310.164 69.113.13.218 10.1.1.10
10.1.1.101 arp 29696
1365501007.827 24840.254 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1365551938.461 86340.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 157440
1365551953.810 86274.445 69.113.13.218 10.1.1.46
10.1.1.10 arp 107904
1365551964.909 86302.922 69.113.13.218 10.1.1.50
10.1.1.10 arp 152960
1365551889.353 86206.047 69.113.13.218 10.1.1.60
10.1.1.10 arp 108544
1365475483.795 786.242 69.113.13.218 10.1.1.101
10.1.1.10 arp 896
1365502197.518 24840.180 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
Then I set these three running on the machine with the database
(argus-clients-3.0.7.8)
/usr/local/bin/radium -f /usr/local/argus/SNKradium.conf -d
/usr/local/bin/rastream -S localhost:9603 -f /usr/local/argus/SNKstream.sh
-M time 1h -B 15 -w /data/argus/%Y/%m/%d/argus.%Y.%m.%d.%H -d
/usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w
mysql://argus@localhost/argus/matrix_%Y_%m_%d
<mysql://argus@localhost/argus/matrix_%25Y_%25m_%25d> -m srcid matrix proto
-s ltime
dur srcid saddr daddr proto bytes -d
# cat /usr/local/argus/SNKradium.conf
RADIUM_DAEMON=no
RADIUM_CLASSIFIER_FILE=/usr/local/argus/SNKlabel.conf
RADIUM_ACCESS_PORT=9603
RADIUM_ARGUS_SERVER=rodnel-new:561
The SNKstream.sh file doesn't do anything but gzip the file.
Now I get these results:
The MySQL table is a bit unusual but not absolutely awful:
mysql> select ltime,dur,srcid,saddr, daddr, proto, bytes from
matrix_2013_05_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10';
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
| ltime | dur | srcid | saddr | daddr
| proto | bytes |
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
| udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
| icmp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.50 | 10.1.1.10
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
| tcp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
255.255.255.255 | udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.60
| tcp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
| tcp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.60 | 10.1.1.10
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.45 | 10.1.1.10
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.127
| udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
| udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45
| icmp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.126
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
167.206.245.130 | udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
113.37.91.61 | icmp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 |
113.37.91.61 | tcp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.126 | 10.1.1.10
| arp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.71
| udp | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.71 | 10.1.1.10
| arp | 4609938798 |
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
20 rows in set (0.00 sec)
The files created by rastream look correct:
racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
daddr proto bytes - host 10.1.1.10
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368143927.352 86340.555 69.113.13.218 10.1.1.10
255.255.255.255 udp 1346514
1368143957.087 86335.609 69.113.13.218 10.1.1.10
10.1.1.50 udp 1674059
1368106993.711 44648.406 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1368143651.470 85679.195 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1368106993.205 44648.055 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1368110304.637 0.001 69.113.13.218 10.1.1.10
10.1.1.71 udp 188
1368106995.036 44648.059 69.113.13.218 10.1.1.10
113.37.91.61 tcp 35811
1368143944.782 86342.789 69.113.13.218 10.1.1.10
10.1.1.45 tcp 173423905*
1368143942.334 86354.383 69.113.13.218 10.1.1.10
10.1.1.50 tcp 3945119719
1368143931.073 86329.680 69.113.13.218 10.1.1.10
10.1.1.60 tcp 429295
1368106993.205 44648.055 69.113.13.218 10.1.1.10
10.1.1.45 icmp 2718
1368143957.087 86335.609 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1439680
1368106987.571 44641.789 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1368143870.011 85965.906 69.113.13.218 10.1.1.10
10.1.1.50 arp 60032
1368106986.534 44640.969 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1368143949.756 86280.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 160128
1368143942.065 86316.039 69.113.13.218 10.1.1.50
10.1.1.10 arp 158848
1368143875.921 86209.273 69.113.13.218 10.1.1.60
10.1.1.10 arp 109440
1368110304.635 0.000 69.113.13.218 10.1.1.71
10.1.1.10 arp 128
1368108176.467 44641.297 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
Then we come to the output of rasql which for some reason informs me way too
many times that something on my network (10.1.1.0/25) sent a bunch of
traffic to somewhere NB: this is the one place where I see CIDR notation and
that might be is a clue.
rasql -u -r mysql://argus@localhost/argus/matrix_2013_05_09 -M sql=" saddr =
'10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0/25
0.0.0.0/4 ip 4609938798
So I copied one day's files from the system running argus to a clean
directory on the MySQL machine and ran the rasqlinsert incantation that I
always used to use:
rasqlinsert -M time 1d -r * -w
mysql://argus@localhost/argus/testMatrix_%Y_%m_%d
<mysql://argus@localhost/argus/testMatrix_%25Y_%25m_%25d> -m srcid matrix
proto -s
ltime dur srcid saddr daddr proto bytes
and got results like the ones I used to get:
rasql -u -r mysql://argus@localhost/argus/testMatrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368143991.233 86389.844 69.113.13.218 10.1.1.60
10.1.1.10 tcp 429589
1368143944.782 86342.789 69.113.13.218 10.1.1.45
10.1.1.10 tcp 173086923*
1368143957.087 86335.609 69.113.13.218 10.1.1.50
10.1.1.10 udp 1674059
1368143957.087 86335.609 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1439680
1368143942.065 86316.039 69.113.13.218 10.1.1.50
10.1.1.10 arp 158848
1368143942.334 86313.445 69.113.13.218 10.1.1.50
10.1.1.10 tcp 3945115585
1368143987.364 86340.555 69.113.13.218 10.1.1.10
255.255.255.255 udp 1346514
1368143996.241 86329.594 69.113.13.218 10.1.1.60
10.1.1.10 arp 109568
1368143949.756 86280.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 160128
1368143870.011 85965.906 69.113.13.218 10.1.1.10
10.1.1.50 arp 60032
1368143651.470 85679.195 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1368106993.205 44648.055 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1368106993.205 44648.055 69.113.13.218 10.1.1.45
10.1.1.10 icmp 2718
1368106986.534 44640.969 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1368106993.711 44648.406 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1368106987.571 44641.789 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1368106995.036 44648.059 69.113.13.218 10.1.1.10
113.37.91.61 tcp 35811
1368108176.467 44641.297 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
1368110304.635 0.000 69.113.13.218 10.1.1.71
10.1.1.10 arp 128
1368110304.637 0.001 69.113.13.218 10.1.1.71
10.1.1.10 udp 188
Then I used the files created by rastream to do the same thing (remember
that these files came from the same radium feed as fed the rasqlinsert that
wasn't so good)
cd /data/argus/2013/05/09
rasqlinsert -M time 1d -r * -w
mysql://argus@localhost/argus/test2Matrix_%Y_%m_%d
<mysql://argus@localhost/argus/test2Matrix_%25Y_%25m_%25d> -m srcid matrix
proto -s
ltime dur srcid saddr daddr proto bytes
and got the results that I expected:
rasql -u -r mysql://argus@localhost/argus/test2Matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368143931.073 86329.680 69.113.13.218 10.1.1.60
10.1.1.10 tcp 429295
1368143944.782 86342.789 69.113.13.218 10.1.1.45
10.1.1.10 tcp 173423905*
1368143957.087 86335.609 69.113.13.218 10.1.1.50
10.1.1.10 udp 1674059
1368143957.087 86335.609 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1439680
1368143942.065 86316.039 69.113.13.218 10.1.1.50
10.1.1.10 arp 158848
1368143942.334 86313.445 69.113.13.218 10.1.1.50
10.1.1.10 tcp 3945115585
1368143927.352 86280.539 69.113.13.218 10.1.1.10
255.255.255.255 udp 1345782
1368143875.921 86209.273 69.113.13.218 10.1.1.60
10.1.1.10 arp 109440
1368143949.756 86280.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 160128
1368143870.011 85965.906 69.113.13.218 10.1.1.10
10.1.1.50 arp 60032
1368143651.470 85679.195 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1368106993.205 44648.055 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1368106993.205 44648.055 69.113.13.218 10.1.1.45
10.1.1.10 icmp 2718
1368106986.534 44640.969 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1368106993.711 44648.406 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1368106987.571 44641.789 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1368106995.036 44648.059 69.113.13.218 10.1.1.10
113.37.91.61 tcp 35811
1368108176.467 44641.297 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
1368110304.635 0.000 69.113.13.218 10.1.1.71
10.1.1.10 arp 128
1368110304.637 0.001 69.113.13.218 10.1.1.71
10.1.1.10 udp 188
Just in case the -M cache is making a difference, I included it in a test
and it didn't break anything:
rasqlinsert -M time 1d -r * -M cache -w
mysql://argus@localhost/argus/test3Matrix_%Y_%m_%d
<mysql://argus@localhost/argus/test3Matrix_%25Y_%25m_%25d> -m srcid matrix
proto -s
ltime dur srcid saddr daddr proto bytes
rasql -u -r mysql://argus@localhost/argus/test3Matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368143991.233 86389.844 69.113.13.218 10.1.1.60
10.1.1.10 tcp 429589
1368143944.782 86342.789 69.113.13.218 10.1.1.45
10.1.1.10 tcp 173086923*
1368143957.087 86335.609 69.113.13.218 10.1.1.50
10.1.1.10 udp 1674059
1368143957.087 86335.609 69.113.13.218 10.1.1.10
10.1.1.50 icmp 1439680
1368143942.065 86316.039 69.113.13.218 10.1.1.50
10.1.1.10 arp 158848
1368143942.334 86313.445 69.113.13.218 10.1.1.50
10.1.1.10 tcp 3945115585
1368143987.364 86340.555 69.113.13.218 10.1.1.10
255.255.255.255 udp 1346514
1368143996.241 86329.594 69.113.13.218 10.1.1.60
10.1.1.10 arp 109568
1368143949.756 86280.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 160128
1368143870.011 85965.906 69.113.13.218 10.1.1.10
10.1.1.50 arp 60032
1368143651.470 85679.195 69.113.13.218 10.1.1.10
10.1.1.127 udp 61560
1368106993.205 44648.055 69.113.13.218 10.1.1.10
10.1.1.45 udp 2130
1368106993.205 44648.055 69.113.13.218 10.1.1.45
10.1.1.10 icmp 2718
1368106986.534 44640.969 69.113.13.218 10.1.1.10
10.1.1.126 arp 384
1368106993.711 44648.406 69.113.13.218 10.1.1.10
167.206.245.130 udp 5766
1368106987.571 44641.789 69.113.13.218 10.1.1.10
113.37.91.61 icmp 612
1368106995.036 44648.059 69.113.13.218 10.1.1.10
113.37.91.61 tcp 35811
1368108176.467 44641.297 69.113.13.218 10.1.1.126
10.1.1.10 arp 384
1368110304.635 0.000 69.113.13.218 10.1.1.71
10.1.1.10 arp 128
1368110304.637 0.001 69.113.13.218 10.1.1.71
10.1.1.10 udp 188
I kill CIDR notation in my ~/.rarc file to see what happens (I dropped the
current table and restarted the clients) and it is looking much better
rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_10 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368153616.837 60.160 69.113.13.218 10.1.1.60
10.1.1.10 tcp 588
1368154260.248 700.975 69.113.13.218 10.1.1.50
10.1.1.10 udp 13899
1368154260.248 700.975 69.113.13.218 10.1.1.10
10.1.1.50 icmp 10688
1368153868.734 307.102 69.113.13.218 10.1.1.50
10.1.1.10 tcp 425755
1368154283.605 721.920 69.113.13.218 10.1.1.60
10.1.1.10 arp 1408
1368154247.544 675.886 69.113.13.218 10.1.1.10
255.255.255.255 udp 12078
1368153964.784 360.043 69.113.13.218 10.1.1.45
10.1.1.10 tcp 10108
1368154172.306 566.982 69.113.13.218 10.1.1.50
10.1.1.10 arp 1280
1368154209.756 600.000 69.113.13.218 10.1.1.45
10.1.1.10 arp 1408
1368153742.552 0.000 69.113.13.218 10.1.1.10
10.1.1.127 udp 513
1368153908.043 67.891 69.113.13.218 10.1.1.10
10.1.1.50 arp 256
The fix is not retroactive, NB: the testMatrix, test2Matrix, and test3Matrix
tables were all generated by rasqlinsert with the .rarc containing
RA_CIDR_ADDRESS_FORMAT="yes" and they were fine so it looks like an
interaction between CIDR notation and rasqlinsert -S from a radium source
rasql -u -r mysql://argus:argus@localhost/argus/matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
LastTime Dur SrcId SrcAddr
DstAddr Proto TotBytes
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
1368144006.774 86377.883 69.113.13.218 10.1.1.0
0.0.0.0 ip 4609938798
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130511/747f9371/attachment.html>
More information about the argus
mailing list