new database support - clients on server
Carter Bullard
carter at qosient.com
Tue Mar 31 17:53:51 EDT 2009
Gentle people,
I have refreshed the argus-clients-3.0.4.beta.4.tar.gz code on the dev
server.
This fixes a memory problem in processing a lot of files at once,
where some
systems were getting 'can't alloc memory'. This was due to poor memory
management with parsing buffers after files were done. This has been
fixed.
ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.2.beta.4.tar.gz
New features include 'strftime()' specification of mysql database
table names,
so that you can have persistent mysql database daemons that load up data
into, say, daily tables. I run this program as a daemon and it tracks
every
IP address that I see throughout my networks on a daily basis. Very
handy:
rasqlinsert -S data.stream -w mysql://root@localhost/ratop/
ratopFlows_%Y_%m_%d \
-F rasqlinsert.rarc -d -M rmon -m srcid smac saddr - ip
The data.stream is the output of a radium() that connects to all the
probes at QoSient
World headquarters. I have 8 now, but they change from 30 or more to
only 5.
The rasqlinsert.rarc file specifies the RA_TIMEOUT_VALUE, which I set
to 86400,
because I want daily tables, so I don't want the entries to timeout
until a full day
has passed. The file also specifies a RA_UPDATE_VALUE, which I set to
1 second.
This specifies how often data will be flushed to the database.
So, the contents of rasqlinsert.rarc is:
RA_TIMEOUT_INTERVAL=86400
RA_UPDATE_INTERVAL=1.0
The table will grow as new IP addresses come in, and every day at
midnight, a
new table is generated with the new date. Data is written as activity
demands,
and if an entry is not idle, you get accumulated totals in each table.
You use rasql() to read the daily tables:
rasql -r mysql://root@localhost/ratop/ratopFlows_2009_03_31 \
-s stime dur srcid smac saddr spkts dpkts sbytes dbytes state
and you'll get all the IP addresses, the ethernet addresses they are
behind, and
the probes where they were seen. You can pipe this info into
rasort() for instance
to get a topN like stat:
rasql -r mysql://root@localhost/ratop/ratopFlows_2009_03_31 -w -
| \
rasort -m pkts -w - | ra -N 20 -s stime dur srcid smac saddr
spkts dpkts sbytes dbytes state
This will give you the top 20 IP addresses. Put a filter anywhere
along the pipeline
to get a specific top 20, like local addresses, remote addresses,
etc....
To get statistics per ethernet address. This is a bit tricky, as
racluster(), by default, wants
to correct the records direction, and so you have to tell it not to
"fix" the records.
rasql -r mysql://root@localhost/ratop/ratopFlows_2009_03_31 -w -
| \
racluster -M nocorrect -m smac -s stime dur smac spkts dpkts
sbytes dbytes state
Here is a variation that is very helpful for me. The idea is to get
the number
of IP addresses behind specific ethernet addresses, so you can tell what
is a router, and what is not.
Use the "-M dsrs='-agr'" option to reset the per flow aggregation
counters, so that
when racluster() merges the records, the agr stats will reflect the
number of IPs.
rasql -M dsrs="-agr" -r mysql://root@localhost/ratop/
ratopFlows_2009_03_31 -w - | \
racluster -M nocorrect -m srcid smac -s stime dur smac spkts
dpkts sbytes dbytes
StartTime Dur SrcId
SrcMac Trans SrcPkts DstPkts
2009/03/31 16:30:50.734864 3714.24438 192.168.0.1 0:9:5b:
36:a:33 1 435 434
2009/03/31 16:38:42.422914 3120.16967 192.168.0.1 0:b:db:
5c:e5:7c 1 3110 2120
2009/03/31 16:30:58.678280 3530.90747 192.168.0.1 0:12:3f:bc:
58:a4 1 12 10
2009/03/31 16:50:21.456184 1942.19873 192.168.0.1 0:12:3f:bc:
64:59 1 37 30
2009/03/31 16:32:08.858857 3601.39575 192.168.0.1 0:16:cb:ad:
90:11 2 879 731
2009/03/31 16:30:50.734864 3715.22729 192.168.0.1 0:1d:b5:bf:
6f:c5 285 45467 30658
2009/03/31 17:07:23.880961 0.000000 192.168.0.1 0:21:5a:
39:d7:a2 1 1 0
2009/03/31 16:30:58.238273 3707.72387 192.168.0.1
0:23:32:2f:ac:9c 2 21335 33655
2009/03/31 16:30:51.626989 3710.23388 192.168.0.1 0:23:6c:7f:
6b:5 2 5175 8487
2009/03/31 17:07:23.737794 922.218384 192.168.0.1 1:0:5e:
0:0:fb 1 0 19
2009/03/31 16:38:30.320967 3098.85253 192.168.0.1 1:0:5e:
7f:ff:fa 1 0 36
2009/03/31 17:07:23.737799 922.218384 192.168.0.1
33:33:0:0:0:fb 1 0 18
2009/03/31 16:30:58.238273 3689.16552 192.168.0.1
ff:ff:ff:ff:ff:ff 2 0 253
2009/03/31 16:30:50.182576 3718.12548 192.168.0.68 0:b:db:
59:14:93 1 86947 85273
2009/03/31 17:05:41.727871 358.660889 192.168.0.68 0:b:db:
5c:e5:7c 1 706 1577
2009/03/31 16:30:52.934368 3683.00903 192.168.0.68 0:16:cb:ad:
90:11 2 1771 1722
2009/03/31 16:32:22.071646 3623.89575 192.168.0.68 0:1d:b5:bf:
6f:c5 76 33655 21135
2009/03/31 17:07:23.884802 0.000000 192.168.0.68 0:21:5a:
39:d7:a2 1 1 0
2009/03/31 16:30:50.182576 3718.12548 192.168.0.68
0:23:32:2f:ac:9c 2 110460 123526
2009/03/31 16:33:29.774269 3361.38623 192.168.0.68 0:23:6c:7f:
6b:5 2 572 553
2009/03/31 17:07:23.741546 922.218445 192.168.0.68 1:0:5e:
0:0:fb 1 0 19
2009/03/31 16:38:30.324729 3098.85400 192.168.0.68 1:0:5e:
7f:ff:fa 1 0 36
2009/03/31 17:07:23.741583 922.218445 192.168.0.68
33:33:0:0:0:fb 1 0 18
2009/03/31 16:30:58.241872 3689.16699 192.168.0.68
ff:ff:ff:ff:ff:ff 2 0 253
2009/03/31 16:30:49.978306 3718.12451 192.168.0.70 0:b:db:
59:14:93 1 26214 26877
2009/03/31 16:30:49.978306 3718.12451 192.168.0.70 0:12:3f:bc:
58:a4 1 26889 26224
2009/03/31 16:32:08.858901 3601.39575 192.168.0.70 0:16:cb:ad:
90:11 2 85 0
2009/03/31 16:30:58.678131 3530.90771 192.168.0.70 0:1d:b5:bf:
6f:c5 5 10 12
2009/03/31 17:07:23.880982 0.000000 192.168.0.70 0:21:5a:
39:d7:a2 1 1 0
2009/03/31 16:30:58.238191 3689.16552 192.168.0.70
0:23:32:2f:ac:9c 2 200 0
2009/03/31 16:33:29.770430 3361.38476 192.168.0.70 0:23:6c:7f:
6b:5 2 40 0
2009/03/31 17:07:23.737767 922.218262 192.168.0.70 1:0:5e:
0:0:fb 1 0 19
2009/03/31 16:38:30.321023 3098.85253 192.168.0.70 1:0:5e:
7f:ff:fa 1 0 36
2009/03/31 17:07:23.737774 922.218445 192.168.0.70
33:33:0:0:0:fb 1 0 18
2009/03/31 16:30:58.238191 3689.16552 192.168.0.70
ff:ff:ff:ff:ff:ff 2 0 253
2009/03/31 16:30:50.742664 3714.23999 192.168.2.65 0:9:5b:
36:a:32 4 539 402
2009/03/31 16:30:50.742664 3714.23999 192.168.2.65
0:11:d9:15:8a:47 1 584 420
2009/03/31 16:30:58.241904 3689.16699 192.168.2.65 0:1d:
4f:ff:a6:ad 2 144 0
2009/03/31 16:34:02.477332 3372.38452 192.168.2.65 1:0:5e:
0:0:fb 1 0 78
2009/03/31 17:07:23.741726 922.218445 192.168.2.65
33:33:0:0:0:fb 1 0 6
2009/03/31 16:30:55.730294 3692.76831 192.168.2.65
ff:ff:ff:ff:ff:ff 2 0 361
2009/03/31 16:30:50.734692 3715.22753 207.237.36.98
0:11:5d:c0:d4:1 377 47307 31367
2009/03/31 16:30:50.734692 3715.22753 207.237.36.98 0:1d:b5:bf:
6f:c0 1 31367 47209
2009/03/31 16:31:08.268467 3674.41186 207.237.36.98
ff:ff:ff:ff:ff:ff 1 0 98
So, what have we got, there are 5 probes, each have a set of routers,
some multicast
ethernet addresses, and some of the ethernets have both IPv4 and IPv6
addresses.
Many of the ethernet addresses are seen by multiple probes, so
0:1d:b5:bf:6f:c5
for example, is seen by probe 192.168.0.1 (which is the internal
router interface for the company),
192.168.0.68 and 192.168.0.70, which are both hosts on the same LAN as
192.168.0.1.
207.237.36.98
Hope this is helpful, and be sure and grab the software for testing!!!!!
Thanks for all the help!!!!
Carter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090331/7623e820/attachment.html>
More information about the argus
mailing list