using database support to track IP addresses

[Carter Bullard]

Gentle people,
Just a quick note to demonstrate how to keep a list of all the IP  
addresses that
visit your host, network, enterprise, whatever (based on argus probe  
placement).

Having a database of IP addresses is very useful for Network Forensics  
and Response.
Easy to get an answer to the question "Have I ever seen this IP  
address before?".
Read entries out of a database like this to get the start and last  
times, and
then go look at your primitive logs to see what was up.

I have many probes, and so I like to keep the IP addresses with their  
source id and
the ethernet address where they came from (assuming you collect mac  
layer
information in your argus records).

I use rasqlinsert() running as a daemon, persistently connected to,  
say, my
exterior border argus().  This simple example will generate a database  
that
holds every IP address that ever touched my network.  With this simple
configuration, there are no entry timeouts, so it keeps them all.   
Other options
can time them out, switch tables based on timed intervals, etc.....

You can use this table to ask, "have I ever seen this IP address?",  
and get a
response very quickly, with  times the address has been active, its  
total
pkts, and bytes,  rate, load, which probes saw it, etc....


    rasqlinsert -S argus.src -w mysql://user:pass@localhost/data/ 
ipAddrs \\
        -M rmon cache -m srcid smac saddr  \\
        -s stime dur srcid smac saddr spkts dpkts sbytes dbytes - ip


Thats it.  You don't have to have the specific "data" database in mysql,
  if its not there, rasqlinsert() will create it.  Same goes for the  
ipAddrs
table that it will also create if it needs to.  (this means you have  
to have
CREATE capabilities in mysql() for this strategy to work.  If you don't
want to give this user these privileges, then create the database and
table for the user (and use rasqlinsert() to do it).

This is what its doing:

    "-S argus.src"
       attach to a remote argus data stream source

    "-w mysql://user:pass@localhost/data/ipAddrs"
       write to a mysql database named "data" that is on localhost  
(you can have it write anywhere
       you are configured to write, so put a hostname or IP address if  
you are writing to a remote
       database), using user with password.  If the account doesn't  
have a password, just omit it
       from the option (user at ....).  These values can be put in  
the .rarc file if you like.  If you have
       the user and the host in the .rarc, then the option is  
"mysql:///data/ipAddrs"

    "-M rmon cache"
       this generates data for each single IP address (rmon) and use  
the database as a cache.
       the cache part is really interesting in that, if you kill the  
program and restart it, you just
       pick up where you left off, without any loss of data.

    "-m srcid smac saddr"
       This modifies the flow key so that the database use the source  
id (srcid) the source mac
       address (smac) and the source IP address (saddr) as the  
database primary key.  This
       creates a database where  you can get the ethernet addresses  
that a particular argus
       "sees", the IP addresses, and you get the binding of ethernet  
to IP, so you can keep up
       with which ethernet addresses are routers, which are hosts  
etc......

       This line is REALLY IMPORTANT, as it defines the schema for the  
database table
       that rasqlinsert() will create, if it needs to, and use.  Be  
sure and use the same
       schema everytime you run the program, or mysql() will complain  
if the table exists
       and it doesn't have the same schema.

    "-s stime dur srcid smac saddr spkts dpkts sbytes dbytes"
       this specifies what attributes will be in the database.
       Put whatever you like in here, but you don't need to put much,  
as we
       insert a binary argus record in each row, incase you want to  
get other
       data, like user data, ttl, ip_id if they didn't get aggregated  
out.
       You have to have the key fields in this list or rasqlinsert()  
will complain.

    "- ip"
      this filters input records so that you only work with IP  
addresses extracted from IP
      flow records.  We can extract IP addrs from things like arp  
flows, and we don't
      necessarily want those addresses in this specific database table.

rasqlinsert() has a lot of knobs so you can control how many entries  
to keep, how
long to keep them, how often it updates entries in the database  
etc....  The default
values should be fine as a starting point.

You can access the data use mysql(), perl whatever.  I use rasql() and  
rasqlinsert().

If you use this command:
    rasqlinsert -r mysql://user:pass@localhost/data/ipAddrs

you will get a realtime curses window that looks and acts just like  
ratop().  this will allow
you to sort, filter, search etc....... and keep a real-time screen  
up.  You can change
the screen refresh rate, which basically controls who often you poll  
the database
for data, with commands like ":u" and change the value that pops up.

If you use rasql(), this program extracts the argus records that are  
in the rows of
interest.  If you want to get the database to select only specific  
records, use the
'-M sql="where clause"' option.   Something like this works very well:

    rasql -r mysql://user:pass@localhost/data/ipAddrs -M sql="smac=\"  
0:1d:b5:bf:6f:c5\""  \
       -s stime dur srcid smac spkts dpkts

If you use mysql() to read the rows, don't use "SELECT *" like  
commands, as it will
try to write out the binary blob that holds the argus record, and that  
looks weird.
specify the columns you specifically like.  If you don't want to have  
argus records in
the database, run rasqlinsert() with the "-M norec" option.


Hope this is useful.  Send suggestions/opinions/reactions/whatever to  
the email list!!!!

Carter




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090521/090b7b7b/attachment.bin>