Request for improvement
Carter Bullard
carter at qosient.com
Mon Oct 22 16:22:50 EDT 2012
Hey Paul,
> In testing the -t switch I found it to be very fast in searching for the records, but there was something interesting about it. Once it had found all the records, ra didn't exit. It continued to search.
Using argus-clients-3.0.6.x or argus-clients-3.0.7.x, " ra -t time-time " will sequentially search until it
reaches EOF. This is by design, and allows ra() to process files that are not sorted.
If you would like to use the fastest time based searches, you should use rasqltimeindex() to index
your files based on seconds, and then use rasql() to find the records. See the manpage for rasqltimeindex.
Once you run rasqltimeindex(), you will find a " Seconds " table in your database, with this schema:
mysql> desc Seconds;
+-----------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+-------+
| probe | int(10) unsigned | NO | | NULL | |
| second | int(10) unsigned | NO | MUL | NULL | |
| fileindex | int(11) | NO | | NULL | |
| ostart | int(10) unsigned | NO | | NULL | |
| ostop | int(10) unsigned | NO | | NULL | |
+-----------+------------------+------+-----+---------+-------+
5 rows in set (0.00 sec)
This generates a database table of the byte offsets for argus data that contains
each second in the file. When you make a query with rasql(), using a time filter,
it will look for a Seconds table in the database you specify (or the one in your
~/.rarc file), and then it will read all the data from all the files that match.
I run this on all my data using rastream(), and I can resolve any short time based query
in less that 0.2 seconds covering data going back 3 years. Printing data may take
some time, but finding out where the data is and its byte offsets is very fast.
thoth:ramysql carter$ time rasql -t -157d+2m -q
real 0m0.067s
user 0m0.035s
sys 0m0.011s
thoth:ramysql carter$ time rasql -t -4014h+2m -q
real 0m0.076s
user 0m0.042s
sys 0m0.012s
Here is some example debug output from one of my queries. I'm using the " -q " option
to supress printing out of the argus records.
thoth:ramysql carter$ rasql -t 2012/04/23.11:40:28+5s -D2 -q
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.881297 ArgusInitAddrtoname (0x1748000, 0x0, 0x0)
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.881360 ArgusParseInit(0x1748000, NULL)
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.905893 RaMySQLInit () RaSource (null) RaArchive (null) RaFormat (null)
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.905967 SQL Query SELECT * from Seconds WHERE second >= 1335195628 and second <= 1335195633
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906783 ArgusAddFileList (0x1748000, //Volumes/Data/Archive/QoSient/192.168.0.1/2012/04/23/argus.2012.04.23.11.40.00, 1, 36288, 99904) returning 1
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906803 RaSQLProcessQueue: filename //Volumes/Data/Archive/QoSient/192.168.0.1/2012/04/23/argus.2012.04.23.11.40.00 ostart 36288 ostop 99904
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906828 ArgusAddFileList (0x1748000, //Volumes/Data/Archive/QoSient/207.237.36.98/2012/04/23/argus.2012.04.23.11.40.00, 1, 77588, 155788) returning 1
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906844 RaSQLProcessQueue: filename //Volumes/Data/Archive/QoSient/207.237.36.98/2012/04/23/argus.2012.04.23.11.40.00 ostart 77588 ostop 155788
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906867 ArgusAddFileList (0x1748000, //Volumes/Data/Archive/QoSient/192.168.0.70/2012/04/23/argus.2012.04.23.11.40.00, 1, 13508, 15412) returning 1
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906882 RaSQLProcessQueue: filename //Volumes/Data/Archive/QoSient/192.168.0.70/2012/04/23/argus.2012.04.23.11.40.00 ostart 13508 ostop 15412
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906895 RaSQLProcessQueue(0x69d49160)
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906969 ArgusReadConnection() read 16 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.906986 ArgusReadConnection() read 112 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.907019 ArgusParseInit(0x1748000 0x19ee000
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.908727 main: ArgusReadFileStream (//Volumes/Data/Archive/QoSient/192.168.0.1/2012/04/23/argus.2012.04.23.11.40.00) done
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.908764 ArgusReadConnection() read 16 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.908777 ArgusReadConnection() read 112 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.912791 ArgusParseInit(0x1748000 0x1a50000
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.914826 main: ArgusReadFileStream (//Volumes/Data/Archive/QoSient/207.237.36.98/2012/04/23/argus.2012.04.23.11.40.00) done
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.914888 ArgusReadConnection() read 16 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.914903 ArgusReadConnection() read 112 bytes
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.915577 ArgusParseInit(0x1748000 0x1ab2000
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.915698 main: ArgusReadFileStream (//Volumes/Data/Archive/QoSient/192.168.0.70/2012/04/23/argus.2012.04.23.11.40.00) done
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.915718 main: reading files completed
rasql[6655.8041bc78ff7f0000]: 2012/10/22.16:11:39.915735 ArgusShutDown (0)
Take a look at rasqltimeindex().
Carter
On Oct 16, 2012, at 11:03 AM, Paul Schmehl <pschmehl_lists at tx.rr.com> wrote:
> I'm working on a project the design of which is to insert datetime, srcip, dstip, srcport,dstport, sbytes,dbytes,state into a db.
>
> We'll right a front end for this in php (probably) that allows us to answer questions like when was the first time this IP talked to a host on our network and what host was it.
>
> Since the data are very small, searches should be very fast.
>
> Along with that, we want to design it so that once you've located that initial conversation, you can pull up the packets from the argus files (we won't be storing payloads in the db) and view the actual payload so you can figure out what a host is up to.
>
> So, we can take the info in the db and do something like this:
>
> ra -t yyyy/mm/dd.HH:MM:SS -r /path/to/argus/file - host x.x.x.x and return the payloads for that host for that second (or time range)
>
> In testing the -t switch I found it to be very fast in searching for the records, but there was something interesting about it. Once it had found all the records, ra didn't exit. It continued to search.
>
> ISTM the software ought to be able to do something like this;
>
> search for records with a certain timestamp
> once it has a return, continue searching until 10 sequential records don't match, then exit.
>
> Since timestamps are sequential, there's no point in continuing the search once they're found. There's not going to be any more.
>
> I'm not a programmer, but I imagine this wouldn't be too hard to code up - unless I'm unaware of some quirk that would make it very difficult.
>
> --
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *******************************************
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20121022/463b1091/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20121022/463b1091/attachment.bin>
More information about the argus
mailing list