Argus-info Digest, Vol 42, Issue 1
Carter Bullard
carter at qosient.com
Thu Feb 5 16:47:51 EST 2009
Hey Mark,
I'm going to move this thread to the argus mailing list, so we
all can see/read.
Hmmm, from your email, I'm still wondering what partitioning means.
Is this a formal function in mysql that I would have to configure
at say, database creation or table creation time?
So the idea is that the new db support programs would be able to
do what you are doing using near real-time streams as the data
source (files too, but the stream is the most problematic), so that the
data is available in almost near real-time to multiple clients of
interest.
My thought is that we would create a database for raw data, that
is based on a year, and it contains a table for each day. The program
rasqlinsert() would create the databases and new tables, like rasplit()
does when it creates new directories and files, on demand as data for
a given year/day is received. If you are storing raw flow data, then
we
would just append the data to each table as it comes in, just like
rasplit. If you wanted something a bit more sophisticated, like the
list of active IP addresses, then rasqlinsert() would do the clustering,
using the db table as a backing store. So rasqlinsert() can have a
small memory footprint, and cache data as needed, referring to the
database table, only when there are changes.
With the same argus data stream, we can manage a number of
strategies. We could maintain IP address tables, IP matrix tables,
MAC address tables, port tables, whatever, limited of course on
performance.
At least that is the design. So rasqlinsert() would attach to your
primary argus flow stream, and it would maintain whatever
tables you specified, in near real-time. You would then use
various rasql* programs to read the data, and periodically, you
would have to drop tables, based on your retention strategy.
How does that sound?
Carter
On Feb 5, 2009, at 2:11 PM, Mark Bartlett wrote:
> Hey Carter... I ask about "Partitioning" because I have the ARGUS
> stuff feeding my database which consists of a few tables:
>
> argus table - contains all fields contained in the argus data I am
> collecting (saddr (sip), daddr (dstip), proto (protocol, times, etc)
> proto_tbl table - contains all protocol numbers (just to make viewing
> the data easier on analysts, etc, they see TCP instead of 6) Links to
> Protocol in Argus table.
> probe_tbl table - contains 'probe information' corresponds with the
> ARGUS_MONITOR_ID= set in the argus.conf.. (links to srcid field in
> Argus table) So we can give the "probe" a name and assign
> location,etc..
>
> Also a location table and host table.. Wanted to be able to run
> reports against locations that contain multiple 'probes'... The host
> table was something I wanted to do so I could add information about a
> specific host when investigating, or add dns names, 'role' (email,
> web, proxy, etc), etc... This is a work in progress..
>
> Problem I run into is that my argus table just builds and builds and
> builds... Basically like a flat file that keeps growing in size. So
> my next step was going to be to create a couple of partitioning
> scripts to solve this problem.. My thought was to have 2 databases, 1
> to contain the last week of argus data and the 2nd to contain the last
> day of data.
>
> DB 1 would be 'partitioned' by Day - This DB would be used to create
> Weekly trending reports on Number of events, Bytes, etc.
> DB 2 would be 'partitioned' by HOUR - This DB would be used for the
> 'current' day of argus data, for more real time queries..
>
> By partitioning the database (s) I will see a huge performance boost
> in my query times.. Currently it takes about 10 minutes to come back
> on a simple query against my db with 8 million rows... That's def.
> unacceptable in our environment where we are trying to use the ARGUS
> data to validate 'traffic' from a host after/before a SNORT alert it
> triggered.
>
> Below is my db script. As you can see we are NOT storing all fields
> from the argus data.
>
> CREATE TABLE IF NOT EXISTS argus (
> srcid varchar(80) NOT NULL default '',
> stime time NOT NULL default '00:00:00',
> sdate date NOT NULL default '2006-06-08',
> ltime time NOT NULL default '00:00:00',
> ldate date NOT NULL default '2006-06-08',
> dur dec(30,6) unsigned NOT NULL default '0',
> saddr varchar(15) NOT NULL default '',
> daddr varchar(15) NOT NULL default '',
> proto tinyint(3) unsigned NOT NULL default '0',
> sport smallint(5) unsigned NOT NULL default '0',
> dport smallint(5) unsigned NOT NULL default '0',
> bytes int(10) unsigned NOT NULL default '0',
> sbytes int(10) unsigned NOT NULL default '0',
> dbytes int(10) unsigned NOT NULL default '0',
> pkts int(10) unsigned NOT NULL default '0',
> spkts int(10) unsigned NOT NULL default '0',
> dpkts int(10) unsigned NOT NULL default '0',
> dir varchar(15) NOT NULL default '',
> trans int(10) unsigned NOT NULL default '0',
> seq int(10) unsigned NOT NULL default '0',
> flags varchar(15) NOT NULL default ''
> );
>
>
>
> /*This will create the proto_tbl. The proto_tbl contains the 'real'
> names of the Protocols. argus has a field within the argus table
> called protocol which contains the numeric value for the protocol.
> This table contains the actual protocol names. This table is used to
> display the protocol names instead of the numbers.*/
> CREATE TABLE IF NOT EXISTS proto_tbl (
> proto_id smallint(3) unsigned NOT NULL default '0',
> proto_name varchar(25) NOT NULL default '',
> proto_long_name varchar(40) NOT NULL default '',
> proto_reference varchar(30) NOT NULL default '',
> PRIMARY KEY (proto_id)
> );
>
>
> /*This will create the probe_tbl. The probe_tbl contains additional
> information for probes used in the environment. In the argus.conf
> file we are using the ARGUS_MONITOR_ID option to specify a probe
> label.*/
> CREATE TABLE IF NOT EXISTS `probe_tbl` (
> probe_id smallint(4) NOT NULL AUTO_INCREMENT,
> probe_label varchar(80) NOT NULL default '',
> probe_position int(1) NOT NULL default '0',
> probename_g varchar(80) NOT NULL default '',
> location_id int(6) unsigned NOT NULL default '0',
> company_id int(6) NOT NULL default '-',
> PRIMARY KEY (probe_id)
> );
>
>
>
>
> On Thu, Feb 5, 2009 at 1:42 PM, Carter Bullard <carter at qosient.com>
> wrote:
>> Hey Mark,
>> So lets move this db question to the mailing list, so that all can
>> see. I, not being a database person, am not sure what partioning
>> means.
>> Carter
>>
>> On Feb 5, 2009, at 1:32 PM, Mark Bartlett wrote:
>>
>>> Also created a web front end to query the ARGUS db... Screen shot
>>> attached.
>>>
>>> Hopefully I can use your DB and just modify my web stuff...
>>>
>>> Carter, does your DB have partitioning built in?? I KNOW that ARGUS
>>> data can fill a DB rather quickly. We are monitoring one segment of
>>> our network and see 130K/hr and 2mil/day..
>>>
>>> Cheers
>>> mark
>>>
>>> On Thu, Feb 5, 2009 at 1:27 PM, Mark Bartlett <mabartle at gmail.com>
>>> wrote:
>>>>
>>>> Hey Carter..
>>>> I'll def. be a user of the mysql support... I'm currently doing it
>>>> with the mysqlimport command after I generate a flat file with
>>>> your ra
>>>> tool....
>>>>
>>>> Mysql Functionality would be AWESOME!
>>>>
>>>> mark
>>>>
>>>> On Thu, Feb 5, 2009 at 11:03 AM, CS Lee <geek00l at gmail.com> wrote:
>>>>>
>>>>> hi Carter,
>>>>>
>>>>> I grab your control plane presentation from here -
>>>>>
>>>>> https://tools.netsa.cert.org/wiki/display/tt/FloCon+2009
>>>>>
>>>>> I'm definitely a user, can't guarantee to be customer unless my
>>>>> boss
>>>>> says
>>>>> yes ;]
>>>>>
>>>>> The mysql support is definitely a long wait ;]
>>>>>
>>>>> On Thu, Feb 5, 2009 at 11:51 PM, Carter Bullard <carter at qosient.com
>>>>> >
>>>>> wrote:
>>>>>>
>>>>>> Thanks, I don't think I gave them slides notes, .... didn't
>>>>>> know if it
>>>>>> was intelligible without some form of narrative.
>>>>>> So I have a bunch of mysql database support for argus almost
>>>>>> ready
>>>>>> for public consumption, would you be a potential customer/user?
>>>>>> I use it primarily to hold real-time "ratop" like state so
>>>>>> multiple
>>>>>> people/programs can work off of the data at the same time, but
>>>>>> some people have had good luck using it to hold flow data and
>>>>>> to keep track of say all the IP addresses they've seen since the
>>>>>> beginning, in real-time.
>>>>>> Carter
>>>>>>
>>>>>> On Feb 4, 2009, at 9:11 PM, CS Lee wrote:
>>>>>>
>>>>>> hi Carter,
>>>>>>
>>>>>> Thanks. Things are all good, btw, I read your presentation in
>>>>>> Flocon2009,
>>>>>> that's really cool stuff.
>>>>>>
>>>>>> On Thu, Feb 5, 2009 at 3:21 AM, Carter Bullard <carter at qosient.com
>>>>>> >
>>>>>> wrote:
>>>>>>>
>>>>>>> Hmmmm,
>>>>>>> I have them in the development directory, here. Let me see
>>>>>>> what's up.
>>>>>>> If we can fix the ongoing v2 -> v3 conversion problem soon, I'll
>>>>>>> release
>>>>>>> another round of client programs, and it will definitely have
>>>>>>> them in
>>>>>>> there.
>>>>>>> How are things?
>>>>>>> Carter
>>>>>>> On Feb 4, 2009, at 1:22 PM, CS Lee wrote:
>>>>>>>
>>>>>>> hi carter,
>>>>>>>
>>>>>>> i can't find both rauserdata() and raservice() in argus 3.0
>>>>>>> distribution
>>>>>>> which I downloaded from
>>>>>>>
>>>>>>> ftp://qosient.com/dev/argus-3.0/
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 3, 2009 at 11:46 PM,
>>>>>>> <argus-info-request at lists.andrew.cmu.edu> wrote:
>>>>>>>>
>>>>>>>> Send Argus-info mailing list submissions to
>>>>>>>> argus-info at lists.andrew.cmu.edu
>>>>>>>>
>>>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>>>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>>> argus-info-request at lists.andrew.cmu.edu
>>>>>>>>
>>>>>>>> You can reach the person managing the list at
>>>>>>>> argus-info-owner at lists.andrew.cmu.edu
>>>>>>>>
>>>>>>>> When replying, please edit your Subject line so it is more
>>>>>>>> specific
>>>>>>>> than "Re: Contents of Argus-info digest..."
>>>>>>>>
>>>>>>>>
>>>>>>>> Today's Topics:
>>>>>>>>
>>>>>>>> 1. ArgusGenerateRecord: packet size type not defined
>>>>>>>> (Michael Grinnell)
>>>>>>>> 2. argus usage (Oguz Yarimtepe)
>>>>>>>> 3. Re: argus usage (Stewart Gray)
>>>>>>>> 4. Re: ArgusGenerateRecord: packet size type not defined
>>>>>>>> (Peter Van Epp)
>>>>>>>> 5. Re: argus usage (Peter Van Epp)
>>>>>>>> 6. Re: argus usage (Oguz Yarimtepe)
>>>>>>>> 7. Re: ArgusGenerateRecord: packet size type not defined
>>>>>>>> (Carter Bullard)
>>>>>>>> 8. argus user data buffer analysis (Carter Bullard)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>> Message: 1
>>>>>>>> Date: Mon, 02 Feb 2009 13:39:44 -0500
>>>>>>>> From: Michael Grinnell <grinnell at american.edu>
>>>>>>>> Subject: [ARGUS] ArgusGenerateRecord: packet size type not
>>>>>>>> defined
>>>>>>>> To: argus-info <argus-info at lists.andrew.cmu.edu>
>>>>>>>> Message-ID: <49873DF0.6090305 at american.edu>
>>>>>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Periodically Argus dies on my test system with the error
>>>>>>>> "ArgusGenerateRecord: packet size type not defined." The time
>>>>>>>> between
>>>>>>>> these errors varies, sometimes it's only a minute or two
>>>>>>>> after argus
>>>>>>>> starts, other times it can be > 15 minutes. I've tried
>>>>>>>> running a
>>>>>>>> simultaneous tcpdump, then running the resulting capture file
>>>>>>>> through
>>>>>>>> argus, but I can't replicate the error. I also don't see any
>>>>>>>> glaring
>>>>>>>> errors in the capture file around the time it dies. This
>>>>>>>> happens
>>>>>>>> with
>>>>>>>> argus 3.0.0 and with argus 3.0.1 beta2. The system is
>>>>>>>> running CentOS
>>>>>>>> 5.2 and is listening on a dedicated interface (NC7782, bnx2
>>>>>>>> driver)
>>>>>>>> to a
>>>>>>>> span port off of a Cisco switch. I have also updated to the
>>>>>>>> newest
>>>>>>>> bnx2
>>>>>>>> drivers, but it still recurs. I'm trying to scare up another
>>>>>>>> NIC to
>>>>>>>> try
>>>>>>>> as well.
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Michael Grinnell
>>>>>>>> Information Security Engineer
>>>>>>>> The American University
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 2
>>>>>>>> Date: Mon, 02 Feb 2009 22:50:20 +0200
>>>>>>>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>>>>>>>> Subject: [ARGUS] argus usage
>>>>>>>> To: argus-info at lists.andrew.cmu.edu
>>>>>>>> Message-ID: <1233607820.7050.12.camel at gezenti>
>>>>>>>> Content-Type: text/plain
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I want some information about argus capabilities. I tried
>>>>>>>> argus a bit
>>>>>>>> and read the wiki. I want to learn whether it is possible to
>>>>>>>> get
>>>>>>>> application level information from a flow record by using
>>>>>>>> argus, like
>>>>>>>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>>>>>>> As i understood from its usage it is possible to get these
>>>>>>>> information
>>>>>>>> indirectly by analyzing the argus output but maybe there is a
>>>>>>>> way
>>>>>>>> that
>>>>>>>> argus serves that i don't know.
>>>>>>>>
>>>>>>>> Thanx.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 3
>>>>>>>> Date: Tue, 3 Feb 2009 10:26:06 +1300
>>>>>>>> From: "Stewart Gray" <Stewart.Gray at safecom.co.nz>
>>>>>>>> Subject: Re: [ARGUS] argus usage
>>>>>>>> To: <argus-info at lists.andrew.cmu.edu>
>>>>>>>> Message-ID:
>>>>>>>>
>>>>>>>>
>>>>>>>> <4C08B31BFF3B5D42BCAC255ADDD6EE5EA8769D at tasbe116.advancedsolutions.co.nz
>>>>>>>> >
>>>>>>>>
>>>>>>>> Content-Type: text/plain; charset="us-ascii"
>>>>>>>>
>>>>>>>> Hi Oguz,
>>>>>>>>
>>>>>>>> Argus supports the Berkeley packet filter (as other apps like
>>>>>>>> tcpdump
>>>>>>>> and ngrep do) so you're able to lock queries down to port
>>>>>>>> numbers,
>>>>>>>> specific hosts etc. Knowing the port number an application
>>>>>>>> runs on
>>>>>>>> means
>>>>>>>> it fairly easy to pull out transactions for a particular
>>>>>>>> protocol.
>>>>>>>> Keep
>>>>>>>> in mind it doesn't operate at layer 7, flow information for a
>>>>>>>> protocol
>>>>>>>> such as HTTP will be very similar to SMTP. If you're after more
>>>>>>>> in-depth
>>>>>>>> analysis of a transaction, i.e. what happened during an http
>>>>>>>> connection
>>>>>>>> (GET, POST, HEAD etc) you're probably best to be taking full
>>>>>>>> packet
>>>>>>>> captures alongside argus so you can run them through another
>>>>>>>> tool
>>>>>>>> such
>>>>>>>> as wireshark.
>>>>>>>>
>>>>>>>> Have a look at this wiki page for some common usages -
>>>>>>>> http://nsmwiki.org/index.php?title=Argus.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Stewart
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From:
>>>>>>>> argus-info-bounces+stewart.gray=safecom.co.nz at lists.andrew.cmu.edu
>>>>>>>>
>>>>>>>> [mailto:argus-info-bounces+stewart.gray=safecom.co.nz at lists.andrew.cmu.e
>>>>>>>> du] On Behalf Of Oguz Yarimtepe
>>>>>>>> Sent: Tuesday, 3 February 2009 9:50 a.m.
>>>>>>>> To: argus-info at lists.andrew.cmu.edu
>>>>>>>> Subject: [ARGUS] argus usage
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I want some information about argus capabilities. I tried
>>>>>>>> argus a bit
>>>>>>>> and read the wiki. I want to learn whether it is possible to
>>>>>>>> get
>>>>>>>> application level information from a flow record by using
>>>>>>>> argus, like
>>>>>>>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>>>>>>> As i understood from its usage it is possible to get these
>>>>>>>> information
>>>>>>>> indirectly by analyzing the argus output but maybe there is a
>>>>>>>> way
>>>>>>>> that
>>>>>>>> argus serves that i don't know.
>>>>>>>>
>>>>>>>> Thanx.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #####################################################################################
>>>>>>>> Important: This electronic message and attachments (if any) are
>>>>>>>> confidential
>>>>>>>> and may be legally privileged. If you are not the intended
>>>>>>>> recipient
>>>>>>>> do
>>>>>>>> not
>>>>>>>> copy, disclose or use the contents in any way. Please let us
>>>>>>>> know by
>>>>>>>> return
>>>>>>>> e-mail immediately and then destroy this message.
>>>>>>>>
>>>>>>>>
>>>>>>>> #####################################################################################
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 4
>>>>>>>> Date: Mon, 2 Feb 2009 20:51:05 -0800
>>>>>>>> From: Peter Van Epp <vanepp at sfu.ca>
>>>>>>>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not
>>>>>>>> defined
>>>>>>>> To: argus-info at lists.andrew.cmu.edu
>>>>>>>> Message-ID: <20090203045105.GA26496 at sfu.ca>
>>>>>>>> Content-Type: text/plain; charset=us-ascii
>>>>>>>>
>>>>>>>> On Mon, Feb 02, 2009 at 01:39:44PM -0500, Michael Grinnell
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Periodically Argus dies on my test system with the error
>>>>>>>>> "ArgusGenerateRecord: packet size type not defined." The time
>>>>>>>>> between
>>>>>>>>> these errors varies, sometimes it's only a minute or two
>>>>>>>>> after argus
>>>>>>>>> starts, other times it can be > 15 minutes. I've tried
>>>>>>>>> running a
>>>>>>>>> simultaneous tcpdump, then running the resulting capture file
>>>>>>>>> through
>>>>>>>>> argus, but I can't replicate the error. I also don't see any
>>>>>>>>> glaring
>>>>>>>>> errors in the capture file around the time it dies. This
>>>>>>>>> happens
>>>>>>>>> with
>>>>>>>>> argus 3.0.0 and with argus 3.0.1 beta2. The system is running
>>>>>>>>> CentOS
>>>>>>>>> 5.2 and is listening on a dedicated interface (NC7782, bnx2
>>>>>>>>> driver)
>>>>>>>>> to
>>>>>>>>> a
>>>>>>>>> span port off of a Cisco switch. I have also updated to the
>>>>>>>>> newest
>>>>>>>>> bnx2
>>>>>>>>> drivers, but it still recurs. I'm trying to scare up
>>>>>>>>> another NIC to
>>>>>>>>> try
>>>>>>>>> as well.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Michael Grinnell
>>>>>>>>> Information Security Engineer
>>>>>>>>> The American University
>>>>>>>>
>>>>>>>> Setting
>>>>>>>>
>>>>>>>> ARGUS_PACKET_CAPTURE_FILE="/var/log/argus/packet.out"
>>>>>>>>
>>>>>>>> in an argus.rc file will capture the input packets from pcap
>>>>>>>> in to
>>>>>>>> the
>>>>>>>> specified file. If you can get lucky and get a failure before
>>>>>>>> you run
>>>>>>>> out of
>>>>>>>> disk space one of the last packets in the file should tell us
>>>>>>>> what
>>>>>>>> argus
>>>>>>>> isn't
>>>>>>>> liking. On a busy link this file will get large fast but if it
>>>>>>>> sometimes
>>>>>>>> fails quickly you may be lucky (you are also likely to see
>>>>>>>> packet
>>>>>>>> loss
>>>>>>>> due to
>>>>>>>> the disk I/O on the sensor but hopefully the fault will still
>>>>>>>> occur).
>>>>>>>> It looks like the argus record is malformed (it is
>>>>>>>> complaining
>>>>>>>> that it
>>>>>>>> doesn't recognize the type in argus/ArgusModeler.c at line
>>>>>>>> 2904 in
>>>>>>>> the
>>>>>>>> argus-3.0.1.beta.2 code). A dump of the offending packet
>>>>>>>> should tell
>>>>>>>> Carter
>>>>>>>> why (or if the incoming packet is corrupted which is also
>>>>>>>> possible).
>>>>>>>>
>>>>>>>> Peter Van Epp
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 5
>>>>>>>> Date: Mon, 2 Feb 2009 21:06:59 -0800
>>>>>>>> From: Peter Van Epp <vanepp at sfu.ca>
>>>>>>>> Subject: Re: [ARGUS] argus usage
>>>>>>>> To: argus-info at lists.andrew.cmu.edu
>>>>>>>> Message-ID: <20090203050659.GA8930 at sfu.ca>
>>>>>>>> Content-Type: text/plain; charset=us-ascii
>>>>>>>>
>>>>>>>> On Mon, Feb 02, 2009 at 10:50:20PM +0200, Oguz Yarimtepe wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I want some information about argus capabilities. I tried
>>>>>>>>> argus a
>>>>>>>>> bit
>>>>>>>>> and read the wiki. I want to learn whether it is possible to
>>>>>>>>> get
>>>>>>>>> application level information from a flow record by using
>>>>>>>>> argus,
>>>>>>>>> like
>>>>>>>>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>>>>>>>> As i understood from its usage it is possible to get these
>>>>>>>>> information
>>>>>>>>> indirectly by analyzing the argus output but maybe there is
>>>>>>>>> a way
>>>>>>>>> that
>>>>>>>>> argus serves that i don't know.
>>>>>>>>>
>>>>>>>>> Thanx.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Depends on what you need. If you enable user data capture
>>>>>>>> (the
>>>>>>>> -U
>>>>>>>> option on the argus) it will capture up to the first 512
>>>>>>>> bytes of the
>>>>>>>> user
>>>>>>>> data of the flow. That may or may not give you enough
>>>>>>>> information
>>>>>>>> about
>>>>>>>> the
>>>>>>>> flow to do what you want. Note that on a fast link best
>>>>>>>> results are
>>>>>>>> going to
>>>>>>>> occur using a DAG card as the data capture adds a fairly
>>>>>>>> heavy load
>>>>>>>> to
>>>>>>>> the
>>>>>>>> server. To display the data with ra (for instance) you need
>>>>>>>> to use
>>>>>>>> the
>>>>>>>> -s
>>>>>>>> command to add suser and duser to the output (as in
>>>>>>>>
>>>>>>>> ra -r argus_file -n -s +suser:512 -s +duser:512
>>>>>>>>
>>>>>>>> which will tack the user data on the end of the line. This of
>>>>>>>> course
>>>>>>>> raises a
>>>>>>>> number of sticky privacy issues that you need to have
>>>>>>>> considered and
>>>>>>>> gotten
>>>>>>>> approved by appropriate management of the link you are
>>>>>>>> tapping (which
>>>>>>>> may or
>>>>>>>> may not be you :-)).
>>>>>>>>
>>>>>>>> Peter Van Epp
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 6
>>>>>>>> Date: Tue, 3 Feb 2009 07:59:33 +0200
>>>>>>>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>>>>>>>> Subject: Re: [ARGUS] argus usage
>>>>>>>> To: argus-info at lists.andrew.cmu.edu
>>>>>>>> Message-ID:
>>>>>>>> <20831c740902022159q176610e5nb954f5d96e211d13 at mail.gmail.com
>>>>>>>> >
>>>>>>>> Content-Type: text/plain; charset="iso-8859-9"
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Depends on what you need. If you enable user data
>>>>>>>>> capture (the
>>>>>>>>> -U
>>>>>>>>> option on the argus) it will capture up to the first 512
>>>>>>>>> bytes of
>>>>>>>>> the
>>>>>>>>> user
>>>>>>>>> data of the flow. That may or may not give you enough
>>>>>>>>> information
>>>>>>>>> about the
>>>>>>>>> flow to do what you want. Note that on a fast link best
>>>>>>>>> results are
>>>>>>>>> going
>>>>>>>>> to
>>>>>>>>> occur using a DAG card as the data capture adds a fairly
>>>>>>>>> heavy load
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> server. To display the data with ra (for instance) you need
>>>>>>>>> to use
>>>>>>>>> the
>>>>>>>>> -s
>>>>>>>>> command to add suser and duser to the output (as in
>>>>>>>>>
>>>>>>>>> ra -r argus_file -n -s +suser:512 -s +duser:512
>>>>>>>>>
>>>>>>>>> which will tack the user data on the end of the line. This
>>>>>>>>> of course
>>>>>>>>> raises
>>>>>>>>> a
>>>>>>>>> number of sticky privacy issues that you need to have
>>>>>>>>> considered and
>>>>>>>>> gotten
>>>>>>>>> approved by appropriate management of the link you are tapping
>>>>>>>>> (which
>>>>>>>>> may
>>>>>>>>> or
>>>>>>>>> may not be you :-)).
>>>>>>>>>
>>>>>>>>> Peter Van Epp
>>>>>>>>
>>>>>>>>
>>>>>>>> What i am willing to do is to characterize the network
>>>>>>>> traffic by
>>>>>>>> using
>>>>>>>> some characteristics derived from flow information. My final
>>>>>>>> decision
>>>>>>>> about
>>>>>>>> a flow record will be whether the flow belongs to a chat
>>>>>>>> session, a
>>>>>>>> mail
>>>>>>>> transfer, a FTP connection, a web browsing, ...
>>>>>>>>
>>>>>>>> I had discovered Bro which has identifiers related with high
>>>>>>>> level
>>>>>>>> protocols. The protocol family that it supports is not as
>>>>>>>> much as
>>>>>>>> Argus
>>>>>>>> does
>>>>>>>> so i was planning to go on with Argus.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> O?uz Yar?mtepe
>>>>>>>> www.loopbacking.info
>>>>>>>> -------------- next part --------------
>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>> URL:
>>>>>>>>
>>>>>>>> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/2e9c8a9c/attachment-0001.html
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 7
>>>>>>>> Date: Tue, 3 Feb 2009 10:07:03 -0500
>>>>>>>> From: Carter Bullard <carter at qosient.com>
>>>>>>>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not
>>>>>>>> defined
>>>>>>>> To: Michael Grinnell <grinnell at american.edu>
>>>>>>>> Cc: argus-info <argus-info at lists.andrew.cmu.edu>
>>>>>>>> Message-ID: <D2F59B8C-3239-475E-AE5C-4DCE7C819151 at qosient.com>
>>>>>>>> Content-Type: text/plain; charset=US-ASCII; format=flowed;
>>>>>>>> delsp=yes
>>>>>>>>
>>>>>>>> Hey Michael,
>>>>>>>> Wow, pretty obvious bug in the flow's packet size reporting
>>>>>>>> feature.
>>>>>>>> Try this patch, and we'll see if it doesn't work for you.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>>
>>>>>>>> ==== //depot/argus/argus/argus/ArgusModeler.c#62 -
>>>>>>>> /home/carter/argus/
>>>>>>>> argus/argus/ArgusModeler.c ====
>>>>>>>> 2872c2872
>>>>>>>> < if (psize->src.psizemax > 0)
>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> if (psize->dst.psizemax > 0)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 2, 2009, at 1:39 PM, Michael Grinnell wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Periodically Argus dies on my test system with the error
>>>>>>>>> "ArgusGenerateRecord: packet size type not defined." The time
>>>>>>>>> between these errors varies, sometimes it's only a minute or
>>>>>>>>> two
>>>>>>>>> after argus starts, other times it can be > 15 minutes.
>>>>>>>>> I've tried
>>>>>>>>> running a simultaneous tcpdump, then running the resulting
>>>>>>>>> capture
>>>>>>>>> file through argus, but I can't replicate the error. I also
>>>>>>>>> don't
>>>>>>>>> see any glaring errors in the capture file around the time
>>>>>>>>> it dies.
>>>>>>>>> This happens with argus 3.0.0 and with argus 3.0.1 beta2. The
>>>>>>>>> system is running CentOS 5.2 and is listening on a dedicated
>>>>>>>>> interface (NC7782, bnx2 driver) to a span port off of a Cisco
>>>>>>>>> switch. I have also updated to the newest bnx2 drivers, but
>>>>>>>>> it
>>>>>>>>> still recurs. I'm trying to scare up another NIC to try as
>>>>>>>>> well.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Michael Grinnell
>>>>>>>>> Information Security Engineer
>>>>>>>>> The American University
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Message: 8
>>>>>>>> Date: Tue, 3 Feb 2009 10:46:45 -0500
>>>>>>>> From: Carter Bullard <carter at qosient.com>
>>>>>>>> Subject: [ARGUS] argus user data buffer analysis
>>>>>>>> To: Oguz Yarimtepe <comp.ogz at gmail.com>
>>>>>>>> Cc: argus-info at lists.andrew.cmu.edu
>>>>>>>> Message-ID: <8FDB29B8-7B9A-401B-8466-B38557690D00 at qosient.com>
>>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>>
>>>>>>>> Hey Oguz,
>>>>>>>> We have undocumented programs in the argus-3.0.0 distribution
>>>>>>>> that do something similar to what you are interested in, I
>>>>>>>> think.
>>>>>>>> These are undocumented because its a bit complex, and I haven't
>>>>>>>> had a chance to describe them yet.
>>>>>>>>
>>>>>>>> If you are interested in helping to make these types of
>>>>>>>> programs
>>>>>>>> useful, I'll be more than happy to describe how they work, and
>>>>>>>> share my experiences and help to make them better.
>>>>>>>>
>>>>>>>> The undocumented program rauserdata() will analyze the user
>>>>>>>> data portion of argus records, and generate a signature pattern
>>>>>>>> configuration for the protocols it encounters in the data set.
>>>>>>>> The algorithm is very simple, and pretty powerful, in that it
>>>>>>>> makes
>>>>>>>> no assumptions about the user data. But, you have to be
>>>>>>>> careful
>>>>>>>> with the data that you give the engine. Below is a little
>>>>>>>> background.
>>>>>>>>
>>>>>>>> I have found that for the set of protocols you listed, the
>>>>>>>> first 32 bytes of data is all that is needed to reliably
>>>>>>>> identify the
>>>>>>>> protocol type. This is because, each of your protocols have
>>>>>>>> unique greeting identifiers, and for the ones you listed, the
>>>>>>>> identifiers are all in ascii.
>>>>>>>>
>>>>>>>> Because argus provides multiple status reports for long lived
>>>>>>>> flows, not all argus records for a given flow will contain the
>>>>>>>> "first N byte" signatures that you are seeking.
>>>>>>>>
>>>>>>>> Using racluster() on your 'primitive' argus data will usually
>>>>>>>> provide
>>>>>>>> you with the "first N bytes" of user data, so that your
>>>>>>>> search for
>>>>>>>> tokens and patterns can be reliable.
>>>>>>>>
>>>>>>>> Try this out for a while to see if you get anything useful:
>>>>>>>>
>>>>>>>> racluster -r /a/days/worth/of/data/of/interest/* -w /tmp/
>>>>>>>> day.cache
>>>>>>>>
>>>>>>>> rauserdata -r /tmp/day.cache | less
>>>>>>>>
>>>>>>>> You should get an output that is arraigned by 'protocol/port'
>>>>>>>> and
>>>>>>>> you should see a set of source and destination user data
>>>>>>>> buffers
>>>>>>>> that have the "greatest likelyhood" patterns for that "proto/
>>>>>>>> port"
>>>>>>>> pair.
>>>>>>>>
>>>>>>>> ra() prints the user data buffer with an ASCII encoding by
>>>>>>>> default,
>>>>>>>> and
>>>>>>>> so you should see some patterns in the buffers it outputs.
>>>>>>>> if you see a '.', that is generally a non printable character.
>>>>>>>>
>>>>>>>> The ides is to build up configuration files of signatures using
>>>>>>>> rauserdata(),
>>>>>>>> and the program raservices(), will take the rauserdata()
>>>>>>>> output as
>>>>>>>> a configuration file, and label flows with the tags that
>>>>>>>> identify the
>>>>>>>> protocol.
>>>>>>>>
>>>>>>>> Give it a try, and I'd love to see/hear your comments.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>>
>>>>>>>> On Feb 3, 2009, at 12:59 AM, Oguz Yarimtepe wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Depends on what you need. If you enable user data capture
>>>>>>>>> (the -U
>>>>>>>>> option on the argus) it will capture up to the first 512
>>>>>>>>> bytes of
>>>>>>>>> the user
>>>>>>>>> data of the flow. That may or may not give you enough
>>>>>>>>> information
>>>>>>>>> about the
>>>>>>>>> flow to do what you want. Note that on a fast link best
>>>>>>>>> results are
>>>>>>>>> going to
>>>>>>>>> occur using a DAG card as the data capture adds a fairly
>>>>>>>>> heavy load
>>>>>>>>> to the
>>>>>>>>> server. To display the data with ra (for instance) you need
>>>>>>>>> to use
>>>>>>>>> the -s
>>>>>>>>> command to add suser and duser to the output (as in
>>>>>>>>>
>>>>>>>>> ra -r argus_file -n -s +suser:512 -s +duser:512
>>>>>>>>>
>>>>>>>>> which will tack the user data on the end of the line. This
>>>>>>>>> of course
>>>>>>>>> raises a
>>>>>>>>> number of sticky privacy issues that you need to have
>>>>>>>>> considered and
>>>>>>>>> gotten
>>>>>>>>> approved by appropriate management of the link you are tapping
>>>>>>>>> (which may or
>>>>>>>>> may not be you :-)).
>>>>>>>>>
>>>>>>>>> Peter Van Epp
>>>>>>>>>
>>>>>>>>> What i am willing to do is to characterize the network
>>>>>>>>> traffic by
>>>>>>>>> using some characteristics derived from flow information. My
>>>>>>>>> final
>>>>>>>>> decision about a flow record will be whether the flow
>>>>>>>>> belongs to a
>>>>>>>>> chat session, a mail transfer, a FTP connection, a web
>>>>>>>>> browsing, ...
>>>>>>>>>
>>>>>>>>> I had discovered Bro which has identifiers related with high
>>>>>>>>> level
>>>>>>>>> protocols. The protocol family that it supports is not as
>>>>>>>>> much as
>>>>>>>>> Argus does so i was planning to go on with Argus.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> O?uz Yar?mtepe
>>>>>>>>> www.loopbacking.info
>>>>>>>>
>>>>>>>> Carter Bullard
>>>>>>>> CEO/President
>>>>>>>> QoSient, LLC
>>>>>>>> 150 E 57th Street Suite 12D
>>>>>>>> New York, New York 10022
>>>>>>>>
>>>>>>>> +1 212 588-9133 Phone
>>>>>>>> +1 212 588-9134 Fax
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -------------- next part --------------
>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>> URL:
>>>>>>>>
>>>>>>>> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/3f7a6300/attachment.html
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Argus-info mailing list
>>>>>>>> Argus-info at lists.andrew.cmu.edu
>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>>>>>>>>
>>>>>>>>
>>>>>>>> End of Argus-info Digest, Vol 42, Issue 1
>>>>>>>> *****************************************
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> CS Lee<geek00L[at]gmail.com>
>>>>>>>
>>>>>>> http://geek00l.blogspot.com
>>>>>>> http://defcraft.net
>>>>>>>
>>>>>>> Carter Bullard
>>>>>>> CEO/President
>>>>>>> QoSient, LLC
>>>>>>> 150 E 57th Street Suite 12D
>>>>>>> New York, New York 10022
>>>>>>> +1 212 588-9133 Phone
>>>>>>> +1 212 588-9134 Fax
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>>
>>>>>> CS Lee<geek00L[at]gmail.com>
>>>>>>
>>>>>> http://geek00l.blogspot.com
>>>>>> http://defcraft.net
>>>>>>
>>>>>> Carter Bullard
>>>>>> CEO/President
>>>>>> QoSient, LLC
>>>>>> 150 E 57th Street Suite 12D
>>>>>> New York, New York 10022
>>>>>> +1 212 588-9133 Phone
>>>>>> +1 212 588-9134 Fax
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>>
>>>>> CS Lee<geek00L[at]gmail.com>
>>>>>
>>>>> http://geek00l.blogspot.com
>>>>> http://defcraft.net
>>>>>
>>>>
>>> <argus-dash.jpg>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>>
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
More information about the argus
mailing list