Argus-info Digest, Vol 42, Issue 1
CS Lee
geek00l at gmail.com
Thu Feb 5 11:03:04 EST 2009
hi Carter,
I grab your control plane presentation from here -
https://tools.netsa.cert.org/wiki/display/tt/FloCon+2009
I'm definitely a user, can't guarantee to be customer unless my boss says
yes ;]
The mysql support is definitely a long wait ;]
On Thu, Feb 5, 2009 at 11:51 PM, Carter Bullard <carter at qosient.com> wrote:
> Thanks, I don't think I gave them slides notes, .... didn't know if itwas
> intelligible without some form of narrative.
>
> So I have a bunch of mysql database support for argus almost ready
> for public consumption, would you be a potential customer/user?
>
> I use it primarily to hold real-time "ratop" like state so multiple
> people/programs can work off of the data at the same time, but
> some people have had good luck using it to hold flow data and
> to keep track of say all the IP addresses they've seen since the
> beginning, in real-time.
>
> Carter
>
>
> On Feb 4, 2009, at 9:11 PM, CS Lee wrote:
>
> hi Carter,
>
> Thanks. Things are all good, btw, I read your presentation in Flocon2009,
> that's really cool stuff.
>
> On Thu, Feb 5, 2009 at 3:21 AM, Carter Bullard <carter at qosient.com> wrote:
>
>> Hmmmm,I have them in the development directory, here. Let me see what's
>> up.
>> If we can fix the ongoing v2 -> v3 conversion problem soon, I'll release
>> another round of client programs, and it will definitely have them in
>> there.
>>
>> How are things?
>>
>> Carter
>>
>> On Feb 4, 2009, at 1:22 PM, CS Lee wrote:
>>
>> hi carter,
>>
>> i can't find both rauserdata() and raservice() in argus 3.0 distribution
>> which I downloaded from
>>
>> ftp://qosient.com/dev/argus-3.0/
>>
>> Thanks!
>>
>>
>> On Tue, Feb 3, 2009 at 11:46 PM, <argus-info-request at lists.andrew.cmu.edu
>> > wrote:
>>
>>> Send Argus-info mailing list submissions to
>>> argus-info at lists.andrew.cmu.edu
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>>> or, via email, send a message with subject or body 'help' to
>>> argus-info-request at lists.andrew.cmu.edu
>>>
>>> You can reach the person managing the list at
>>> argus-info-owner at lists.andrew.cmu.edu
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Argus-info digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. ArgusGenerateRecord: packet size type not defined
>>> (Michael Grinnell)
>>> 2. argus usage (Oguz Yarimtepe)
>>> 3. Re: argus usage (Stewart Gray)
>>> 4. Re: ArgusGenerateRecord: packet size type not defined
>>> (Peter Van Epp)
>>> 5. Re: argus usage (Peter Van Epp)
>>> 6. Re: argus usage (Oguz Yarimtepe)
>>> 7. Re: ArgusGenerateRecord: packet size type not defined
>>> (Carter Bullard)
>>> 8. argus user data buffer analysis (Carter Bullard)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Mon, 02 Feb 2009 13:39:44 -0500
>>> From: Michael Grinnell <grinnell at american.edu>
>>> Subject: [ARGUS] ArgusGenerateRecord: packet size type not defined
>>> To: argus-info <argus-info at lists.andrew.cmu.edu>
>>> Message-ID: <49873DF0.6090305 at american.edu>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> Hi,
>>>
>>> Periodically Argus dies on my test system with the error
>>> "ArgusGenerateRecord: packet size type not defined." The time between
>>> these errors varies, sometimes it's only a minute or two after argus
>>> starts, other times it can be > 15 minutes. I've tried running a
>>> simultaneous tcpdump, then running the resulting capture file through
>>> argus, but I can't replicate the error. I also don't see any glaring
>>> errors in the capture file around the time it dies. This happens with
>>> argus 3.0.0 and with argus 3.0.1 beta2. The system is running CentOS
>>> 5.2 and is listening on a dedicated interface (NC7782, bnx2 driver) to a
>>> span port off of a Cisco switch. I have also updated to the newest bnx2
>>> drivers, but it still recurs. I'm trying to scare up another NIC to try
>>> as well.
>>>
>>> Thoughts?
>>>
>>> --
>>> Michael Grinnell
>>> Information Security Engineer
>>> The American University
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Mon, 02 Feb 2009 22:50:20 +0200
>>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>>> Subject: [ARGUS] argus usage
>>> To: argus-info at lists.andrew.cmu.edu
>>> Message-ID: <1233607820.7050.12.camel at gezenti>
>>> Content-Type: text/plain
>>>
>>> Hi,
>>>
>>> I want some information about argus capabilities. I tried argus a bit
>>> and read the wiki. I want to learn whether it is possible to get
>>> application level information from a flow record by using argus, like
>>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>> As i understood from its usage it is possible to get these information
>>> indirectly by analyzing the argus output but maybe there is a way that
>>> argus serves that i don't know.
>>>
>>> Thanx.
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Tue, 3 Feb 2009 10:26:06 +1300
>>> From: "Stewart Gray" <Stewart.Gray at safecom.co.nz>
>>> Subject: Re: [ARGUS] argus usage
>>> To: <argus-info at lists.andrew.cmu.edu>
>>> Message-ID:
>>> <
>>> 4C08B31BFF3B5D42BCAC255ADDD6EE5EA8769D at tasbe116.advancedsolutions.co.nz>
>>>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> Hi Oguz,
>>>
>>> Argus supports the Berkeley packet filter (as other apps like tcpdump
>>> and ngrep do) so you're able to lock queries down to port numbers,
>>> specific hosts etc. Knowing the port number an application runs on means
>>> it fairly easy to pull out transactions for a particular protocol. Keep
>>> in mind it doesn't operate at layer 7, flow information for a protocol
>>> such as HTTP will be very similar to SMTP. If you're after more in-depth
>>> analysis of a transaction, i.e. what happened during an http connection
>>> (GET, POST, HEAD etc) you're probably best to be taking full packet
>>> captures alongside argus so you can run them through another tool such
>>> as wireshark.
>>>
>>> Have a look at this wiki page for some common usages -
>>> http://nsmwiki.org/index.php?title=Argus.
>>>
>>> Cheers,
>>>
>>> Stewart
>>>
>>> -----Original Message-----
>>> From: argus-info-bounces+stewart.gray=safecom.co.nz at lists.andrew.cmu.edu
>>> [mailto:argus-info-bounces+stewart.gray<argus-info-bounces%2Bstewart.gray>
>>> =safecom.co.nz at lists.andrew.cmu.e
>>> du] On Behalf Of Oguz Yarimtepe
>>> Sent: Tuesday, 3 February 2009 9:50 a.m.
>>> To: argus-info at lists.andrew.cmu.edu
>>> Subject: [ARGUS] argus usage
>>>
>>> Hi,
>>>
>>> I want some information about argus capabilities. I tried argus a bit
>>> and read the wiki. I want to learn whether it is possible to get
>>> application level information from a flow record by using argus, like
>>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>> As i understood from its usage it is possible to get these information
>>> indirectly by analyzing the argus output but maybe there is a way that
>>> argus serves that i don't know.
>>>
>>> Thanx.
>>>
>>>
>>> #####################################################################################
>>> Important: This electronic message and attachments (if any) are
>>> confidential
>>> and may be legally privileged. If you are not the intended recipient do
>>> not
>>> copy, disclose or use the contents in any way. Please let us know by
>>> return
>>> e-mail immediately and then destroy this message.
>>>
>>> #####################################################################################
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 4
>>> Date: Mon, 2 Feb 2009 20:51:05 -0800
>>> From: Peter Van Epp <vanepp at sfu.ca>
>>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not defined
>>> To: argus-info at lists.andrew.cmu.edu
>>> Message-ID: <20090203045105.GA26496 at sfu.ca>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> On Mon, Feb 02, 2009 at 01:39:44PM -0500, Michael Grinnell wrote:
>>> > Hi,
>>> >
>>> > Periodically Argus dies on my test system with the error
>>> > "ArgusGenerateRecord: packet size type not defined." The time between
>>> > these errors varies, sometimes it's only a minute or two after argus
>>> > starts, other times it can be > 15 minutes. I've tried running a
>>> > simultaneous tcpdump, then running the resulting capture file through
>>> > argus, but I can't replicate the error. I also don't see any glaring
>>> > errors in the capture file around the time it dies. This happens with
>>> > argus 3.0.0 and with argus 3.0.1 beta2. The system is running CentOS
>>> > 5.2 and is listening on a dedicated interface (NC7782, bnx2 driver) to
>>> a
>>> > span port off of a Cisco switch. I have also updated to the newest
>>> bnx2
>>> > drivers, but it still recurs. I'm trying to scare up another NIC to
>>> try
>>> > as well.
>>> >
>>> > Thoughts?
>>> >
>>> > --
>>> > Michael Grinnell
>>> > Information Security Engineer
>>> > The American University
>>>
>>> Setting
>>>
>>> ARGUS_PACKET_CAPTURE_FILE="/var/log/argus/packet.out"
>>>
>>> in an argus.rc file will capture the input packets from pcap in to the
>>> specified file. If you can get lucky and get a failure before you run out
>>> of
>>> disk space one of the last packets in the file should tell us what argus
>>> isn't
>>> liking. On a busy link this file will get large fast but if it sometimes
>>> fails quickly you may be lucky (you are also likely to see packet loss
>>> due to
>>> the disk I/O on the sensor but hopefully the fault will still occur).
>>> It looks like the argus record is malformed (it is complaining
>>> that it
>>> doesn't recognize the type in argus/ArgusModeler.c at line 2904 in the
>>> argus-3.0.1.beta.2 code). A dump of the offending packet should tell
>>> Carter
>>> why (or if the incoming packet is corrupted which is also possible).
>>>
>>> Peter Van Epp
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 5
>>> Date: Mon, 2 Feb 2009 21:06:59 -0800
>>> From: Peter Van Epp <vanepp at sfu.ca>
>>> Subject: Re: [ARGUS] argus usage
>>> To: argus-info at lists.andrew.cmu.edu
>>> Message-ID: <20090203050659.GA8930 at sfu.ca>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> On Mon, Feb 02, 2009 at 10:50:20PM +0200, Oguz Yarimtepe wrote:
>>> > Hi,
>>> >
>>> > I want some information about argus capabilities. I tried argus a bit
>>> > and read the wiki. I want to learn whether it is possible to get
>>> > application level information from a flow record by using argus, like
>>> > HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>>> > As i understood from its usage it is possible to get these information
>>> > indirectly by analyzing the argus output but maybe there is a way that
>>> > argus serves that i don't know.
>>> >
>>> > Thanx.
>>> >
>>>
>>> Depends on what you need. If you enable user data capture (the -U
>>> option on the argus) it will capture up to the first 512 bytes of the
>>> user
>>> data of the flow. That may or may not give you enough information about
>>> the
>>> flow to do what you want. Note that on a fast link best results are going
>>> to
>>> occur using a DAG card as the data capture adds a fairly heavy load to
>>> the
>>> server. To display the data with ra (for instance) you need to use the -s
>>> command to add suser and duser to the output (as in
>>>
>>> ra -r argus_file -n -s +suser:512 -s +duser:512
>>>
>>> which will tack the user data on the end of the line. This of course
>>> raises a
>>> number of sticky privacy issues that you need to have considered and
>>> gotten
>>> approved by appropriate management of the link you are tapping (which may
>>> or
>>> may not be you :-)).
>>>
>>> Peter Van Epp
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 6
>>> Date: Tue, 3 Feb 2009 07:59:33 +0200
>>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>>> Subject: Re: [ARGUS] argus usage
>>> To: argus-info at lists.andrew.cmu.edu
>>> Message-ID:
>>> <20831c740902022159q176610e5nb954f5d96e211d13 at mail.gmail.com>
>>> Content-Type: text/plain; charset="iso-8859-9"
>>>
>>> >
>>> > Depends on what you need. If you enable user data capture (the
>>> -U
>>> > option on the argus) it will capture up to the first 512 bytes of the
>>> user
>>> > data of the flow. That may or may not give you enough information about
>>> the
>>> > flow to do what you want. Note that on a fast link best results are
>>> going
>>> > to
>>> > occur using a DAG card as the data capture adds a fairly heavy load to
>>> the
>>> > server. To display the data with ra (for instance) you need to use the
>>> -s
>>> > command to add suser and duser to the output (as in
>>> >
>>> > ra -r argus_file -n -s +suser:512 -s +duser:512
>>> >
>>> > which will tack the user data on the end of the line. This of course
>>> raises
>>> > a
>>> > number of sticky privacy issues that you need to have considered and
>>> gotten
>>> > approved by appropriate management of the link you are tapping (which
>>> may
>>> > or
>>> > may not be you :-)).
>>> >
>>> > Peter Van Epp
>>>
>>>
>>> What i am willing to do is to characterize the network traffic by using
>>> some characteristics derived from flow information. My final decision
>>> about
>>> a flow record will be whether the flow belongs to a chat session, a mail
>>> transfer, a FTP connection, a web browsing, ...
>>>
>>> I had discovered Bro which has identifiers related with high level
>>> protocols. The protocol family that it supports is not as much as Argus
>>> does
>>> so i was planning to go on with Argus.
>>>
>>>
>>> --
>>> O?uz Yar?mtepe
>>> www.loopbacking.info
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/2e9c8a9c/attachment-0001.html
>>>
>>> ------------------------------
>>>
>>> Message: 7
>>> Date: Tue, 3 Feb 2009 10:07:03 -0500
>>> From: Carter Bullard <carter at qosient.com>
>>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not defined
>>> To: Michael Grinnell <grinnell at american.edu>
>>> Cc: argus-info <argus-info at lists.andrew.cmu.edu>
>>> Message-ID: <D2F59B8C-3239-475E-AE5C-4DCE7C819151 at qosient.com>
>>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>>
>>> Hey Michael,
>>> Wow, pretty obvious bug in the flow's packet size reporting feature.
>>> Try this patch, and we'll see if it doesn't work for you.
>>>
>>> Carter
>>>
>>> ==== //depot/argus/argus/argus/ArgusModeler.c#62 - /home/carter/argus/
>>> argus/argus/ArgusModeler.c ====
>>> 2872c2872
>>> < if (psize->src.psizemax > 0)
>>> ---
>>> > if (psize->dst.psizemax > 0)
>>>
>>>
>>> On Feb 2, 2009, at 1:39 PM, Michael Grinnell wrote:
>>>
>>> > Hi,
>>> >
>>> > Periodically Argus dies on my test system with the error
>>> > "ArgusGenerateRecord: packet size type not defined." The time
>>> > between these errors varies, sometimes it's only a minute or two
>>> > after argus starts, other times it can be > 15 minutes. I've tried
>>> > running a simultaneous tcpdump, then running the resulting capture
>>> > file through argus, but I can't replicate the error. I also don't
>>> > see any glaring errors in the capture file around the time it dies.
>>> > This happens with argus 3.0.0 and with argus 3.0.1 beta2. The
>>> > system is running CentOS 5.2 and is listening on a dedicated
>>> > interface (NC7782, bnx2 driver) to a span port off of a Cisco
>>> > switch. I have also updated to the newest bnx2 drivers, but it
>>> > still recurs. I'm trying to scare up another NIC to try as well.
>>> >
>>> > Thoughts?
>>> >
>>> > --
>>> > Michael Grinnell
>>> > Information Security Engineer
>>> > The American University
>>> >
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 8
>>> Date: Tue, 3 Feb 2009 10:46:45 -0500
>>> From: Carter Bullard <carter at qosient.com>
>>> Subject: [ARGUS] argus user data buffer analysis
>>> To: Oguz Yarimtepe <comp.ogz at gmail.com>
>>> Cc: argus-info at lists.andrew.cmu.edu
>>> Message-ID: <8FDB29B8-7B9A-401B-8466-B38557690D00 at qosient.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Hey Oguz,
>>> We have undocumented programs in the argus-3.0.0 distribution
>>> that do something similar to what you are interested in, I think.
>>> These are undocumented because its a bit complex, and I haven't
>>> had a chance to describe them yet.
>>>
>>> If you are interested in helping to make these types of programs
>>> useful, I'll be more than happy to describe how they work, and
>>> share my experiences and help to make them better.
>>>
>>> The undocumented program rauserdata() will analyze the user
>>> data portion of argus records, and generate a signature pattern
>>> configuration for the protocols it encounters in the data set.
>>> The algorithm is very simple, and pretty powerful, in that it makes
>>> no assumptions about the user data. But, you have to be careful
>>> with the data that you give the engine. Below is a little background.
>>>
>>> I have found that for the set of protocols you listed, the
>>> first 32 bytes of data is all that is needed to reliably identify the
>>> protocol type. This is because, each of your protocols have
>>> unique greeting identifiers, and for the ones you listed, the
>>> identifiers are all in ascii.
>>>
>>> Because argus provides multiple status reports for long lived
>>> flows, not all argus records for a given flow will contain the
>>> "first N byte" signatures that you are seeking.
>>>
>>> Using racluster() on your 'primitive' argus data will usually provide
>>> you with the "first N bytes" of user data, so that your search for
>>> tokens and patterns can be reliable.
>>>
>>> Try this out for a while to see if you get anything useful:
>>>
>>> racluster -r /a/days/worth/of/data/of/interest/* -w /tmp/day.cache
>>>
>>> rauserdata -r /tmp/day.cache | less
>>>
>>> You should get an output that is arraigned by 'protocol/port' and
>>> you should see a set of source and destination user data buffers
>>> that have the "greatest likelyhood" patterns for that "proto/port" pair.
>>>
>>> ra() prints the user data buffer with an ASCII encoding by default, and
>>> so you should see some patterns in the buffers it outputs.
>>> if you see a '.', that is generally a non printable character.
>>>
>>> The ides is to build up configuration files of signatures using
>>> rauserdata(),
>>> and the program raservices(), will take the rauserdata() output as
>>> a configuration file, and label flows with the tags that identify the
>>> protocol.
>>>
>>> Give it a try, and I'd love to see/hear your comments.
>>>
>>> Carter
>>>
>>> On Feb 3, 2009, at 12:59 AM, Oguz Yarimtepe wrote:
>>>
>>> >
>>> >
>>> > Depends on what you need. If you enable user data capture
>>> > (the -U
>>> > option on the argus) it will capture up to the first 512 bytes of
>>> > the user
>>> > data of the flow. That may or may not give you enough information
>>> > about the
>>> > flow to do what you want. Note that on a fast link best results are
>>> > going to
>>> > occur using a DAG card as the data capture adds a fairly heavy load
>>> > to the
>>> > server. To display the data with ra (for instance) you need to use
>>> > the -s
>>> > command to add suser and duser to the output (as in
>>> >
>>> > ra -r argus_file -n -s +suser:512 -s +duser:512
>>> >
>>> > which will tack the user data on the end of the line. This of course
>>> > raises a
>>> > number of sticky privacy issues that you need to have considered and
>>> > gotten
>>> > approved by appropriate management of the link you are tapping
>>> > (which may or
>>> > may not be you :-)).
>>> >
>>> > Peter Van Epp
>>> >
>>> > What i am willing to do is to characterize the network traffic by
>>> > using some characteristics derived from flow information. My final
>>> > decision about a flow record will be whether the flow belongs to a
>>> > chat session, a mail transfer, a FTP connection, a web browsing, ...
>>> >
>>> > I had discovered Bro which has identifiers related with high level
>>> > protocols. The protocol family that it supports is not as much as
>>> > Argus does so i was planning to go on with Argus.
>>> >
>>> >
>>> > --
>>> > O?uz Yar?mtepe
>>> > www.loopbacking.info
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E 57th Street Suite 12D
>>> New York, New York 10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/3f7a6300/attachment.html
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Argus-info mailing list
>>> Argus-info at lists.andrew.cmu.edu
>>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>>>
>>>
>>> End of Argus-info Digest, Vol 42, Issue 1
>>> *****************************************
>>>
>>
>>
>>
>> --
>> Best Regards,
>>
>> CS Lee<geek00L[at]gmail.com>
>>
>> http://geek00l.blogspot.com
>> http://defcraft.net
>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
> http://defcraft.net
>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>
--
Best Regards,
CS Lee<geek00L[at]gmail.com>
http://geek00l.blogspot.com
http://defcraft.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090206/09abbafe/attachment.html>
More information about the argus
mailing list