Argus-info Digest, Vol 42, Issue 1
Carter Bullard
carter at qosient.com
Thu Feb 5 10:51:08 EST 2009
Thanks, I don't think I gave them slides notes, .... didn't know if it
was intelligible without some form of narrative.
So I have a bunch of mysql database support for argus almost ready
for public consumption, would you be a potential customer/user?
I use it primarily to hold real-time "ratop" like state so multiple
people/programs can work off of the data at the same time, but
some people have had good luck using it to hold flow data and
to keep track of say all the IP addresses they've seen since the
beginning, in real-time.
Carter
On Feb 4, 2009, at 9:11 PM, CS Lee wrote:
> hi Carter,
>
> Thanks. Things are all good, btw, I read your presentation in
> Flocon2009, that's really cool stuff.
>
> On Thu, Feb 5, 2009 at 3:21 AM, Carter Bullard <carter at qosient.com>
> wrote:
> Hmmmm,
> I have them in the development directory, here. Let me see what's up.
> If we can fix the ongoing v2 -> v3 conversion problem soon, I'll
> release
> another round of client programs, and it will definitely have them in
> there.
>
> How are things?
>
> Carter
>
> On Feb 4, 2009, at 1:22 PM, CS Lee wrote:
>
>> hi carter,
>>
>> i can't find both rauserdata() and raservice() in argus 3.0
>> distribution which I downloaded from
>>
>> ftp://qosient.com/dev/argus-3.0/
>>
>> Thanks!
>>
>>
>> On Tue, Feb 3, 2009 at 11:46 PM, <argus-info-request at lists.andrew.cmu.edu
>> > wrote:
>> Send Argus-info mailing list submissions to
>> argus-info at lists.andrew.cmu.edu
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>> or, via email, send a message with subject or body 'help' to
>> argus-info-request at lists.andrew.cmu.edu
>>
>> You can reach the person managing the list at
>> argus-info-owner at lists.andrew.cmu.edu
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Argus-info digest..."
>>
>>
>> Today's Topics:
>>
>> 1. ArgusGenerateRecord: packet size type not defined
>> (Michael Grinnell)
>> 2. argus usage (Oguz Yarimtepe)
>> 3. Re: argus usage (Stewart Gray)
>> 4. Re: ArgusGenerateRecord: packet size type not defined
>> (Peter Van Epp)
>> 5. Re: argus usage (Peter Van Epp)
>> 6. Re: argus usage (Oguz Yarimtepe)
>> 7. Re: ArgusGenerateRecord: packet size type not defined
>> (Carter Bullard)
>> 8. argus user data buffer analysis (Carter Bullard)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 02 Feb 2009 13:39:44 -0500
>> From: Michael Grinnell <grinnell at american.edu>
>> Subject: [ARGUS] ArgusGenerateRecord: packet size type not defined
>> To: argus-info <argus-info at lists.andrew.cmu.edu>
>> Message-ID: <49873DF0.6090305 at american.edu>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hi,
>>
>> Periodically Argus dies on my test system with the error
>> "ArgusGenerateRecord: packet size type not defined." The time
>> between
>> these errors varies, sometimes it's only a minute or two after argus
>> starts, other times it can be > 15 minutes. I've tried running a
>> simultaneous tcpdump, then running the resulting capture file through
>> argus, but I can't replicate the error. I also don't see any glaring
>> errors in the capture file around the time it dies. This happens
>> with
>> argus 3.0.0 and with argus 3.0.1 beta2. The system is running CentOS
>> 5.2 and is listening on a dedicated interface (NC7782, bnx2 driver)
>> to a
>> span port off of a Cisco switch. I have also updated to the newest
>> bnx2
>> drivers, but it still recurs. I'm trying to scare up another NIC
>> to try
>> as well.
>>
>> Thoughts?
>>
>> --
>> Michael Grinnell
>> Information Security Engineer
>> The American University
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 02 Feb 2009 22:50:20 +0200
>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>> Subject: [ARGUS] argus usage
>> To: argus-info at lists.andrew.cmu.edu
>> Message-ID: <1233607820.7050.12.camel at gezenti>
>> Content-Type: text/plain
>>
>> Hi,
>>
>> I want some information about argus capabilities. I tried argus a bit
>> and read the wiki. I want to learn whether it is possible to get
>> application level information from a flow record by using argus, like
>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>> As i understood from its usage it is possible to get these
>> information
>> indirectly by analyzing the argus output but maybe there is a way
>> that
>> argus serves that i don't know.
>>
>> Thanx.
>>
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 3 Feb 2009 10:26:06 +1300
>> From: "Stewart Gray" <Stewart.Gray at safecom.co.nz>
>> Subject: Re: [ARGUS] argus usage
>> To: <argus-info at lists.andrew.cmu.edu>
>> Message-ID:
>> <4C08B31BFF3B5D42BCAC255ADDD6EE5EA8769D at tasbe116.advancedsolutions.co.nz
>> >
>>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi Oguz,
>>
>> Argus supports the Berkeley packet filter (as other apps like tcpdump
>> and ngrep do) so you're able to lock queries down to port numbers,
>> specific hosts etc. Knowing the port number an application runs on
>> means
>> it fairly easy to pull out transactions for a particular protocol.
>> Keep
>> in mind it doesn't operate at layer 7, flow information for a
>> protocol
>> such as HTTP will be very similar to SMTP. If you're after more in-
>> depth
>> analysis of a transaction, i.e. what happened during an http
>> connection
>> (GET, POST, HEAD etc) you're probably best to be taking full packet
>> captures alongside argus so you can run them through another tool
>> such
>> as wireshark.
>>
>> Have a look at this wiki page for some common usages -
>> http://nsmwiki.org/index.php?title=Argus.
>>
>> Cheers,
>>
>> Stewart
>>
>> -----Original Message-----
>> From: argus-info-bounces+stewart.gray=safecom.co.nz at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+stewart.gray=safecom.co.nz at lists.andrew.cmu.e
>> du] On Behalf Of Oguz Yarimtepe
>> Sent: Tuesday, 3 February 2009 9:50 a.m.
>> To: argus-info at lists.andrew.cmu.edu
>> Subject: [ARGUS] argus usage
>>
>> Hi,
>>
>> I want some information about argus capabilities. I tried argus a bit
>> and read the wiki. I want to learn whether it is possible to get
>> application level information from a flow record by using argus, like
>> HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>> As i understood from its usage it is possible to get these
>> information
>> indirectly by analyzing the argus output but maybe there is a way
>> that
>> argus serves that i don't know.
>>
>> Thanx.
>>
>> #####################################################################################
>> Important: This electronic message and attachments (if any) are
>> confidential
>> and may be legally privileged. If you are not the intended
>> recipient do not
>> copy, disclose or use the contents in any way. Please let us know
>> by return
>> e-mail immediately and then destroy this message.
>> #####################################################################################
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Mon, 2 Feb 2009 20:51:05 -0800
>> From: Peter Van Epp <vanepp at sfu.ca>
>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not
>> defined
>> To: argus-info at lists.andrew.cmu.edu
>> Message-ID: <20090203045105.GA26496 at sfu.ca>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Mon, Feb 02, 2009 at 01:39:44PM -0500, Michael Grinnell wrote:
>> > Hi,
>> >
>> > Periodically Argus dies on my test system with the error
>> > "ArgusGenerateRecord: packet size type not defined." The time
>> between
>> > these errors varies, sometimes it's only a minute or two after
>> argus
>> > starts, other times it can be > 15 minutes. I've tried running a
>> > simultaneous tcpdump, then running the resulting capture file
>> through
>> > argus, but I can't replicate the error. I also don't see any
>> glaring
>> > errors in the capture file around the time it dies. This happens
>> with
>> > argus 3.0.0 and with argus 3.0.1 beta2. The system is running
>> CentOS
>> > 5.2 and is listening on a dedicated interface (NC7782, bnx2
>> driver) to a
>> > span port off of a Cisco switch. I have also updated to the
>> newest bnx2
>> > drivers, but it still recurs. I'm trying to scare up another NIC
>> to try
>> > as well.
>> >
>> > Thoughts?
>> >
>> > --
>> > Michael Grinnell
>> > Information Security Engineer
>> > The American University
>>
>> Setting
>>
>> ARGUS_PACKET_CAPTURE_FILE="/var/log/argus/packet.out"
>>
>> in an argus.rc file will capture the input packets from pcap in to
>> the
>> specified file. If you can get lucky and get a failure before you
>> run out of
>> disk space one of the last packets in the file should tell us what
>> argus isn't
>> liking. On a busy link this file will get large fast but if it
>> sometimes
>> fails quickly you may be lucky (you are also likely to see packet
>> loss due to
>> the disk I/O on the sensor but hopefully the fault will still occur).
>> It looks like the argus record is malformed (it is
>> complaining that it
>> doesn't recognize the type in argus/ArgusModeler.c at line 2904 in
>> the
>> argus-3.0.1.beta.2 code). A dump of the offending packet should
>> tell Carter
>> why (or if the incoming packet is corrupted which is also possible).
>>
>> Peter Van Epp
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Mon, 2 Feb 2009 21:06:59 -0800
>> From: Peter Van Epp <vanepp at sfu.ca>
>> Subject: Re: [ARGUS] argus usage
>> To: argus-info at lists.andrew.cmu.edu
>> Message-ID: <20090203050659.GA8930 at sfu.ca>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Mon, Feb 02, 2009 at 10:50:20PM +0200, Oguz Yarimtepe wrote:
>> > Hi,
>> >
>> > I want some information about argus capabilities. I tried argus a
>> bit
>> > and read the wiki. I want to learn whether it is possible to get
>> > application level information from a flow record by using argus,
>> like
>> > HTTP, HTTPS, IMAP, POP, SMTP, FTP ...
>> > As i understood from its usage it is possible to get these
>> information
>> > indirectly by analyzing the argus output but maybe there is a way
>> that
>> > argus serves that i don't know.
>> >
>> > Thanx.
>> >
>>
>> Depends on what you need. If you enable user data capture
>> (the -U
>> option on the argus) it will capture up to the first 512 bytes of
>> the user
>> data of the flow. That may or may not give you enough information
>> about the
>> flow to do what you want. Note that on a fast link best results are
>> going to
>> occur using a DAG card as the data capture adds a fairly heavy load
>> to the
>> server. To display the data with ra (for instance) you need to use
>> the -s
>> command to add suser and duser to the output (as in
>>
>> ra -r argus_file -n -s +suser:512 -s +duser:512
>>
>> which will tack the user data on the end of the line. This of
>> course raises a
>> number of sticky privacy issues that you need to have considered
>> and gotten
>> approved by appropriate management of the link you are tapping
>> (which may or
>> may not be you :-)).
>>
>> Peter Van Epp
>>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Tue, 3 Feb 2009 07:59:33 +0200
>> From: Oguz Yarimtepe <comp.ogz at gmail.com>
>> Subject: Re: [ARGUS] argus usage
>> To: argus-info at lists.andrew.cmu.edu
>> Message-ID:
>> <20831c740902022159q176610e5nb954f5d96e211d13 at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-9"
>>
>> >
>> > Depends on what you need. If you enable user data capture
>> (the -U
>> > option on the argus) it will capture up to the first 512 bytes of
>> the user
>> > data of the flow. That may or may not give you enough information
>> about the
>> > flow to do what you want. Note that on a fast link best results
>> are going
>> > to
>> > occur using a DAG card as the data capture adds a fairly heavy
>> load to the
>> > server. To display the data with ra (for instance) you need to
>> use the -s
>> > command to add suser and duser to the output (as in
>> >
>> > ra -r argus_file -n -s +suser:512 -s +duser:512
>> >
>> > which will tack the user data on the end of the line. This of
>> course raises
>> > a
>> > number of sticky privacy issues that you need to have considered
>> and gotten
>> > approved by appropriate management of the link you are tapping
>> (which may
>> > or
>> > may not be you :-)).
>> >
>> > Peter Van Epp
>>
>>
>> What i am willing to do is to characterize the network traffic by
>> using
>> some characteristics derived from flow information. My final
>> decision about
>> a flow record will be whether the flow belongs to a chat session, a
>> mail
>> transfer, a FTP connection, a web browsing, ...
>>
>> I had discovered Bro which has identifiers related with high level
>> protocols. The protocol family that it supports is not as much as
>> Argus does
>> so i was planning to go on with Argus.
>>
>>
>> --
>> O?uz Yar?mtepe
>> www.loopbacking.info
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/2e9c8a9c/attachment-0001.html
>>
>> ------------------------------
>>
>> Message: 7
>> Date: Tue, 3 Feb 2009 10:07:03 -0500
>> From: Carter Bullard <carter at qosient.com>
>> Subject: Re: [ARGUS] ArgusGenerateRecord: packet size type not
>> defined
>> To: Michael Grinnell <grinnell at american.edu>
>> Cc: argus-info <argus-info at lists.andrew.cmu.edu>
>> Message-ID: <D2F59B8C-3239-475E-AE5C-4DCE7C819151 at qosient.com>
>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>
>> Hey Michael,
>> Wow, pretty obvious bug in the flow's packet size reporting feature.
>> Try this patch, and we'll see if it doesn't work for you.
>>
>> Carter
>>
>> ==== //depot/argus/argus/argus/ArgusModeler.c#62 - /home/carter/
>> argus/
>> argus/argus/ArgusModeler.c ====
>> 2872c2872
>> < if (psize->src.psizemax > 0)
>> ---
>> > if (psize->dst.psizemax > 0)
>>
>>
>> On Feb 2, 2009, at 1:39 PM, Michael Grinnell wrote:
>>
>> > Hi,
>> >
>> > Periodically Argus dies on my test system with the error
>> > "ArgusGenerateRecord: packet size type not defined." The time
>> > between these errors varies, sometimes it's only a minute or two
>> > after argus starts, other times it can be > 15 minutes. I've tried
>> > running a simultaneous tcpdump, then running the resulting capture
>> > file through argus, but I can't replicate the error. I also don't
>> > see any glaring errors in the capture file around the time it dies.
>> > This happens with argus 3.0.0 and with argus 3.0.1 beta2. The
>> > system is running CentOS 5.2 and is listening on a dedicated
>> > interface (NC7782, bnx2 driver) to a span port off of a Cisco
>> > switch. I have also updated to the newest bnx2 drivers, but it
>> > still recurs. I'm trying to scare up another NIC to try as well.
>> >
>> > Thoughts?
>> >
>> > --
>> > Michael Grinnell
>> > Information Security Engineer
>> > The American University
>> >
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 8
>> Date: Tue, 3 Feb 2009 10:46:45 -0500
>> From: Carter Bullard <carter at qosient.com>
>> Subject: [ARGUS] argus user data buffer analysis
>> To: Oguz Yarimtepe <comp.ogz at gmail.com>
>> Cc: argus-info at lists.andrew.cmu.edu
>> Message-ID: <8FDB29B8-7B9A-401B-8466-B38557690D00 at qosient.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hey Oguz,
>> We have undocumented programs in the argus-3.0.0 distribution
>> that do something similar to what you are interested in, I think.
>> These are undocumented because its a bit complex, and I haven't
>> had a chance to describe them yet.
>>
>> If you are interested in helping to make these types of programs
>> useful, I'll be more than happy to describe how they work, and
>> share my experiences and help to make them better.
>>
>> The undocumented program rauserdata() will analyze the user
>> data portion of argus records, and generate a signature pattern
>> configuration for the protocols it encounters in the data set.
>> The algorithm is very simple, and pretty powerful, in that it makes
>> no assumptions about the user data. But, you have to be careful
>> with the data that you give the engine. Below is a little
>> background.
>>
>> I have found that for the set of protocols you listed, the
>> first 32 bytes of data is all that is needed to reliably identify the
>> protocol type. This is because, each of your protocols have
>> unique greeting identifiers, and for the ones you listed, the
>> identifiers are all in ascii.
>>
>> Because argus provides multiple status reports for long lived
>> flows, not all argus records for a given flow will contain the
>> "first N byte" signatures that you are seeking.
>>
>> Using racluster() on your 'primitive' argus data will usually provide
>> you with the "first N bytes" of user data, so that your search for
>> tokens and patterns can be reliable.
>>
>> Try this out for a while to see if you get anything useful:
>>
>> racluster -r /a/days/worth/of/data/of/interest/* -w /tmp/day.cache
>>
>> rauserdata -r /tmp/day.cache | less
>>
>> You should get an output that is arraigned by 'protocol/port' and
>> you should see a set of source and destination user data buffers
>> that have the "greatest likelyhood" patterns for that "proto/port"
>> pair.
>>
>> ra() prints the user data buffer with an ASCII encoding by default,
>> and
>> so you should see some patterns in the buffers it outputs.
>> if you see a '.', that is generally a non printable character.
>>
>> The ides is to build up configuration files of signatures using
>> rauserdata(),
>> and the program raservices(), will take the rauserdata() output as
>> a configuration file, and label flows with the tags that identify the
>> protocol.
>>
>> Give it a try, and I'd love to see/hear your comments.
>>
>> Carter
>>
>> On Feb 3, 2009, at 12:59 AM, Oguz Yarimtepe wrote:
>>
>> >
>> >
>> > Depends on what you need. If you enable user data capture
>> > (the -U
>> > option on the argus) it will capture up to the first 512 bytes of
>> > the user
>> > data of the flow. That may or may not give you enough information
>> > about the
>> > flow to do what you want. Note that on a fast link best results are
>> > going to
>> > occur using a DAG card as the data capture adds a fairly heavy load
>> > to the
>> > server. To display the data with ra (for instance) you need to use
>> > the -s
>> > command to add suser and duser to the output (as in
>> >
>> > ra -r argus_file -n -s +suser:512 -s +duser:512
>> >
>> > which will tack the user data on the end of the line. This of
>> course
>> > raises a
>> > number of sticky privacy issues that you need to have considered
>> and
>> > gotten
>> > approved by appropriate management of the link you are tapping
>> > (which may or
>> > may not be you :-)).
>> >
>> > Peter Van Epp
>> >
>> > What i am willing to do is to characterize the network traffic by
>> > using some characteristics derived from flow information. My final
>> > decision about a flow record will be whether the flow belongs to a
>> > chat session, a mail transfer, a FTP connection, a web
>> browsing, ...
>> >
>> > I had discovered Bro which has identifiers related with high level
>> > protocols. The protocol family that it supports is not as much as
>> > Argus does so i was planning to go on with Argus.
>> >
>> >
>> > --
>> > O?uz Yar?mtepe
>> > www.loopbacking.info
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20090203/3f7a6300/attachment.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> Argus-info mailing list
>> Argus-info at lists.andrew.cmu.edu
>> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>>
>>
>> End of Argus-info Digest, Vol 42, Issue 1
>> *****************************************
>>
>>
>>
>> --
>> Best Regards,
>>
>> CS Lee<geek00L[at]gmail.com>
>>
>> http://geek00l.blogspot.com
>> http://defcraft.net
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
> http://defcraft.net
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20090205/556d5703/attachment.html>
More information about the argus
mailing list