Traffic Profiling

Fri Feb 2 14:29:35 EST 2007

Hey Tony,
On the xml front, there were a number of sites that would use the xml to
blow data into webpages and some were doing databases, but the
performance really sucks when you get into the 10's of records a sec.

I use native mysql() library calls out of C routines and perl scripts to
do the database stuff that we do, but our strategy is not to import
huge amounts of argus data into mysql, but to poke indexes into mysql
and still use the unix filesystem files as they come off the probe.
The tools that I will transition after the argus-3.0 release are generic
enough that they could be used to put it all in the database, and we're
trying to be smart about it, so for low volume sites, it may be all  
that is
needed.

Carter

On Feb 2, 2007, at 2:14 PM, twebster at blackhillscorp.com wrote:

>
> Carter Bullard <carter at qosient.com> wrote on 02/02/2007 11:22:32 AM:
>
> > Hey Tony,
> > Don't forget the program rapolicy(), where you give it a Cisco
> > access control list and argus spits out violations.  People use
> > this to tryout new configurations before installation.  I'll have it
> > back in the argus-3.0 distribution in a week or so.  If it helps,
> > or better if it doesn't definitely send some email to the list.
>
>
> Hmm, just read about rapolicy, looks very interesting and  
> promising.  The more I learn about Argus, the more I am impressed.   
> Thanks for the great tool!
>
> I'll test it out on a 2.0 install and look forward to it be  
> included in 3.0.
>
>
> >
> > Just a data point.  raxml() hasn't been dropped, it's just going  
> to be
> > last in the list of things to do before the release.  I've almost  
> got it
> > done.
>
>
> Have you or others used raxml to import into a database for  
> querying?  Any scripts available?
>
> thank you,
> tony
>
>
> >
> > Carter
> >
> > On Feb 2, 2007, at 12:51 PM, twebster at blackhillscorp.com wrote:
> >
> >
> > "Philipp E. Letschert" <phil at uni-koblenz.de> wrote on 02/02/2007  
> 05:06:24 AM:
> >
> > > On Thu, Feb 01, 2007 at 11:29:15AM -0700,  
> twebster at blackhillscorp.comwrote:
> > > > I am currently in the process of an internal firewall  
> implementation.  I
> > > > will be implementing firewalls on all of our internal server  
> networks.  I
> > > > find the most difficult part of this project is simple "data
> > management".
> > > > How do I easily create server/port documentation so that I  
> can correctly
> > > > write firewall rules for each and every server.   I need to know
> > > > source/destination ip address and destination port/protocol  
> for every
> > > > server.  Since we do not currently have one central repository
> > documenting
> > > > every server, I am going to need to perform network  
> reconnaissance and
> > > > traffic analysis on each network.
> > > >
> > > > To begin this process, I intend to use our current Argus  
> archive to
> > > > profile traffic to/from our server network.  I need to  
> develop amethod to
> > > > query Argus for each individual IP address that is currently  
> in use and
> > > > document the port/protocol utilization for each address.
> > > >
> > > > Now, we are using Argus 3.0 and I have written several useful
> > queries that
> > > > give me the information I need, e.g. saddr, daddr, dport,  
> bytes.  The
> > > > problem I face, is how do I make this "easier".  Do I need to  
> script and
> > > > automate some of these queries?  Should I export this data into
> > a database
> > > > for other types of queries?
> > >
> > > You will need some additional scripting to generate a helpful  
> result on the
> > > port and destination usage of individual addresses. A database  
> might
> > > be helpful,
> > > but is not neccessarily needed.
> > >
> > > > 1.  Knowing that others have used Argus to profile networks,  
> what methods
> > > > work?  Did you develop any scripts to automate the process?
> > >
> > > I guess most users have developed some scripts for working with  
> their data,
> > > but these are most-likely for very specific purposes. I'm  
> afraid you have to
> > > create on your own. But this is an interesting topic for me,  
> unfortunately
> > > I dont have a ready to use script yet...
> >
> >
> > I have had little luck searching and browsing the internet for my
> > specific needs, I believe you are correct, most people are creating
> > their own solution.  I find this an interesting topic as well and
> > have always been curious how "large" data centers and such are able
> > to document their systems and more importantly, keep the  
> documents current.
> >
> >
> > >
> > > > 2.  Are there current methods for importing Argus results  
> into adatabase?
> > >
> > > None, that I know of, I'm thinking on database support for the  
> ArgusEye GUI,
> > > but this is somewhere in the future...
> > >
> > > > 3. Also, is xml an option, does raxml still exist for Argus 3.
> > >
> > > XML is not very useful for large data sets, this is probably  
> the reason
> > > why raxml dropped from 3.x?
> >
> >
> > My knowledge of XML is limited, I just thought it might be an option
> > for easy import into a database.  Thanks for the update.
> >
> > >
> > > > 4.  Any additional suggestions how to go about managing the  
> documentation
> > > > and organization of this data?
> > >
> > > One important suggestion:
> > >
> > > Automatically generating firewall rules from existing traffic  
> is always a
> > > *bad* idea. If your purpose is getting things working, just set  
> your rules
> > > on allow all. If your purpose is security there is no other way,
> > as evaluating
> > > each host (or maybe ranges of hosts with same usage e.g.  
> workstations)
> > > manually. From the recorded traffic you can then check if your
> > ruleswork (e.g.
> > > no traffic occurs that is not allowed) but not the other way  
> round.
> > >
> > > But of course an automatically generated list can help you in  
> evaluating
> > > if and *why* certain traffic is needed for a specific host.
> >
> >
> > Yes, automatically generated firewall rules would be a disaster but
> > my plan was to use network profiling via Argus, NMAP, and TCPDump to
> > create all of the allow rules with an explicit permit and then watch
> > ACL counters and custom captures to validate any ips/ports that have
> > been missed.  Then, when the explicit permit is no longer being hit,
> > change it to deny.
> >
> > For other people interested, I have found the following SANS
> > document quite useful: Firewall Analysis and Operation Modes
> > available in the SANS Reading Room
> >
> > http://www.sans.org/reading_room/whitepapers/firewalls/1659.php?
> > portal=5cda226dee2c95bcf238fc6f4266c67a
> >
> >
> > >
> > >
> > > Regards, Philipp
> >
> > thanks for the input,
> > tony

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20070202/ce233816/attachment.html>