Database design concerns

Sat Oct 27 11:01:29 EDT 2012

Hey Paul,
Glad to see that you're on my track with regard to how to do something completely different.

There are a lot of reasons to provide network situational awareness other than IDS, and
Argus is trying to provide a network activity awareness data model that can support a lot
of functions, performance analysis, operations management and security.

There are huge benefits taking this approach.  As network link speeds increase, the
network activity awareness processing has to scale, but the other functions, because
they are down stream, not integrated into the packet processing, don't have to scale to
the same degree.

OK, so with regard to finding things quickly, with a lot of pre-processing, you can get to
the answer of a query without having to process a lot of primitive data, each time.  Each
day table of unique IP addresses, in my situation, holds about 30K of entries, so to
discover if an IP address is actually in the data, isn't expensive at all.   That is 30K
unique IP addrs, against many GB of flow data per day.  For large enterprises, the
number of unique IP addresses is in the 400-600K addresses a day.

Because the IP address table has a lot of utility, its worth generating and maintaining
in real time, but also worth keeping around forever.

Interesting, the number of matrix entries isn't much different, as the interest in foreign
addresses isn't shared among a large number of local entities.  The matrix data, I've
found, runs around 1.25x the unique addresses.  But with NAT, its only 1x, as all the
foreign addresses are talking to only one IP address.

Carter

On Oct 26, 2012, at 9:41 PM, Paul Schmehl <pschmehl_lists at tx.rr.com> wrote:

> That's very generous of you.  Thank you.
> 
> I'd be interested in working on the code so that it could become part of argus, if that's acceptable to Carter.  A web app that was tied to the queries he described that could search across 365 days and return data quickly would be extremely valuable, I would think.  And would open up argus to a much wider audience as well.
> 
> I'm still stunned that Carter can search that much data so quickly.  He's doing something I don't fully understand yet, but I will be poring over his site until I grasp it.  I'm convinced that argus can provide us with the value we haven't been getting from traditional IDS technologies.
> 
> Just like antivirus, IDS relies on knowing what the attack is before it occurs.  It's the old whack-a-mole game that's been played in security for years.
> 
> We need a new paradigm.  Argus captures everything.  If you have everything, you can find anything.  If you can find anything, you can find the thing that caused the problem.  You can see the attack step by step, as it happened.  That makes it much easier to design effective defenses and prevent repeated failures.
> 
> The problem is capturing everything means a ton of data, and you have to be able to search that in a reasonable amount of time or its usefulness decreases rapidly.  Using ra to search for things has been useful but slow. That's why I'm looking for a faster way to get the answers we need.
> 
> After reading Carter's two emails, I realized that I don't understand much at all about argus.  He's doing things I didn't even realize are possible. So if I can setup our stuff so it works like his AND present the results in a web app, THAT would be nirvana.
> 
> --On October 26, 2012 8:42:38 PM -0400 Mark Bartlett <mabartle at gmail.com> wrote:
> 
>> 
>> Hey Paul,
>> 
>> I wrote a web gui for argus in php before Carter had added the db
>> component to argus (I had created my own mysql db and was inserting
>> batches of argus data).  I would be more than willing to offer up what I
>> have done and you can change the select statements to match the db schema
>> which argus resides now.  Let me know if you are interested.
>> 
>> Bartlett
>> On Oct 26, 2012 6:05 PM, "Paul Schmehl" <pschmehl_lists at tx.rr.com> wrote:
>> 
>> --On October 26, 2012 3:11:25 PM -0400 Carter Bullard
>> <carter at qosient.com> wrote:
>> 
>> 
>> Hey Paul,
>> No problem at all !!!!!  If there is a problem, its that we don't have
>> enough
>> documentation on this topic for you to read.  I was excited that you
>> brought
>> up the topics that you did, as I'm hoping that it will generate some
>> thoughts
>> on the " first pass " support we put into argus-3.0.4+.
>> 
>> 
>> I think the best thing for you to do is to keep doing what you're doing,
>> and
>> if there is anything I can help you with, I'm happy to do so.  I still
>> think you're
>> onto something with the partitions, but I don't know enough about them to
>> know if they are a distraction, or something useful.
>> 
>> 
>> 
>> Here's some basics I've learned over the past 72 hours.  Partitions
>> break up a table into lots of mini-tables.  They don't recommend having
>> more than 50 partitions on a table.  Partitions make sense if the bulk
>> of the selects will only search one partition.  In the case of argus
>> data this would make sense because a lot of queries are done against time
>> frames less than 24 hours in size.  Even if a search crossed daily
>> boundaries, it would usually only require searching two partitions.
>> 
>> However, it sounds like your approach has already solved the problem of
>> searching massive amounts of data quickly, so I'm not sure partitions
>> would improve performance.  In fact they may be more trouble than
>> they're worth.
>> 
>> My problem is I don't yet understand what you're doing and how all the
>> various utilities you've written tie into the greater picture.
>> 
>> The other issue I have is that our department is expanding, and some of
>> the newer analysts don't have the same level of expertise that I do with
>> unix commandline apps.  So I was looking for a way to "webify" argus
>> data and searches so that the analysts can use an interface they're more
>> familiar with than bash.
>> 
>> 
>> I just don't want you to think that argus is half done.
>> 
>> 
>> You and I both know that many open source apps leave much to be desired.
>> You often have to cobble together several different apps to create
>> something worthwhile for daily use by graphics-demanding younger folks.
>> 
>> I realize now, after your explanations, that is not the case with argus.
>> I'm going to have to pore through the docs to figure out how I can do
>> what you're doing without bugging the daylights out of you.  If I can
>> put a web front-end on it, so much the better.
>> 
>> 
>> I'm thinking its getting done, but you never know how people like their
>> 
>> software cooked ;o)
>> 
>> 
>> 
>> 
>> There's often a dichotomy between what the users want and what software
>> gives them.  I frequently wonder why some software is so counter
>> intuitive and why features you would think would be there aren't.  I
>> think the gap is at least partly explained by the different thought
>> patterns of people who design software and those who use it.
>> 
>> Paul Schmehl, Senior Infosec Analyst
>> As if it wasn't already obvious, my opinions
>> are my own and not those of my employer.
>> *******************************************
>> "It is as useless to argue with those who have
>> renounced the use of reason as to administer
>> medication to the dead." Thomas Jefferson
>> "There are some ideas so wrong that only a very
>> intelligent person could believe in them." George Orwell
>> 
>> 
>> 
> 
> 
> 
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *******************************************
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20121027/e7b53216/attachment.bin>