Database design concerns

[Paul Schmehl]

That's very generous of you.  Thank you.

I'd be interested in working on the code so that it could become part of 
argus, if that's acceptable to Carter.  A web app that was tied to the 
queries he described that could search across 365 days and return data 
quickly would be extremely valuable, I would think.  And would open up 
argus to a much wider audience as well.

I'm still stunned that Carter can search that much data so quickly.  He's 
doing something I don't fully understand yet, but I will be poring over his 
site until I grasp it.  I'm convinced that argus can provide us with the 
value we haven't been getting from traditional IDS technologies.

Just like antivirus, IDS relies on knowing what the attack is before it 
occurs.  It's the old whack-a-mole game that's been played in security for 
years.

We need a new paradigm.  Argus captures everything.  If you have 
everything, you can find anything.  If you can find anything, you can find 
the thing that caused the problem.  You can see the attack step by step, as 
it happened.  That makes it much easier to design effective defenses and 
prevent repeated failures.

The problem is capturing everything means a ton of data, and you have to be 
able to search that in a reasonable amount of time or its usefulness 
decreases rapidly.  Using ra to search for things has been useful but slow. 
That's why I'm looking for a faster way to get the answers we need.

After reading Carter's two emails, I realized that I don't understand much 
at all about argus.  He's doing things I didn't even realize are possible. 
So if I can setup our stuff so it works like his AND present the results in 
a web app, THAT would be nirvana.

--On October 26, 2012 8:42:38 PM -0400 Mark Bartlett <mabartle at gmail.com> 
wrote:

>
> Hey Paul,
>
> I wrote a web gui for argus in php before Carter had added the db
> component to argus (I had created my own mysql db and was inserting
> batches of argus data).  I would be more than willing to offer up what I
> have done and you can change the select statements to match the db schema
> which argus resides now.  Let me know if you are interested.
>
> Bartlett
> On Oct 26, 2012 6:05 PM, "Paul Schmehl" <pschmehl_lists at tx.rr.com> wrote:
>
> --On October 26, 2012 3:11:25 PM -0400 Carter Bullard
> <carter at qosient.com> wrote:
>
>
> Hey Paul,
> No problem at all !!!!!  If there is a problem, its that we don't have
> enough
> documentation on this topic for you to read.  I was excited that you
> brought
> up the topics that you did, as I'm hoping that it will generate some
> thoughts
> on the " first pass " support we put into argus-3.0.4+.
>
>
> I think the best thing for you to do is to keep doing what you're doing,
> and
> if there is anything I can help you with, I'm happy to do so.  I still
> think you're
> onto something with the partitions, but I don't know enough about them to
> know if they are a distraction, or something useful.
>
>
>
> Here's some basics I've learned over the past 72 hours.  Partitions
> break up a table into lots of mini-tables.  They don't recommend having
> more than 50 partitions on a table.  Partitions make sense if the bulk
> of the selects will only search one partition.  In the case of argus
> data this would make sense because a lot of queries are done against time
> frames less than 24 hours in size.  Even if a search crossed daily
> boundaries, it would usually only require searching two partitions.
>
> However, it sounds like your approach has already solved the problem of
> searching massive amounts of data quickly, so I'm not sure partitions
> would improve performance.  In fact they may be more trouble than
> they're worth.
>
> My problem is I don't yet understand what you're doing and how all the
> various utilities you've written tie into the greater picture.
>
> The other issue I have is that our department is expanding, and some of
> the newer analysts don't have the same level of expertise that I do with
> unix commandline apps.  So I was looking for a way to "webify" argus
> data and searches so that the analysts can use an interface they're more
> familiar with than bash.
>
>
> I just don't want you to think that argus is half done.
>
>
> You and I both know that many open source apps leave much to be desired.
> You often have to cobble together several different apps to create
> something worthwhile for daily use by graphics-demanding younger folks.
>
> I realize now, after your explanations, that is not the case with argus.
> I'm going to have to pore through the docs to figure out how I can do
> what you're doing without bugging the daylights out of you.  If I can
> put a web front-end on it, so much the better.
>
>
> I'm thinking its getting done, but you never know how people like their
>
> software cooked ;o)
>
>
>
>
> There's often a dichotomy between what the users want and what software
> gives them.  I frequently wonder why some software is so counter
> intuitive and why features you would think would be there aren't.  I
> think the gap is at least partly explained by the different thought
> patterns of people who design software and those who use it.
>
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *******************************************
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
>
>
>



Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
*******************************************
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell