Database design concerns

[Paul Schmehl]

--On October 26, 2012 3:11:25 PM -0400 Carter Bullard <carter at qosient.com> 
wrote:

> Hey Paul,
> No problem at all !!!!!  If there is a problem, its that we don't have
> enough
> documentation on this topic for you to read.  I was excited that you
> brought
> up the topics that you did, as I'm hoping that it will generate some
> thoughts
> on the " first pass " support we put into argus-3.0.4+.
>
>
> I think the best thing for you to do is to keep doing what you're doing,
> and
> if there is anything I can help you with, I'm happy to do so.  I still
> think you're
> onto something with the partitions, but I don't know enough about them to
> know if they are a distraction, or something useful.
>

Here's some basics I've learned over the past 72 hours.  Partitions break 
up a table into lots of mini-tables.  They don't recommend having more than 
50 partitions on a table.  Partitions make sense if the bulk of the selects 
will only search one partition.  In the case of argus data this would make 
sense because a lot of queries are done against time frames less than 24 
hours in size.  Even if a search crossed daily boundaries, it would usually 
only require searching two partitions.

However, it sounds like your approach has already solved the problem of 
searching massive amounts of data quickly, so I'm not sure partitions would 
improve performance.  In fact they may be more trouble than they're worth.

My problem is I don't yet understand what you're doing and how all the 
various utilities you've written tie into the greater picture.

The other issue I have is that our department is expanding, and some of the 
newer analysts don't have the same level of expertise that I do with unix 
commandline apps.  So I was looking for a way to "webify" argus data and 
searches so that the analysts can use an interface they're more familiar 
with than bash.
>
> I just don't want you to think that argus is half done.

You and I both know that many open source apps leave much to be desired. 
You often have to cobble together several different apps to create 
something worthwhile for daily use by graphics-demanding younger folks.

I realize now, after your explanations, that is not the case with argus. 
I'm going to have to pore through the docs to figure out how I can do what 
you're doing without bugging the daylights out of you.  If I can put a web 
front-end on it, so much the better.

> I'm thinking its getting done, but you never know how people like their 
software cooked ;o)
>

There's often a dichotomy between what the users want and what software 
gives them.  I frequently wonder why some software is so counter intuitive 
and why features you would think would be there aren't.  I think the gap is 
at least partly explained by the different thought patterns of people who 
design software and those who use it.

Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
*******************************************
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell