rasqlinsert crashing

[Carter Bullard]

Hey Mike,
You are running rasqlinsert() so that it just inserts primitive argus data into a table.
This will generate the least performing database strategy, and no, for very large
feeds, writing primitive data into the database is not advised.

So do you know how many insertions you are trying to do per second?  If its over
1-2,000 then its probably not going to work.  That will be the best guide.    

I use the database to hold specific realtime views of the data stream, such as
the IP matrix, and a list of all the IP addresses seen.  I want a fast system for the
data I will look at the most frequently, and when I need more, I go to the primitive
data.

rasqlinsert() is an aggregator, just like ratop(), and the logic ratop() uses for buffering
data and scheduling updates to the curses screen, is the same logic for regulating
updates to the database table, but you are not using any of these features.

Lots of places to make the code better/more efficient, no debate about that.
I'll fix the signal() problem.

Carter

On Apr 21, 2010, at 2:24 PM, Mike wrote:

> carter at qosient.com wrote:
>> Hey Mike,
>> For mysql support, you should use the development version of the clients, as there have been quite a number of bug fixes there.
>>   Http://qosient.com/argus/dev
>> The rasqlinsert() ihere does what you would like.
>> Carter 
> 
> Thank you for the pointer, seems more stable now. Using rasqlinsert with
> the %Y_$m_%d is handy for keeping daily flow tables, and man do they get
> big I had no idea we had so many flows. My tables are multi-gigabyte
> already, and most sql queries I run against it are revealing peformance
> problems I never thought about. In one instance, a simple perl script
> using DBI::Mysql just hangs every time on "prepare("SELECT
> saddr,daddr,bytes ..." and grows to monstrous virtual memory
> proportions. And I also notice there's no indexes in the autogenerated
> tables of rasqlinsert, which I assume (?) is intentional for performance
> reasons?
> 
> 	The performance of my host, under argus/rasqlinsert, seems awfully bad
> and out of line with my experience in other mysql based apps even with
> big tables like this. I have been doing some digging trying to see about
> why this might be so and so far I haven't come up with much. One issue I
> did notice however, was that rasqlinsert seems to have some calls to
> localtime/getlocaltime in the wrong place, resulting in about 170 calls
> to stat64(/etc/localtime) and gettimeofday for every argus packet
> received. It also seems to draw too much in from ratop, which may have
> helped speed development but which also makes it more difficult to
> follow. It also has trouble (all argus client progs do) with dying when
> sent a kill, necessitating a the magic -9 sword. Of course, not to look
> a gift horse in the you know what, I am wondering wether sql really is
> going to be preferable to straight argus
> files for large #'s of flows?
> 
> Mike-
> 
> 
> 

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100422/188b8eb1/attachment.bin>