Problems with piping Argus output to rasplit

[Carter Bullard]

Hey Nick
The problem is that argus closes the rasplit, because argus has  
processed
and generated more records than rasplit() has consumed, with the records
waiting to be read have exceeded the 100K record threshold.

When argus is reading from the network it makes sense for argus to close
the reader, but when its reading from a file, it doesn't, since we're  
not in
a terrible hurry.  We should have some flow control so that it waits a  
bit
for rasplit to catch up.

I can do something about this this weekend.  For a work around, maybe
nice the argus priority down, and up the priority of the rasplit?   
maybe write
to intermediate files?  I know that probably is too much diskspace?

Carter

On Mar 21, 2008, at 1:55 PM, Nick Diel wrote:

> I am having some problems piping Argus to rasplit.  I have a large  
> set of pcaps I am turning into argus records.  I have about 700gigs  
> of pcaps with a baseline of 10,000 packets/second and peaks at  
> 17,000 packets/second.
>
> I am using the following command:
> mergecap -a -w - pcap/*.pcap | argus -AJR -r - -w - | rasplit -r - - 
> w argus. -a 4 -M size 1000m
> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428622  
> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428879  
> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>
> rasplit dies with in 30 seconds of starting (no messages to stderr),  
> though it has created a 65mb files.  racount shows the 65mb file has  
> no flows.  I can't tell if the Argus error message is before or  
> after rasplit dies.
>
> The disk system is a fairly fast raid 5, 4 cpu cores, 2 gig of ram.   
> Barely any ram is in use when rasplit dies.
>
> Any thoughts?
>
> Nick
>
>

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax