Problems with piping Argus output to rasplit

[Nick Diel]

Carter,

I will play around with nice. 

I have been using intermediate files, am fortunate to have some disk 
space to work with.  I am just working on streamlining the operation as 
we start using Argus more and more.

Nick

Carter Bullard wrote:
> Hey Nick
> The problem is that argus closes the rasplit, because argus has processed
> and generated more records than rasplit() has consumed, with the records
> waiting to be read have exceeded the 100K record threshold.
>
> When argus is reading from the network it makes sense for argus to close
> the reader, but when its reading from a file, it doesn't, since we're 
> not in
> a terrible hurry.  We should have some flow control so that it waits a 
> bit
> for rasplit to catch up.
>
> I can do something about this this weekend.  For a work around, maybe
> nice the argus priority down, and up the priority of the rasplit?  
> maybe write
> to intermediate files?  I know that probably is too much diskspace?
>
> Carter
>
> On Mar 21, 2008, at 1:55 PM, Nick Diel wrote:
>
>> I am having some problems piping Argus to rasplit.  I have a large 
>> set of pcaps I am turning into argus records.  I have about 700gigs 
>> of pcaps with a baseline of 10,000 packets/second and peaks at 17,000 
>> packets/second.
>>
>> I am using the following command:
>> mergecap -a -w - pcap/*.pcap | argus -AJR -r - -w - | rasplit -r - -w 
>> argus. -a 4 -M size 1000m
>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428622 
>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428879 
>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>
>> rasplit dies with in 30 seconds of starting (no messages to stderr), 
>> though it has created a 65mb files.  racount shows the 65mb file has 
>> no flows.  I can't tell if the Argus error message is before or after 
>> rasplit dies.
>>
>> The disk system is a fairly fast raid 5, 4 cpu cores, 2 gig of ram.  
>> Barely any ram is in use when rasplit dies.
>>
>> Any thoughts?
>>
>> Nick
>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>