Problems with piping Argus output to rasplit

[Carter Bullard]

Hey Nick,
One other thing you can use, is argus()'s realtime feature.  Although
its a little tricky.  For simulation/testing purposes, you can have  
argus
read packets at the rate they were received, or some multiple.  It maybe
that you can get by with reading at 50-100x realtime, which may be
slower than you're reading now, and that will give your rasplit(0 time
to process the output records.  Try playing with the "-T timescale"  
option
to see if you can get good performance without hitting this threshold.

Still a workaround, but maybe helpful.

Carter


On Mar 21, 2008, at 3:17 PM, Nick Diel wrote:

> Carter,
>
> I will play around with nice.
> I have been using intermediate files, am fortunate to have some disk  
> space to work with.  I am just working on streamlining the operation  
> as we start using Argus more and more.
>
> Nick
>
> Carter Bullard wrote:
>> Hey Nick
>> The problem is that argus closes the rasplit, because argus has  
>> processed
>> and generated more records than rasplit() has consumed, with the  
>> records
>> waiting to be read have exceeded the 100K record threshold.
>>
>> When argus is reading from the network it makes sense for argus to  
>> close
>> the reader, but when its reading from a file, it doesn't, since  
>> we're not in
>> a terrible hurry.  We should have some flow control so that it  
>> waits a bit
>> for rasplit to catch up.
>>
>> I can do something about this this weekend.  For a work around, maybe
>> nice the argus priority down, and up the priority of the rasplit?   
>> maybe write
>> to intermediate files?  I know that probably is too much diskspace?
>>
>> Carter
>>
>> On Mar 21, 2008, at 1:55 PM, Nick Diel wrote:
>>
>>> I am having some problems piping Argus to rasplit.  I have a large  
>>> set of pcaps I am turning into argus records.  I have about  
>>> 700gigs of pcaps with a baseline of 10,000 packets/second and  
>>> peaks at 17,000 packets/second.
>>>
>>> I am using the following command:
>>> mergecap -a -w - pcap/*.pcap | argus -AJR -r - -w - | rasplit -r -  
>>> -w argus. -a 4 -M size 1000m
>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428622  
>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428879  
>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>>
>>> rasplit dies with in 30 seconds of starting (no messages to  
>>> stderr), though it has created a 65mb files.  racount shows the  
>>> 65mb file has no flows.  I can't tell if the Argus error message  
>>> is before or after rasplit dies.
>>>
>>> The disk system is a fairly fast raid 5, 4 cpu cores, 2 gig of  
>>> ram.  Barely any ram is in use when rasplit dies.
>>>
>>> Any thoughts?
>>>
>>> Nick
>>>
>>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E. 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>
>

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax