Problems with piping Argus output to rasplit

[Carter Bullard]

You should try a time scale like 50 or 100.
Carter


On Mar 21, 2008, at 3:44 PM, Nick Diel wrote:

> Actually I was playing with it yesterday (even though I didn't  
> exactly know what it did, wasn't sure if it was a sampling rate).
>
> I didn't have much luck, I got to a point where either rasplit died  
> or argus was sleeping way too much.  And I was using a timescale in  
> the range of .1 (quite slow I know).  Now that I know exactly what  
> the timescale does, I do some more testing.
>
> Nick
>
> Carter Bullard wrote:
>> Hey Nick,
>> One other thing you can use, is argus()'s realtime feature.  Although
>> its a little tricky.  For simulation/testing purposes, you can have  
>> argus
>> read packets at the rate they were received, or some multiple.  It  
>> maybe
>> that you can get by with reading at 50-100x realtime, which may be
>> slower than you're reading now, and that will give your rasplit(0  
>> time
>> to process the output records.  Try playing with the "-T timescale"  
>> option
>> to see if you can get good performance without hitting this  
>> threshold.
>>
>> Still a workaround, but maybe helpful.
>>
>> Carter
>>
>>
>> On Mar 21, 2008, at 3:17 PM, Nick Diel wrote:
>>
>>> Carter,
>>>
>>> I will play around with nice.
>>> I have been using intermediate files, am fortunate to have some  
>>> disk space to work with.  I am just working on streamlining the  
>>> operation as we start using Argus more and more.
>>>
>>> Nick
>>>
>>> Carter Bullard wrote:
>>>> Hey Nick
>>>> The problem is that argus closes the rasplit, because argus has  
>>>> processed
>>>> and generated more records than rasplit() has consumed, with the  
>>>> records
>>>> waiting to be read have exceeded the 100K record threshold.
>>>>
>>>> When argus is reading from the network it makes sense for argus  
>>>> to close
>>>> the reader, but when its reading from a file, it doesn't, since  
>>>> we're not in
>>>> a terrible hurry.  We should have some flow control so that it  
>>>> waits a bit
>>>> for rasplit to catch up.
>>>>
>>>> I can do something about this this weekend.  For a work around,  
>>>> maybe
>>>> nice the argus priority down, and up the priority of the  
>>>> rasplit?  maybe write
>>>> to intermediate files?  I know that probably is too much diskspace?
>>>>
>>>> Carter
>>>>
>>>> On Mar 21, 2008, at 1:55 PM, Nick Diel wrote:
>>>>
>>>>> I am having some problems piping Argus to rasplit.  I have a  
>>>>> large set of pcaps I am turning into argus records.  I have  
>>>>> about 700gigs of pcaps with a baseline of 10,000 packets/second  
>>>>> and peaks at 17,000 packets/second.
>>>>>
>>>>> I am using the following command:
>>>>> mergecap -a -w - pcap/*.pcap | argus -AJR -r - -w - | rasplit -r  
>>>>> - -w argus. -a 4 -M size 1000m
>>>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428622  
>>>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428879  
>>>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>>>>
>>>>> rasplit dies with in 30 seconds of starting (no messages to  
>>>>> stderr), though it has created a 65mb files.  racount shows the  
>>>>> 65mb file has no flows.  I can't tell if the Argus error message  
>>>>> is before or after rasplit dies.
>>>>>
>>>>> The disk system is a fairly fast raid 5, 4 cpu cores, 2 gig of  
>>>>> ram.  Barely any ram is in use when rasplit dies.
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>
>>>> Carter Bullard
>>>> CEO/President
>>>> QoSient, LLC
>>>> 150 E. 57th Street Suite 12D
>>>> New York, New York 10022
>>>>
>>>> +1 212 588-9133 Phone
>>>> +1 212 588-9134 Fax
>>>>
>>>>
>>>>
>>>
>>>
>>
>> Carter Bullard
>> CEO/President
>> QoSient, LLC
>> 150 E. 57th Street Suite 12D
>> New York, New York 10022
>>
>> +1 212 588-9133 Phone
>> +1 212 588-9134 Fax
>>
>>
>>
>
>