Problems with piping Argus output to rasplit

[Nick Diel]

Actually I was playing with it yesterday (even though I didn't exactly 
know what it did, wasn't sure if it was a sampling rate).

I didn't have much luck, I got to a point where either rasplit died or 
argus was sleeping way too much.  And I was using a timescale in the 
range of .1 (quite slow I know).  Now that I know exactly what the 
timescale does, I do some more testing.

Nick

Carter Bullard wrote:
> Hey Nick,
> One other thing you can use, is argus()'s realtime feature.  Although
> its a little tricky.  For simulation/testing purposes, you can have argus
> read packets at the rate they were received, or some multiple.  It maybe
> that you can get by with reading at 50-100x realtime, which may be
> slower than you're reading now, and that will give your rasplit(0 time
> to process the output records.  Try playing with the "-T timescale" 
> option
> to see if you can get good performance without hitting this threshold.
>
> Still a workaround, but maybe helpful.
>
> Carter
>
>
> On Mar 21, 2008, at 3:17 PM, Nick Diel wrote:
>
>> Carter,
>>
>> I will play around with nice.
>> I have been using intermediate files, am fortunate to have some disk 
>> space to work with.  I am just working on streamlining the operation 
>> as we start using Argus more and more.
>>
>> Nick
>>
>> Carter Bullard wrote:
>>> Hey Nick
>>> The problem is that argus closes the rasplit, because argus has 
>>> processed
>>> and generated more records than rasplit() has consumed, with the 
>>> records
>>> waiting to be read have exceeded the 100K record threshold.
>>>
>>> When argus is reading from the network it makes sense for argus to 
>>> close
>>> the reader, but when its reading from a file, it doesn't, since 
>>> we're not in
>>> a terrible hurry.  We should have some flow control so that it waits 
>>> a bit
>>> for rasplit to catch up.
>>>
>>> I can do something about this this weekend.  For a work around, maybe
>>> nice the argus priority down, and up the priority of the rasplit?  
>>> maybe write
>>> to intermediate files?  I know that probably is too much diskspace?
>>>
>>> Carter
>>>
>>> On Mar 21, 2008, at 1:55 PM, Nick Diel wrote:
>>>
>>>> I am having some problems piping Argus to rasplit.  I have a large 
>>>> set of pcaps I am turning into argus records.  I have about 700gigs 
>>>> of pcaps with a baseline of 10,000 packets/second and peaks at 
>>>> 17,000 packets/second.
>>>>
>>>> I am using the following command:
>>>> mergecap -a -w - pcap/*.pcap | argus -AJR -r - -w - | rasplit -r - 
>>>> -w argus. -a 4 -M size 1000m
>>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428622 
>>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>>> ArgusWarning: argus[21937]: 20 Mar 08 11:36:52.428879 
>>>> ArgusWriteOutSocket(0xb7eaf008) max queue exceeded 100001
>>>>
>>>> rasplit dies with in 30 seconds of starting (no messages to 
>>>> stderr), though it has created a 65mb files.  racount shows the 
>>>> 65mb file has no flows.  I can't tell if the Argus error message is 
>>>> before or after rasplit dies.
>>>>
>>>> The disk system is a fairly fast raid 5, 4 cpu cores, 2 gig of 
>>>> ram.  Barely any ram is in use when rasplit dies.
>>>>
>>>> Any thoughts?
>>>>
>>>> Nick
>>>>
>>>>
>>>
>>> Carter Bullard
>>> CEO/President
>>> QoSient, LLC
>>> 150 E. 57th Street Suite 12D
>>> New York, New York 10022
>>>
>>> +1 212 588-9133 Phone
>>> +1 212 588-9134 Fax
>>>
>>>
>>>
>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>