rastream fails

Thu Jun 19 11:43:07 EDT 2014

You said that it took 2 days or so to have problems, so it seems that the problem is transient, and the issue is going to be at these transient times.

is rastream on the same box ??  if not how is the network connection between the argus server and the rastream ???  is it 1 Gbps ??
run an argus on your rastream system, so you can figure out if your path is adequate.  is there loss ??

> On Jun 19, 2014, at 10:42 AM, James Grace <jgrac002 at fiu.edu> wrote:
> 
> Hi Carter, 
> 
> As of right now we have a Netoptics 80/20 tap sitting between two 10Gb/s ports, but traffic rarely peaks over 2Gb/s. 
> 
> We have a DAG 8.1SX card with 128Mb of memory doing the heavy lifting and argus attaches itself to this card.
> 
> There is only one argus instance running on this box, and only one rastream client is connecting to the argus server. 
> 
> CPU utilization for argus: 
> 
> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
> 
>  21332 root      20   0 8625m 8.3g 100m S 17.9 71.3   3199:44 argus 
> 
> CPU utilization for rastream:
> 
> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
> 
> 30099 root      20   0 35424 5244 1100 S  6.0  0.0   0:01.39 rastream
> 
> I haven't shortened the record length, so perhaps shortening it down to 82 bytes -- just the header will help.  Thoughts?
> 
> 
> 
> Thanks, 
> 
> -james
> 
> 
> 
> 
> 
> 
>> On Thu, Jun 19, 2014 at 10:22 AM, Carter Bullard <carter at qosient.com> wrote:
>> its queuing up buffers because the consumer isn't running fast enough, or the transport path isn't adequate.  
>> 
>> You need to describe your setup ..
>> so what are you trying to do ??  how many clients ??
>> how many records a second are you generating?? 
>> is the client running properly ???  
>> is the processor pegged out on the clients.
>> loss on the link ????
>> that kind of thing...
>> 
>>> On Jun 19, 2014, at 10:09 AM, James Grace <jgrac002 at fiu.edu> wrote:
>>> 
>>> Hi Carter, 
>>> Thanks for all the help so far. Here are some error messages I've seen when rastream fails.  Looks like my argus server's buffer is filling up:
>>> 
>>> Jun 15 22:49:41 coralreef argus[21332]: 15 Jun 14 22:49:41.001702 ArgusNewFlow() flow key is not correct len equals zero
>>> 
>>> Jun 17 13:21:26 coralreef argus[21332]: 17 Jun 14 13:21:26.134936 ArgusWriteOutSocket(0x7fb6ae016950) max queue exceeded 100001
>>> 
>>> Jun 17 13:21:26 coralreef argus[21332]: 17 Jun 14 13:21:26.138624 ArgusWriteOutSocket(0x7fb6ae016950) max queue exceeded 100001
>>> 
>>> Jun 18 22:54:34 coralreef argus[21332]: 18 Jun 14 22:54:34.557649 ArgusWriteOutSocket(0x7fb6ac0008c0) max queue exceeded 100001
>>> 
>>> 
>>> 
>>> Any tips or tricks to keep the buffer in tact?  
>>> 
>>> 
>>> 
>>> Thanks, 
>>> 
>>> -james
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On Wed, Jun 18, 2014 at 12:20 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> Should be something like (for bash).
>>>> 
>>>>    rastream  options > /home/flows/rastream.err 2>&1
>>>> 
>>>> The D8 maybe too much into, try D4…
>>>> 
>>>> Carter
>>>> 
>>>>> On Jun 18, 2014, at 11:29 AM, James Grace <jgrac002 at fiu.edu> wrote:
>>>>> 
>>>>> Hi Carter, 
>>>>> 
>>>>> I didn't see anything in any of my logs.  The rastream command I'm running is:
>>>>> 
>>>>> rastream -S localhost -M time 5m -w /home/flows/South/%Y/%m/%d/argus.%H.%M -D 8 2>/home/flows/rastream.err &
>>>>> 
>>>>> 
>>>>> 
>>>>> Will this capture the exceptions thrown to stderr appropriately? Or is there a more 'argus-y' way of doing this?
>>>>> 
>>>>> 
>>>>> 
>>>>> Thanks, 
>>>>> 
>>>>> -james
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Wed, Jun 11, 2014 at 11:24 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> Hey James,
>>>>>> rastream doesn’t write out these types of messages, so its a little weird.
>>>>>> Seems like this maybe coming from the script that you’ve got rastream running ???
>>>>>> 
>>>>>> rastream will exit for all the standard reasons that a ra* program
>>>>>> exits for, such as the argus data source hanging up, internal
>>>>>> failure, or in rastream’s case, if there is a major problem forking
>>>>>> the script that it is suppose to run. It should generate syslog messages
>>>>>> to indicate any errors.  Anything in your system log ???
>>>>>> 
>>>>>> 
>>>>>> Carter
>>>>>> 
>>>>>>> On Jun 11, 2014, at 10:07 AM, James Grace <jgrac002 at fiu.edu> wrote:
>>>>>>> 
>>>>>>> Good morning all, 
>>>>>>> 
>>>>>>> I've been using rastream to connect to my argus client to split up traces into 5 minute interval and store them neatly.
>>>>>>> 
>>>>>>> I've been running into a problem after about 48 hours of operation where the rastream process closes. I finally ran it with -D 3 and piped the output to a file so here's the error message I'm receiving:
>>>>>>> 
>>>>>>> Write failed: Broken pipe
>>>>>>> 
>>>>>>> Time spent in user mode   (CPU seconds) : 8.348s
>>>>>>> 
>>>>>>> Time spent in kernel mode (CPU seconds) : 5.514s
>>>>>>> 
>>>>>>> Total time                              : 47:31:15.22s
>>>>>>> 
>>>>>>> CPU utilisation (percentage)            : 0.0%
>>>>>>> 
>>>>>>> Times the process was swapped           : 0
>>>>>>> 
>>>>>>> Times of major page faults              : 74
>>>>>>> 
>>>>>>> Times of minor page faults              : 1583
>>>>>>> 
>>>>>>> Exit 255
>>>>>>> 
>>>>>>> Any clues on this?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> thanks much, 
>>>>>>> 
>>>>>>> -james
>>>>>>> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140619/31ed67ad/attachment.html>