argus/radium problem

Fri Jun 8 14:23:35 EDT 2012

So lets get argus running in the background, with a radium running on your linux
box connecting to it, and then lets fix the radium so it doesn't hurt the argus if / when
it disconnects clients or has congestion issues.  In this situation, radium is reading
all the argus records from the bivio; and radium is there to protect the argus,
from the congestion issues that ra* clients can generate. 

With radium running on the linux box, we should be able to debug it with a lot of
visibility.  Once we get the argus -> radium running, then lets test having another
radium connect to the radium, and then having your ra* clients connect to the
2nd radium.  This should help us to understand how the back flow impacts
both radii.

Carter

On Jun 8, 2012, at 1:52 PM, CS Lee wrote:

> hi Carter,
> 
> Currently it is very hard for me to run argus in foreground in this bivio box, if I push to one cpu and run argus in bivio, it got killed after I connect argus client to it. I have sent you the message in previous email. The argus keeps getting killed message once I attach argus client for a little while.
> 
> 
> On Sat, Jun 9, 2012 at 12:58 AM, Carter Bullard <carter at qosient.com> wrote:
> Hey CS Lee,
> Just going through the messages you sent, so we have some dialog on what is going on.
> 
> OK, so this is the most important to figuring out the argus fatal issues.  After deleting the socket that the client
> was using, somehow the modeler looks to have been re-initialized, or deleted.  This is not good.
> The model->ArgusOutputList is the communication channel from the ArgusModeler to the ArgusOutput
> thread, so if this goes away, broken argus.
> 
> The ArgusModeler->ArgusOutputList goes away only when the something calls ArgusCloseOutput().
> Any chance you can verify, running argus with a "-D 2" option, that in this situation, ArgusCloseOutput()
> is called?
> 
> Carter 
> 
> On Jun 7, 2012, at 11:58 PM, CS Lee wrote:
> 
>> hi Carter,
>> 
>> I run ra in follwoing way and after ra has retrieved 100 records, it exits, however I can't reconnect to argus again -
>> 
>> ra -N 100 -Z b -S 10.0.0.1:561 -s saddr sport daddr flgs proto state
>> 
>> This is the output I get from argus 
>> 
>> argus[27489.48c93490]: 08 Jun 12 10:23:07.144707 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
>> argus[27489.48c93490]: 08 Jun 12 10:23:07.144726 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
>> argus[27489.48c93490]: 08 Jun 12 10:23:07.144786 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
>> argus[27489.48c93490]: 08 Jun 12 10:23:07.144804 ArgusDeleteSocket: list not empty
>> argus[27489.48c93490]: 08 Jun 12 10:23:07.144909 ArgusDeleteSocket (0x58300008) returning
>> argus[27489.498dd490]: 08 Jun 12 10:26:18.708883 ArgusSendFlowRecord (0x1009b008, 0x57595818, 64) no model->ArgusOutputList
>> argus[27489.498dd490]: 08 Jun 12 10:26:19.110344 ArgusSendFlowRecord (0x1009b008, 0x527033d8, 64) no model->ArgusOutputList
>> 
>> If I reconnect to argus again, I get connection refused message
>> 
>> ra[20656]: 10:26:56.109970 connect to 10.0.0.1:561 failed 'Connection timed out'
>> 
>> Thank you.
>> 
>> 
>> 
>> 
>> On Fri, Jun 8, 2012 at 10:12 AM, CS Lee <geek00l at gmail.com> wrote:
>> hi Carter,
>> 
>> I don't know if bivio team has compiled it with threaded support, need to ask them(joel in bivio) if you know him. 
>> 
>> Here's the argus output message when radium connects to it -
>> 
>> ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x32a5da00 msgnum 0x66573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
>> argus[27125.498dd490]: 08 Jun 12 08:26:57.178068 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x329ee15a msgnum 0x31573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
>> argus[27125.498dd490]: 08 Jun 12 08:26:57.196113 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x321c61e1 msgnum 0xac573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
>> argus[27125.498dd490]: 08 Jun 12 08:26:57.220138 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x3227377b msgnum 0xba573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
>> argus[27125.498dd490]: 08 Jun 12 08:26:57.293605 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x32ac4271 msgnum 0x59573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
>> argus[27125.48c93490]: 08 Jun 12 08:26:58.039943 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
>> argus[27125.48c93490]: 08 Jun 12 08:26:58.040039 ArgusFreeListRecord (0x100f679c) returning
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.243664 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.243759 ArgusFreeListRecord (0x100f6b44) returning
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.413949 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.414030 ArgusFreeListRecord (0x100f6eec) returning
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.767378 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
>> argus[27125.48c93490]: 08 Jun 12 08:26:59.767474 ArgusFreeListRecord (0x100f7294) returning
>> argus[27125.48c93490]: 08 Jun 12 08:27:00.112948 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
>> argus[27125.48c93490]: 08 Jun 12 08:27:00.113028 ArgusFreeListRecord (0x100f763c) returning
>> 
>> Then after a while, this appears -
>> 
>> ArgusGenerateInitialMar() returning
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083154 ArgusCheckClientStatus: wrote 128 bytes to client
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083175 ArgusCheckClientStatus() returning
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083192 ArgusLoadList (0x1009b9c0, 0x1009b948) load 19823 objects
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083209 ArgusOutputProcess() 1 client(s) for record 0x4ce4e2a4
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083226 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083244 ArgusFreeListRecord (0x4ce4e2a4) returning
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083260 ArgusOutputProcess() 1 client(s) for record 0x4ce4e64c
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083276 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083293 ArgusFreeListRecord (0x4ce4e64c) returning
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083310 ArgusOutputProcess() 1 client(s) for record 0x4ce4e9f4
>> argus[27125.48c93490]: 08 Jun 12 08:27:09.083326 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
>> 
>> 
>> On Fri, Jun 8, 2012 at 10:02 AM, Carter Bullard <carter at qosient.com> wrote:
>> Well, if it's getting write errors we should be shutting down the connection and deallocating any queued records.  This is useful.  Let me see what I can do with this.
>> 
>> Carter
>> 
>> Sent from my iPad
>> 
>> On Jun 7, 2012, at 9:19 PM, CS Lee <geek00l at gmail.com> wrote:
>> 
>>> hi Carter,
>>> 
>>> Bivio team has sent me the argus rpm with debug enabled, here's what I get from argus -D 5 when argus client sees the idle stream message -
>>> 
>>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594670 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594696 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594723 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> argus[26916.498dd490]: 08 Jun 12 07:20:28.600001 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.48c93490]: 08 Jun 12 07:20:28.603302 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> argus[26916.48c93490]: 08 Jun 12 07:20:28.603332 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> 
>>> argus[26916.48c93490]: 08 Jun 12 07:20:33.554648 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.554671 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563056 ArgusMallocListRecord (904) returned NULL
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563076 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563092 ArgusMallocListRecord (904) returned NULL
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563108 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563124 ArgusMallocListRecord (904) returned NULL
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563140 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563157 ArgusMallocListRecord (904) returned NULL
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563173 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563189 ArgusMallocListRecord (904) returned NULL
>>> 
>>> Argus keeps running however no client can connect to it and retrieve anything.
>>> 
>>> On Fri, Jun 8, 2012 at 2:54 AM, CS Lee <geek00l at gmail.com> wrote:
>>> hi Carter,
>>> 
>>> I able to get argus running in foreground now, however since the rpm they provide doesn't compile with debug enable, I can only see limited output, here's where it breaks.
>>> 
>>> Argus client on linux box
>>> ra[32718]: 01:07:11.134418 ArgusReadStream 10.0.0.1: idle stream: closing
>>> 
>>> Argus on bivio
>>> argus -D 3 -i bond:s0.e0,s0.e1 -mAJZRU 64 -B 10.0.0.1 -P 561
>>> argus[1611]: 08 Jun 12 01:02:46.781824 started
>>> argus[1611]: 08 Jun 12 01:02:46.800912 ArgusGetInterfaceStatus: interface s0.e1 is up
>>> argus[1611]: 08 Jun 12 01:02:46.801064 ArgusGetInterfaceStatus: interface s0.e0 is up
>>> argus[1611]: 08 Jun 12 01:04:15.489853 connect from 10.0.0.3
>>> argus[1611]: 08 Jun 12 01:05:07.432936 connect from 10.0.0.3
>>> 
>>> The following line shows up after I control+c
>>> argus[1611]: 08 Jun 12 01:14:23.524250 ArgusCloseModeler(0x100c7b00) ArgusGenerateListRecord failed
>>> 
>>> Currently I'm working with one of bivio team engineer to see how this issue can be fixed. Just keep this between us. Thank you.
>>> 
>>> 
>>> On Fri, Jun 8, 2012 at 2:40 AM, Carter Bullard <carter at qosient.com> wrote:
>>> Lets see if I find something in the review tonight.
>>> I don't have that much familiarity with Bivio debugging myself, I'll appeal to
>>> some of the Bivio engineers to help out, if  we need to go that far.
>>> But definitely need to get this fixed.
>>> 
>>> Carter
>>> 
>>> 
>>> On Jun 7, 2012, at 1:50 PM, CS Lee wrote:
>>> 
>>>> hi Carter,
>>>> 
>>>> I'm currently deploying this system for Myanmar government(they got out of US Sanction status since May).
>>>> 
>>>> If you can't find any site to have similar issue, I can give you the access of the bivio system and also one of the linux server, currently I don't even share the access with bivio team yet, they just support me via email.
>>>> 
>>>> What do you think? Only thing is I don't know how to fork argus to foreground to see the debug message in bivio system, on linux server everything is straightforward.
>>>> 
>>>> 
>>>> -- 
>>>> Best Regards,
>>>> 
>>>> CS Lee<geek00L[at]gmail.com>
>>>> 
>>>> http://geek00l.blogspot.com
>>>> http://defcraft.net
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Best Regards,
>>> 
>>> CS Lee<geek00L[at]gmail.com>
>>> 
>>> http://geek00l.blogspot.com
>>> http://defcraft.net
>>> 
>>> 
>>> 
>>> -- 
>>> Best Regards,
>>> 
>>> CS Lee<geek00L[at]gmail.com>
>>> 
>>> http://geek00l.blogspot.com
>>> http://defcraft.net
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> 
>> CS Lee<geek00L[at]gmail.com>
>> 
>> http://geek00l.blogspot.com
>> http://defcraft.net
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> 
>> CS Lee<geek00L[at]gmail.com>
>> 
>> http://geek00l.blogspot.com
>> http://defcraft.net
> 
> 
> 
> 
> -- 
> Best Regards,
> 
> CS Lee<geek00L[at]gmail.com>
> 
> http://geek00l.blogspot.com
> http://defcraft.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120608/ca2bc5a2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120608/ca2bc5a2/attachment.bin>