argus/radium problem

Fri Jun 8 12:58:58 EDT 2012

Hey CS Lee,
Just going through the messages you sent, so we have some dialog on what is going on.

OK, so this is the most important to figuring out the argus fatal issues.  After deleting the socket that the client
was using, somehow the modeler looks to have been re-initialized, or deleted.  This is not good.
The model->ArgusOutputList is the communication channel from the ArgusModeler to the ArgusOutput
thread, so if this goes away, broken argus.

The ArgusModeler->ArgusOutputList goes away only when the something calls ArgusCloseOutput().
Any chance you can verify, running argus with a "-D 2" option, that in this situation, ArgusCloseOutput()
is called?

Carter 

On Jun 7, 2012, at 11:58 PM, CS Lee wrote:

> hi Carter,
> 
> I run ra in follwoing way and after ra has retrieved 100 records, it exits, however I can't reconnect to argus again -
> 
> ra -N 100 -Z b -S 10.0.0.1:561 -s saddr sport daddr flgs proto state
> 
> This is the output I get from argus 
> 
> argus[27489.48c93490]: 08 Jun 12 10:23:07.144707 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
> argus[27489.48c93490]: 08 Jun 12 10:23:07.144726 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
> argus[27489.48c93490]: 08 Jun 12 10:23:07.144786 ArgusWriteSocket: write (7, 0x58300058, 280, ...) -1
> argus[27489.48c93490]: 08 Jun 12 10:23:07.144804 ArgusDeleteSocket: list not empty
> argus[27489.48c93490]: 08 Jun 12 10:23:07.144909 ArgusDeleteSocket (0x58300008) returning
> argus[27489.498dd490]: 08 Jun 12 10:26:18.708883 ArgusSendFlowRecord (0x1009b008, 0x57595818, 64) no model->ArgusOutputList
> argus[27489.498dd490]: 08 Jun 12 10:26:19.110344 ArgusSendFlowRecord (0x1009b008, 0x527033d8, 64) no model->ArgusOutputList
> 
> If I reconnect to argus again, I get connection refused message
> 
> ra[20656]: 10:26:56.109970 connect to 10.0.0.1:561 failed 'Connection timed out'
> 
> Thank you.
> 
> 
> 
> 
> On Fri, Jun 8, 2012 at 10:12 AM, CS Lee <geek00l at gmail.com> wrote:
> hi Carter,
> 
> I don't know if bivio team has compiled it with threaded support, need to ask them(joel in bivio) if you know him. 
> 
> Here's the argus output message when radium connects to it -
> 
> ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x32a5da00 msgnum 0x66573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
> argus[27125.498dd490]: 08 Jun 12 08:26:57.178068 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x329ee15a msgnum 0x31573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
> argus[27125.498dd490]: 08 Jun 12 08:26:57.196113 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x321c61e1 msgnum 0xac573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
> argus[27125.498dd490]: 08 Jun 12 08:26:57.220138 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x3227377b msgnum 0xba573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
> argus[27125.498dd490]: 08 Jun 12 08:26:57.293605 ArgusUpdateUDTState(0x49dd65a8, 32) UDT_DATA_PACKET seq 0x32ac4271 msgnum 0x59573422 tstmp 0x216c794 sockid 0x12ce55c8 loss 0
> argus[27125.48c93490]: 08 Jun 12 08:26:58.039943 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
> argus[27125.48c93490]: 08 Jun 12 08:26:58.040039 ArgusFreeListRecord (0x100f679c) returning
> argus[27125.48c93490]: 08 Jun 12 08:26:59.243664 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
> argus[27125.48c93490]: 08 Jun 12 08:26:59.243759 ArgusFreeListRecord (0x100f6b44) returning
> argus[27125.48c93490]: 08 Jun 12 08:26:59.413949 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
> argus[27125.48c93490]: 08 Jun 12 08:26:59.414030 ArgusFreeListRecord (0x100f6eec) returning
> argus[27125.48c93490]: 08 Jun 12 08:26:59.767378 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
> argus[27125.48c93490]: 08 Jun 12 08:26:59.767474 ArgusFreeListRecord (0x100f7294) returning
> argus[27125.48c93490]: 08 Jun 12 08:27:00.112948 ArgusLoadList (0x1009b9c0, 0x1009b948) load 1 objects
> argus[27125.48c93490]: 08 Jun 12 08:27:00.113028 ArgusFreeListRecord (0x100f763c) returning
> 
> Then after a while, this appears -
> 
> ArgusGenerateInitialMar() returning
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083154 ArgusCheckClientStatus: wrote 128 bytes to client
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083175 ArgusCheckClientStatus() returning
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083192 ArgusLoadList (0x1009b9c0, 0x1009b948) load 19823 objects
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083209 ArgusOutputProcess() 1 client(s) for record 0x4ce4e2a4
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083226 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083244 ArgusFreeListRecord (0x4ce4e2a4) returning
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083260 ArgusOutputProcess() 1 client(s) for record 0x4ce4e64c
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083276 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083293 ArgusFreeListRecord (0x4ce4e64c) returning
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083310 ArgusOutputProcess() 1 client(s) for record 0x4ce4e9f4
> argus[27125.48c93490]: 08 Jun 12 08:27:09.083326 ArgusOutputProcess() 1 client(s) not ready fd 4 sock 0x4f500008 start 0
> 
> 
> On Fri, Jun 8, 2012 at 10:02 AM, Carter Bullard <carter at qosient.com> wrote:
> Well, if it's getting write errors we should be shutting down the connection and deallocating any queued records.  This is useful.  Let me see what I can do with this.
> 
> Carter
> 
> Sent from my iPad
> 
> On Jun 7, 2012, at 9:19 PM, CS Lee <geek00l at gmail.com> wrote:
> 
>> hi Carter,
>> 
>> Bivio team has sent me the argus rpm with debug enabled, here's what I get from argus -D 5 when argus client sees the idle stream message -
>> 
>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594670 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594696 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> argus[26916.48c93490]: 08 Jun 12 07:20:28.594723 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> argus[26916.498dd490]: 08 Jun 12 07:20:28.600001 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.48c93490]: 08 Jun 12 07:20:28.603302 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> argus[26916.48c93490]: 08 Jun 12 07:20:28.603332 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> 
>> argus[26916.48c93490]: 08 Jun 12 07:20:33.554648 ArgusWriteSocket: write (7, 0x4d200058, 48, ...) -1
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.554671 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563056 ArgusMallocListRecord (904) returned NULL
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563076 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563092 ArgusMallocListRecord (904) returned NULL
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563108 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563124 ArgusMallocListRecord (904) returned NULL
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563140 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563157 ArgusMallocListRecord (904) returned NULL
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563173 ArgusMallocList memory pool exhausted (1000000 : 1000000)
>> argus[26916.498dd490]: 08 Jun 12 07:20:33.563189 ArgusMallocListRecord (904) returned NULL
>> 
>> Argus keeps running however no client can connect to it and retrieve anything.
>> 
>> On Fri, Jun 8, 2012 at 2:54 AM, CS Lee <geek00l at gmail.com> wrote:
>> hi Carter,
>> 
>> I able to get argus running in foreground now, however since the rpm they provide doesn't compile with debug enable, I can only see limited output, here's where it breaks.
>> 
>> Argus client on linux box
>> ra[32718]: 01:07:11.134418 ArgusReadStream 10.0.0.1: idle stream: closing
>> 
>> Argus on bivio
>> argus -D 3 -i bond:s0.e0,s0.e1 -mAJZRU 64 -B 10.0.0.1 -P 561
>> argus[1611]: 08 Jun 12 01:02:46.781824 started
>> argus[1611]: 08 Jun 12 01:02:46.800912 ArgusGetInterfaceStatus: interface s0.e1 is up
>> argus[1611]: 08 Jun 12 01:02:46.801064 ArgusGetInterfaceStatus: interface s0.e0 is up
>> argus[1611]: 08 Jun 12 01:04:15.489853 connect from 10.0.0.3
>> argus[1611]: 08 Jun 12 01:05:07.432936 connect from 10.0.0.3
>> 
>> The following line shows up after I control+c
>> argus[1611]: 08 Jun 12 01:14:23.524250 ArgusCloseModeler(0x100c7b00) ArgusGenerateListRecord failed
>> 
>> Currently I'm working with one of bivio team engineer to see how this issue can be fixed. Just keep this between us. Thank you.
>> 
>> 
>> On Fri, Jun 8, 2012 at 2:40 AM, Carter Bullard <carter at qosient.com> wrote:
>> Lets see if I find something in the review tonight.
>> I don't have that much familiarity with Bivio debugging myself, I'll appeal to
>> some of the Bivio engineers to help out, if  we need to go that far.
>> But definitely need to get this fixed.
>> 
>> Carter
>> 
>> 
>> On Jun 7, 2012, at 1:50 PM, CS Lee wrote:
>> 
>>> hi Carter,
>>> 
>>> I'm currently deploying this system for Myanmar government(they got out of US Sanction status since May).
>>> 
>>> If you can't find any site to have similar issue, I can give you the access of the bivio system and also one of the linux server, currently I don't even share the access with bivio team yet, they just support me via email.
>>> 
>>> What do you think? Only thing is I don't know how to fork argus to foreground to see the debug message in bivio system, on linux server everything is straightforward.
>>> 
>>> 
>>> -- 
>>> Best Regards,
>>> 
>>> CS Lee<geek00L[at]gmail.com>
>>> 
>>> http://geek00l.blogspot.com
>>> http://defcraft.net
>> 
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> 
>> CS Lee<geek00L[at]gmail.com>
>> 
>> http://geek00l.blogspot.com
>> http://defcraft.net
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> 
>> CS Lee<geek00L[at]gmail.com>
>> 
>> http://geek00l.blogspot.com
>> http://defcraft.net
> 
> 
> 
> -- 
> Best Regards,
> 
> CS Lee<geek00L[at]gmail.com>
> 
> http://geek00l.blogspot.com
> http://defcraft.net
> 
> 
> 
> -- 
> Best Regards,
> 
> CS Lee<geek00L[at]gmail.com>
> 
> http://geek00l.blogspot.com
> http://defcraft.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120608/6f5fc77a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120608/6f5fc77a/attachment.bin>