FWD: RE: Argus, and moving 'live files'

Chris Newton newton at unb.ca
Sat Mar 10 19:11:53 EST 2001


Ok, I didn't have argus compiled with debug turned on... but, I think I do 
now.  I couldn't really find any info on how to do it.. but, it looked like 
from the configure script, it tested for a .debug.  So, I 'touch'ed one of 
those, and recompiled.  The binaries were bigger than my last compile, so I 
assume that worked.

  That version of argus is running now...  it might take a couple/few more 
days for this to occur again.  I'll report as soon as I see it happen.

Thanks for the help

Chris

>===== Original Message From <carter at qosient.com> =====
>If you compiled with debug support (~/.debug file present)
>then you should send some USR1 signals to the process
>with the median PID.  Assuming that argus is pid 200, 201,
>and 202, send 4 USR1 messages to pid 201.
>
>   # kill -USR1 201
>   # kill -USR1 201
>   # kill -USR1 201
>   # kill -USR1 201
>
>And lets see what its saying for itself.  If you need
>to, send as many as 8.  If you don't get anything at
>all, and debug support was compiled in, then there is a
>really good problem to solve.
>
>If you have any gdb() experience, attach to process 201
>and step through to see where it thinks it is.
>
>I'm going out in just a few minutes, so I'll have to
>pay attention to this on Sunday.
>
>Carter
>
>Carter Bullard
>QoSient, LLC
>300 E. 56th Street, Suite 18K
>New York, New York  10022
>
>carter at qosient.com
>Phone +1 212 588-9133
>Fax   +1 212 588-9134
>http://qosient.com
>
>> -----Original Message-----
>> From: Chris Newton [mailto:newton at unb.ca]
>> Sent: Saturday, March 10, 2001 6:39 PM
>> To: Carter Bullard; argus; Peter Van Epp
>> Subject: RE: FWD: RE: Argus, and moving 'live files'
>>
>>
>> Nope:
>>
>> [newton at epic flowlogs]$ ls -l argus-2001-03-10-18:14:50
>> -rw-r--r--    1 root     root       323624 Mar 10 18:14
>> argus-2001-03-10-18:14:50
>>
>>   And, in fact, it seems a little small, compared to the
>> others from the same
>> time.  So, maybe it isn't the moving of the files thats
>> causing the problem...
>>  Here is an ls of the other files from that hour... right up
>> to when it
>> stopped outputing.  So, either that was a slow minute.. or,
>> argus stopped
>> writing to that file part way through the minute.
>>
>>
>> [newton at epic flowlogs]$ ls -l argus-2001-03-10-18:*
>> -rw-r--r--    1 root     root       569956 Mar 10 18:00
>> argus-2001-03-10-18:00:49
>> -rw-r--r--    1 root     root       580148 Mar 10 18:01
>> argus-2001-03-10-18:01:49
>> -rw-r--r--    1 root     root       593456 Mar 10 18:02
>> argus-2001-03-10-18:02:49
>> -rw-r--r--    1 root     root       584564 Mar 10 18:03
>> argus-2001-03-10-18:03:49
>> -rw-r--r--    1 root     root       507604 Mar 10 18:04
>> argus-2001-03-10-18:04:49
>> -rw-r--r--    1 root     root       451776 Mar 10 18:05
>> argus-2001-03-10-18:05:49
>> -rw-r--r--    1 root     root       500492 Mar 10 18:06
>> argus-2001-03-10-18:06:49
>> -rw-r--r--    1 root     root       499104 Mar 10 18:07
>> argus-2001-03-10-18:07:50
>> -rw-r--r--    1 root     root       467036 Mar 10 18:08
>> argus-2001-03-10-18:08:50
>> -rw-r--r--    1 root     root       431164 Mar 10 18:09
>> argus-2001-03-10-18:09:50
>> -rw-r--r--    1 root     root       465376 Mar 10 18:10
>> argus-2001-03-10-18:10:50
>> -rw-r--r--    1 root     root       415984 Mar 10 18:11
>> argus-2001-03-10-18:11:50
>> -rw-r--r--    1 root     root       500576 Mar 10 18:12
>> argus-2001-03-10-18:12:50
>> -rw-r--r--    1 root     root       521964 Mar 10 18:13
>> argus-2001-03-10-18:13:50
>> -rw-r--r--    1 root     root       323624 Mar 10 18:14
>> argus-2001-03-10-18:14:50
>> [newton at epic flowlogs]$
>>
>>
>>
>> >===== Original Message From <carter at qosient.com> =====
>> >Is the file /usr/local/nva/flowlogs/argus-2001-03-10-18:14:50
>> >getting bigger?
>> >
>> >Carter
>> >
>> >Carter Bullard
>> >QoSient, LLC
>> >300 E. 56th Street, Suite 18K
>> >New York, New York  10022
>> >
>> >carter at qosient.com
>> >Phone +1 212 588-9133
>> >Fax   +1 212 588-9134
>> >http://qosient.com
>> >
>> >> -----Original Message-----
>> >> From: Chris Newton [mailto:newton at unb.ca]
>> >> Sent: Saturday, March 10, 2001 6:22 PM
>> >> To: Carter Bullard; argus; Peter Van Epp
>> >> Subject: RE: FWD: RE: Argus, and moving 'live files'
>> >>
>> >>
>> >> Ok, happened again, tonight at 8:12pm.
>> >>
>> >>   Here is the lsof output, below.... which is interesting,
>> >> because it shows
>> >> Argus having open the file
>> >> /usr/local/nva/flowlogs/argus-2001-03-10-18:14:50,
>> >> What is odd about that is that argus was started with this
>> >> command line:
>> >>
>> >> /usr/local/nva/bin/argus -d -F /usr/local/nva/conf/argus.conf -w
>> >> /usr/local/nva/flowlogs/argus.out
>> >>
>> >>   And, I have a program called argproc that runs continually.
>> >>  What it does is
>> >> moves the argus.out file to argus-{data/timestamp}, waits for
>> >> 1 minute (minus
>> >> the amount of time it took to move that file), and does it
>> >> again.  Right now
>> >> it is complaining that the argus.out file doesnt exist when
>> >> it tries to do the
>> >> move.
>> >>
>> >>   And, yup, all three argii are running:
>> >> [root at epic conf]# ps axfw |grep argus
>> >>   519 ?        R     83:22 /usr/local/nva/bin/argus -d -F
>> >> /usr/local/nva/conf/argus.conf -w /usr/local/nva/flowlogs/argus.out
>> >>   522 ?        S      1:59  \_ /usr/local/nva/bin/argus -d -F
>> >> /usr/local/nva/conf/argus.conf -w /usr/local/nva/flowlogs/argus.out
>> >>   523 ?        S      3:16      \_ /usr/local/nva/bin/argus -d -F
>> >> /usr/local/nva/conf/argus.conf -w /usr/local/nva/flowlogs/argus.ou
>> >>
>> >>
>> >>   Some output from top, showing at least one of the argii
>> >> consuming CPU, in
>> >> this case, about 4.1% CPU.
>> >>
>> >>
>> >> 519 root      14   0 12808  12M   628 S       0  4.0 10.1  83:29
>> >> /usr/local/nva/bin/argus -d -F /usr/local/nva/conf/argus.conf -w
>> >>
>> >> Operating system is redhat 6.2, upgraded to 2.4 kernel:
>> >>
>> >> [root at epic conf]# uname -a
>> >> Linux epic.csd.unb.ca 2.4.0-test12 #1 Sat Dec 16 23:51:30 AST
>> >> 2000 i686
>> >> unknown
>> >>
>> >> [root at epic conf]# /usr/sbin/lsof |grep argus
>> >> argus       519   root  cwd    DIR        3,1     4096          2 /
>> >> argus       519   root  rtd    DIR        3,1     4096          2 /
>> >> argus       519   root  txt    REG        3,1   551451     754834
>> >> /usr/local/nva/bin/argus
>> >> argus       519   root  mem    REG        3,1   340663     311343
>> >> /lib/ld-2.1.3.so
>> >> argus       519   root  mem    REG        3,1   527442     311361
>> >> /lib/libm-2.1.3.so
>> >> argus       519   root  mem    REG        3,1  4101324     311350
>> >> /lib/libc-2.1.3.so
>> >> argus       519   root  mem    REG        3,1   246652     311381
>> >> /lib/libnss_files-2.1.3.so
>> >> argus       519   root    0r   REG        3,1     8105     754878
>> >> /usr/local/nva/conf/argus.conf
>> >> argus       519   root    3u  sock        0,0
>> >> 1331 can't
>> >> identify protocol
>> >> argus       519   root    4r  FIFO        0,0
>>   1337 pipe
>> >> argus       519   root    5w  FIFO        0,0
>>   1337 pipe
>> >> argus       519   root    6w   CHR        1,3
>> >> 180352 /dev/null
>> >> argus       522   root  cwd    DIR        3,1     4096          2 /
>> >> argus       522   root  rtd    DIR        3,1     4096          2 /
>> >> argus       522   root  txt    REG        3,1   551451     754834
>> >> /usr/local/nva/bin/argus
>> >> argus       522   root  mem    REG        3,1   340663     311343
>> >> /lib/ld-2.1.3.so
>> >> argus       522   root  mem    REG        3,1   527442     311361
>> >> /lib/libm-2.1.3.so
>> >> argus       522   root  mem    REG        3,1  4101324     311350
>> >> /lib/libc-2.1.3.so
>> >> argus       522   root  mem    REG        3,1   246652     311381
>> >> /lib/libnss_files-2.1.3.so
>> >> argus       522   root    0r   REG        3,1     8105     754878
>> >> /usr/local/nva/conf/argus.conf
>> >> argus       522   root    1u   CHR        5,1
>> >> 180385 /dev/console
>> >> argus       522   root    2u   CHR        5,1
>> >> 180385 /dev/console
>> >> argus       522   root    3u  sock        0,0
>> >> 1331 can't
>> >> identify protocol
>> >> argus       522   root    4r  FIFO        0,0
>>   1337 pipe
>> >> argus       522   root    5w  FIFO        0,0
>>   1337 pipe
>> >> argus       523   root  cwd    DIR        3,1     4096          2 /
>> >> argus       523   root  rtd    DIR        3,1     4096          2 /
>> >> argus       523   root  txt    REG        3,1   551451     754834
>> >> /usr/local/nva/bin/argus
>> >> argus       523   root  mem    REG        3,1   340663     311343
>> >> /lib/ld-2.1.3.so
>> >> argus       523   root  mem    REG        3,1   527442     311361
>> >> /lib/libm-2.1.3.so
>> >> argus       523   root  mem    REG        3,1  4101324     311350
>> >> /lib/libc-2.1.3.so
>> >> argus       523   root  mem    REG        3,1   246652     311381
>> >> /lib/libnss_files-2.1.3.so
>> >> argus       523   root    0r   REG        3,1     8105     754878
>> >> /usr/local/nva/conf/argus.conf
>> >> argus       523   root    1u   CHR        5,1
>> >> 180385 /dev/console
>> >> argus       523   root    2u   CHR        5,1
>> >> 180385 /dev/console
>> >> argus       523   root    3u  sock        0,0
>> >> 1331 can't
>> >> identify protocol
>> >> argus       523   root    4r  FIFO        0,0
>>   1337 pipe
>> >> argus       523   root    5w  FIFO        0,0
>>   1337 pipe
>> >> argus       523   root    6r  FIFO        0,0
>>   1338 pipe
>> >> argus       523   root    7w  FIFO        0,0
>>   1338 pipe
>> >> argus       523   root    8u   REG        3,1   323624     821089
>> >> /usr/local/nva/flowlogs/argus-2001-03-10-18:14:50
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> >===== Original Message From <carter at qosient.com> =====
>> >> >Hey Chris,
>> >> >What others have seen, is that the second process that argus
>> >> >spawns, is either eating up a lot of CPU or none at all.
>> >> >This is the flow record multiplexor, and so if its not doing
>> >> >what its suppose to do, then nothing is going to come out of
>> >> >the argus.
>> >> >
>> >> >There are several debugging strategies to find out what is
>> >> >going on.  The first is to do a simple ps() to make sure that all
>> >> >the processes are there.  In the case of writing out to a file,
>> >> >you should have at least 3 argus processes running all the time.
>> >> >If you do have 3 processes, you can use gdb to attach to each
>> >> >running process, and then step through them for a few
>> >> >instructions to see what they are doing.
>> >> >
>> >> >Another strategy is to turn debug support on for each process.
>> >> >If you've compiled in debug support, then you can send SIGUSR1
>> >> >signals to any argus process to turn on its debug reporting.
>> >> >So as an example, assuming that the 3 processes are 200, 201
>> >> >and 202:
>> >> >
>> >> >   # kill -USR1 202
>> >> >
>> >> >will turn on debug reporting and set the debug level to one.
>> >> >Sending another SIGUSR1 will increment the debug level.  To
>> >> >turn it off, send a SIGUSR2 to the process.
>> >> >
>> >> >   # kill -USR2 202
>> >> >
>> >> >So you can test them all, by getting their debug level to 3 or
>> >> >4 and see what they think is going on.
>> >> >
>> >> >Carter
>> >> >
>> >> >Carter Bullard
>> >> >QoSient, LLC
>> >> >300 E. 56th Street, Suite 18K
>> >> >New York, New York  10022
>> >> >
>> >> >carter at qosient.com
>> >> >Phone +1 212 588-9133
>> >> >Fax   +1 212 588-9134
>> >> >http://qosient.com
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Chris Newton [mailto:newton at unb.ca]
>> >> >> Sent: Thursday, March 08, 2001 1:33 PM
>> >> >> To: Carter Bullard; argus; Peter Van Epp
>> >> >> Subject: RE: FWD: RE: Argus, and moving 'live files'
>> >> >>
>> >> >>
>> >> >> >===== Original Message From <carter at qosient.com> =====
>> >> >> >Hey Guys,
>> >> >> >   Chris, more than likely your problem doesn't have anything
>> >> >> >to do with the file moving itself.  If Argus breaks, you will
>> >> >> >see that your file moving strategy will suddenly stop, as
>> >> >> >there won't be a file to move any more.  So the file moving
>> >> >> >makes the problem much more apparent.
>> >> >>
>> >> >>   Thats whats happening.  I get errors from my script that
>> >> >> the 'argus-output'
>> >> >> file does not exist, and therefore, can't be moved.  Argus is
>> >> >> still running
>> >> >> happily though.
>> >> >>
>> >> >>   It happens out of the blue (the couple of times it has
>> >> >> happened).  The
>> >> >> moving script runs happily along.. then, boom... errors, 'no
>> >> >> such file'.  I
>> >> >> check, sure enough, Argus isn't recreating the new
>> >> >> 'argus-output' file
>> >> >> anymore.  Kill restart argus, everything returns to normal.
>> >> >>
>> >> >> Chris
>> >> >>
>> >>
>> >> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
>> >>
>> >> Chris Newton, Systems Analyst
>> >> Computing Services, University of New Brunswick
>> >> newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)
>> >>
>> >>
>>
>> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
>>
>> Chris Newton, Systems Analyst
>> Computing Services, University of New Brunswick
>> newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)
>>
>>

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Chris Newton, Systems Analyst
Computing Services, University of New Brunswick
newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)



More information about the argus mailing list