Error Starting Argus Daemon
Welland, Neal
N.Welland at warwick.ac.uk
Tue Feb 5 10:30:34 EST 2013
Hi,
I tried recompiling the v3.0.2 daemon without .debug or .devel. Compilation succeeded, but execution of the daemon resulted in the same scenario as before - runs for approx. 1 second, logging traffic on the interface, syslog does not indicate the daemon has stopped.
In an attempt to move things along, I downloaded v3.0.6 (current stable version). I applied the same code changes to get it to compile on Solaris, and also added the scheduling modification to get around permission issues.
Sadly I get the same results as with v3.0.2 described above.
I'm at a loss as to what to do now. Any additional advice would be very welcome.
Regards, Neal.
-----Original Message-----
From: Carter Bullard [mailto:carter at qosient.com]
Sent: 31 January 2013 17:23
To: Welland, Neal
Cc: argus-info at lists.andrew.cmu.edu
Subject: Re: [ARGUS] Error Starting Argus Daemon
Hey Neal,
Yes, if we don't have a cause, we'll have to divine one.
Adding symbols using the " .devel " method will allow us to run under gdb(), and breaking in ArgusShutdown(), or in main() toward the end of the rountine, to try to figure out what the issue is.
Sorry about all this. Argus runs well on Solaris for a lot of sites, not sure why your having so many issues / problems.
Carter
On Jan 31, 2013, at 12:02 PM, "Welland, Neal" <N.Welland at warwick.ac.uk> wrote:
> Hi,
>
> Sorry, no messages printed, other than those from the debug!
>
> Ran with lower debug level:
>
> $ /opt/csw/bin/sudo /usr/local/sbin/argus -D3 -F /etc/argus.conf
> argus[3909.01000000]: 31 Jan 13 16:50:02.803234 ArgusNewModeler()
> returning 80c0f58
> argus[3909.01000000]: 31 Jan 13 16:50:02.810449
> ArgusNewSource(80c0f58) returning 80c3f18
> argus[3909.01000000]: 31 Jan 13 16:50:02.810502 ArgusNewOutput()
> returning retn 84cc230
> argus[3909.01000000]: 31 Jan 13 16:50:02.818848 setArgusID(80c3f34,
> 0x89cd5bc0) done
> argus[3909.01000000]: 31 Jan 13 16:50:02.818897 setArgusPortNum(561)
> returning
> argus[3909.01000000]: 31 Jan 13 16:50:02.818919
> clearArgusDevice(80c3f18) returning
> argus[3909.01000000]: 31 Jan 13 16:50:02.824410 setArgusDevice(nxge0)
> returning
>
> Return code:
>
> $ echo $?
> 0
>
> Same messages in /var/adm/messages.
>
> If I run argus in daemon mode (using the command line toggle):
>
> $ /opt/csw/bin/sudo /usr/local/sbin/argus -F /etc/argus.conf -d
>
> I get the following in /var/adm/messages:
>
> Jan 31 16:54:48 cachi argus[4100]: [ID 586304 daemon.warning] 31 Jan
> 13 16:54:48.106655 started Jan 31 16:54:48 cachi argus[4100]: [ID
> 289320 daemon.warning] 31 Jan 13 16:54:48.114981
> ArgusGetInterfaceStatus: interface nxge0 is up Jan 31 16:54:49 cachi
> argus[4100]: [ID 458095 daemon.warning] 31 Jan 13 16:54:49.116451
> stopped
>
> So, no message on exit.
>
> Is it worth recompiling without debug & symbols etc?
>
> Regards, Neal.
>
> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: 31 January 2013 16:20
> To: Welland, Neal
> Cc: argus-info at lists.andrew.cmu.edu
> Subject: Re: [ARGUS] Error Starting Argus Daemon
>
> * PGP - S/MIME Signed by an unverified key: 31/01/2013 at 16:20:16
>
> Hey Neal,
> No need to run with the " -D 8 " at this point, as we're at least functional.
> 2 or 3 will be more than needed to understand faults and exit status, but most faults generate their own messages, so -D may not be needed.
>
> Does argus print out any message at the point where it exists? It does normally complain before it exits.
>
> Carter
>
> On Jan 31, 2013, at 10:52 AM, "Welland, Neal" <N.Welland at warwick.ac.uk> wrote:
>
>> Hey,
>>
>> While initially preparing my response to you, I noticed that although the interface (nxge0) was plumbed in, it wasn't "UP". After activating the interface, I executed argus once again.
>>
>> $ /opt/csw/bin/sudo /usr/local/sbin/argus -D8 -F /etc/argus.conf
>> argus[1001.01000000]: 31 Jan 13 15:41:14.979050 ArgusCalloc (1, 1772)
>> returning 80c0f58
>> argus[1001.01000000]: 31 Jan 13 15:41:14.979433 ArgusNewModeler()
>> returning 80c0f58
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986316 ArgusCalloc (1,
>> 4227852) returning 80c3f18
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986358
>> ArgusNewSource(80c0f58) returning 80c3f18
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986381 ArgusCalloc (1, 196)
>> returning 84cc230
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986396 ArgusCalloc (1, 88)
>> returning 84cc300
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986407 ArgusNewQueue ()
>> returning 84cc300
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986424 ArgusCalloc (1, 80)
>> returning 84cc360
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986435 ArgusNewList ()
>> returning 84cc360
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986444 ArgusCalloc (1, 80)
>> returning 84cc3b8
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986455 ArgusNewList ()
>> returning 84cc3b8
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986464 ArgusNewOutput()
>> returning retn 84cc230
>> argus[1001.01000000]: 31 Jan 13 15:41:14.986495
>> setArgusMarReportInterval(60) returning
>> argus[1001.01000000]: 31 Jan 13 15:41:14.994743 setArgusID(80c3f34,
>> 0x89cd5bc0) done
>> argus[1001.01000000]: 31 Jan 13 15:41:14.994794 setArgusPortNum(561)
>> returning
>> argus[1001.01000000]: 31 Jan 13 15:41:14.994814
>> clearArgusDevice(80c3f18) returning
>> argus[1001.01000000]: 31 Jan 13 15:41:14.994826 ArgusCalloc (1, 80)
>> returning 84d1220
>> argus[1001.01000000]: 31 Jan 13 15:41:14.994838 ArgusNewList ()
>> returning 84d1220
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995415 ArgusCalloc (1, 40)
>> returning 84d22c0
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995438 ArgusPushFrontList
>> (84d1220, 84d22c0, 1) returning 84d17c9
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995454 setArgusDevice(nxge0)
>> returning
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995590 ArgusDeleteList (0,
>> 2) returning
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995608 ArgusCalloc (1, 80)
>> returning 84d1278
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995619 ArgusNewList ()
>> returning 84d1278
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995630 ArgusCalloc (1, 12)
>> returning 84d1c30
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995641 ArgusPushFrontList
>> (84d1278, 84d1c30, 1) returning 804685f
>> argus[1001.01000000]: 31 Jan 13 15:41:14.995664
>> setArgusMarReportInterval(60) returning
>>
>> The argus process runs for a second or so and then returns. While it is running, it is writing data records to the output file, which can subsequently be read:
>>
>> $ ra -nr cachiArgus/tap.argus -F ~/VboxShare/ra.conf .
>> .
>> 2013-01-31 15:41:15.225568 e tcp 217.155.203.235.48225 ?> 137.205.238.216.49634 1 99 CON s[32]=..\.....f..L.P..*=....../..^...w
>> 2013-01-31 15:41:15.225574 e tcp 217.155.203.235.48225 ?> 172.29.1.49.57710 1 99 CON s[32]=..\.....f..L.P..*=....../..^...w
>> 2013-01-31 15:41:15.225583 e udp 108.61.230.152.14280 -> 137.205.6.76.3074 1 162 REQ s[32]=..7../.MNT....d.cH..`N..4L.q2T..
>> 2013-01-31 15:41:15.225586 e udp 137.205.238.52.13346 -> 94.64.132.223.61657 1 72 REQ s[30]=A............8..O.............
>> 2013-01-31 15:41:15.225595 e udp 137.205.70.232.3074 -> 108.61.230.111.14080 1 122 REQ s[32]=...$....G=T..9 at ......C...3.H.C..
>> 2013-01-31 15:41:15.225596 e udp 137.205.225.25.17773 -> 42.60.61.148.23557 1 79 REQ s[32]=.......C..[.............o.......
>> 2013-01-31 15:41:15.225597 e udp 172.31.16.49.64945 -> 171.36.219.37.26879 1 67 REQ s[25]=..C....qB.....p.....M....
>> 2013-01-31 15:41:15.225625 e tcp 137.205.53.74.50506 -> 173.194.34.155.80 1 66 REQ
>> 2013-01-31 15:41:15.225767 e s tcp 137.205.6.47.1440 -> 137.205.184.19.443 2 132 REQ
>> 2013-01-31 15:41:15.226870 e tcp 81.177.35.226.80 ?> 137.205.238.151.37177 1 1514 CON s[32]=..[.9.9..E...<...c.9....l..j]6^v
>> 2013-01-31 15:41:15.226897 e tcp 137.205.133.42.55758 ?> 2.22.228.8.80 1 858 CON s[32]=GET /shared/app/pulsar/assets/?c
>> 2013-01-31 15:41:15.226913 e tcp 81.177.35.226.80 ?> 172.31.196.15.4533 1 1514 CON s[32]=..[.9.9..E...<...c.9....l..j]6^v
>> 2013-01-31 15:41:15.226961 e tcp 65.54.165.55.443 ?> 137.205.73.138.61744 1 66 CON
>> 2013-01-31 15:41:15.226965 e tcp 137.205.185.216.54677 ?> 157.56.126.55.443 1 60 CON s[1]=.
>>
>> /var/adm/messages shows the process starting and that the interface is up:
>>
>> Jan 31 15:41:15 cachi argus[1001]: [ID 180722 daemon.warning] 31 Jan
>> 13 15:41:15.000208 started Jan 31 15:41:15 cachi argus[1001]: [ID
>> 326406 daemon.warning] 31 Jan 13 15:41:15.008750
>> ArgusGetInterfaceStatus: interface nxge0 is up
>>
>> Here is the latest argus.conf directives with ARGUS_DAEMON removed as suggested:
>>
>> ARGUS_ACCESS_PORT=561
>> ARGUS_CAPTURE_DATA_LEN=32
>> ARGUS_DEBUG_LEVEL=0
>> ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
>> ARGUS_FLOW_STATUS_INTERVAL=5
>> ARGUS_FLOW_TYPE="Bidirectional"
>> ARGUS_GENERATE_APPBYTE_METRIC=yes
>> ARGUS_GENERATE_BIDIRECTIONAL_TIMESTAMPS=yes
>> ARGUS_GENERATE_JITTER_DATA=no
>> ARGUS_GENERATE_MAC_DATA=no
>> ARGUS_GENERATE_PACKET_SIZE=yes
>> ARGUS_GENERATE_RESPONSE_TIME_DATA=no
>> ARGUS_GO_PROMISCUOUS=yes
>> ARGUS_INTERFACE=nxge0
>> ARGUS_MAR_STATUS_INTERVAL=60
>> ARGUS_MONITOR_ID=`hostname`
>> ARGUS_OUTPUT_FILE=/datapool/cachi_int-rz1/tap.argus
>> ARGUS_PID_PATH="/var/run"
>> ARGUS_SET_PID=yes
>> ARGUS_SETGROUP_ID=other
>> ARGUS_SETUSER_ID=argus
>>
>> Progress, but I'm not sure why the process returns.
>>
>> Many thanks, Neal.
>>
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: 31 January 2013 13:50
>> To: Welland, Neal
>> Cc: argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] Error Starting Argus Daemon
>>
>>> Old - S/MIME Signed by an unverified key: 31/01/2013 at 13:50:15
>>
>> Hey Neal,
>> Hmmmm, you've run down the path, but we may need to back up a bit, and
>> test a few things. Having argus, itself, write to an output file is deprecated,
>> as we have a set of clients that do a better job than argus does, and we want argus to use its cycles to process packets, not write to the disk.
>>
>> But, that doesn't mean you can't do it, so lets try a few things.
>> First, with all the changes, lets get it running, verify that it does the basic things, and, then move on, to a reasonable installation.
>>
>> Lets not run as a daemon, in the first pass, either comment out ARGUS_DAEMON in the argus.conf file, or add a " -d " to toggle that option. Be sure to use the " -d "
>> after the "-F argus.conf" option on the command line. Order matters.
>>
>> Don't use the Debug option if you are running argus as a daemon. Thats a lot of debug messages that will be going into the system syslog, and that probably
>> isn't going to perform very well. Use the -D when argus is running from a shell,
>> in the foreground.
>>
>> What interface is argus opening? When argus starts, it should declare which
>> interface is " up " which will tell us which interface it's opening. Is there traffic on
>> that interface? When run as a daemon, argus should send that message to the syslog.
>>
>> Looks to me that argus maybe writing management records, but not data records, so lets see if we can get argus to read an active packet stream.
>>
>> The error messages that you are seeing are very important, because they usually mean that some aspect of argus has failed. Argus is a multi-threaded program, and while its suppose to exit if any of its parts fail, the bug maybe that a part of argus runs, even though most of it is dead. So, we need to fix the socket option error, and the write() error.
>>
>> Can you share your argus.conf file?
>>
>> Carter
>>
>> On Jan 30, 2013, at 10:02 AM, "Welland, Neal" <N.Welland at warwick.ac.uk> wrote:
>>
>>> Hey Cater,
>>>
>>> I can see the daemon is running from a quick ps:
>>>
>>> root 28873 23841 0 14:30:39 pts/1 0:00 /opt/csw/bin/sudo /usr/local/sbin/argus -D8 -F /etc/argus.conf
>>>
>>> The file its configured to write to seems to be having its time stamp updated:
>>>
>>> $ ls -lrt /datapool/cachi_int-rz1
>>> total 16
>>> -rw-r--r-- 1 argus argus 7296 Jan 30 14:49 tap.argus
>>>
>>> But it's size suggests it's not seeing any interface traffic.
>>>
>>> The ra() client isn't installed on this host, and if I try and connect remotely, I get:
>>>
>>> $ ra -nS cachi
>>> ra[16833]: 01-30-13 14:53:56 ArgusReadConnection: 137.205.91.192 connection closed.
>>>
>>> Which produces the following logs on the server:
>>>
>>> Jan 30 14:53:05 cachi argus[28886]: [ID 444193 daemon.notice]
>>> connect from cookie-hostage.warwick.ac.uk with IP options (ignored):
>>> 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> 00 00 00 00 00 00 Jan 30 14:53:05 cachi argus[28886]: [ID 899512
>>> daemon.error] setsockopt IP_OPTIONS NULL: Option not supported by
>>> protocol Jan 30 14:53:05 cachi argus[28886]: [ID 811558
>>> daemon.error]
>>> 30 Jan 13 14:53:05.649044 ArgusInitOutput: write(): No such file or
>>> directory
>>>
>>> I've mounted the partition containing tap.argus on a machine with ra() installed. Running $ ra -nr tap.argus produced NO output.
>>>
>>> Running $ racount -nr tap.argus
>>> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
>>> sum 60 0 0 0 0 0 0
>>>
>>> Which suggests time stamps are being written to the file, but nothing else.
>>>
>>> Regards, Neal.
>>>
>>> -----Original Message-----
>>> From: Carter Bullard [mailto:carter at qosient.com]
>>> Sent: 30 January 2013 14:01
>>> To: Welland, Neal
>>> Cc: argus-info at lists.andrew.cmu.edu
>>> Subject: Re: [ARGUS] Error Starting Argus Daemon
>>>
>>> Hey Neal,
>>> Try connecting to argus with ra() to see if its running.
>>> Argus doesn't log anything unless you subscribe to the data stream.
>>>
>>> Carter
>>>
>>> On Jan 30, 2013, at 8:56 AM, "Welland, Neal" <N.Welland at warwick.ac.uk> wrote:
>>>
>>>> Hey Carter,
>>>>
>>>> I modified argus.c as you suggested:
>>>>
>>>> # diff argus.c argus.c.orig
>>>> 583d582
>>>> < #ifdef ARGUS_NOT_DEFINED
>>>> 614d612
>>>> < #endif
>>>>
>>>> I recompiled and executed: $ /opt/csw/bin/sudo
>>>> /usr/local/sbin/argus
>>>> -D8 -F /etc/argus.conf
>>>>
>>>> Which gave the following output:
>>>>
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.132830 ArgusCalloc (1,
>>>> 1772) returning 80c0f58
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.133228 ArgusNewModeler()
>>>> returning 80c0f58
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140150 ArgusCalloc (1,
>>>> 4227852) returning 80c3f18
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140193
>>>> ArgusNewSource(80c0f58) returning 80c3f18
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140216 ArgusCalloc (1,
>>>> 196) returning 84cc230
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140232 ArgusCalloc (1,
>>>> 88) returning 84cc300
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140244 ArgusNewQueue ()
>>>> returning 84cc300
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140266 ArgusCalloc (1,
>>>> 80) returning 84cc360
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140277 ArgusNewList ()
>>>> returning 84cc360
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140287 ArgusCalloc (1,
>>>> 80) returning 84cc3b8
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140300 ArgusNewList ()
>>>> returning 84cc3b8
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140310 ArgusNewOutput()
>>>> returning retn 84cc230
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.140342
>>>> setArgusMarReportInterval(60) returning
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152243
>>>> setArgusID(80c3f34,
>>>> 0x89cd5bc0) done
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152296
>>>> setArgusPortNum(561) returning
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152318
>>>> clearArgusDevice(80c3f18) returning
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152330 ArgusCalloc (1,
>>>> 80) returning 84d1220
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152341 ArgusNewList ()
>>>> returning 84d1220
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152850 ArgusCalloc (1,
>>>> 40) returning 84d22c0
>>>> argus[27211.01000000]: 30 Jan 13 13:50:53.152875 ArgusPushFrontList
>>>> (
More information about the argus
mailing list