rastream exits after multi day run

[Carter Bullard]

Hey Jesse,
Looks like rastream() isn’t keeping up and radium.1 is dropping the connection.
Your script is a bit heavy, but that runs in a separate process, and shouldn’t
affect rastream, at least that is the design.

In gdb, you can’t run with the daemon flag, so take the ‘-d' out.  You say it crashes ???
If so, can you type ‘where’ so we can see where the fault occurs ??

Run rastream with a ‘-D3’.  That will generate messages about the script and any time shift issues.  ‘-D4’ will add the socket read messages, which maybe of interest, but it may
generate quite a bit of messages.  Try -D3 to see if we see anything interesting.


Carter


> On Oct 29, 2014, at 8:09 PM, Jesse Bowling <jessebowling at gmail.com> wrote:
> 
> Hi,
> 
> I’ve had an issue using rastream recently. What I’m seeing is that rastream will run successfully for a while (usually around 2 days or less) and then will simply exit. The only log messages I get are:
> 
> Oct 15 13:06:24 netf-test radium[33140]: 2014-10-15 13:06:24.908361 started
> Oct 15 13:06:29 netf-test radium[33140]: 2014-10-15 13:06:29.599856 connect from localhost[127.0.0.1]
> Oct 16 10:52:59 netf-test radium[33140]: 2014-10-16 10:52:59.537044 ArgusWriteOutSocket(0xb81269d0) client not processing: disconnecting
> 
> I tried running under gdb, but it quickly finishes/crashes...I’d can run it with -D, but unsure what level might catch what’s happening. Any recommendations? 
> 
> Any ideas of why this might be exiting? Is the bash script too heavy? Would it help to call a wrapper script that simply backgrounds the actual script I want to run?
> 
> Config information below.
> 
> Cheers,
> 
> Jesse
> 
> $ egrep -v '^#|^[ ]*$' /etc/radium.conf 
> RADIUM_DAEMON="yes"
> RADIUM_MONITOR_ID=10.138.3.168
> RADIUM_MAR_STATUS_INTERVAL=60
> RADIUM_ARGUS_SERVER=cisco://10.138.3.168:9995
> RADIUM_ACCESS_PORT=561
> RADIUM_BIND_IP=127.0.0.1
> RADIUM_CLASSIFIER_FILE=/etc/ralabel.conf
> 
> $ egrep -v '^#|^[ ]*$' /etc/ra.conf
> RA_SET_PID="no"
> RA_PID_PATH="/var/run"
> RA_RUN_TIME=0
> RA_GENERATE_BIN_MAR_RECORDS=yes
> RA_PRINT_LABELS=0
> RA_FIELD_SPECIFIER="stime:25 flgs proto saddr sport sco sas dir daddr dport dco das spkts dpkts sbytes dbytes"
> RA_FIELD_DELIMITER=''
> RA_SEPARATE_ADDR_FROM_PORT_WITH_PERIOD="no"
> RA_PRINT_NAMES='none'
> RA_PRINT_RESPONSE_DATA=no
> RA_PRINT_UNIX_TIME=no
> RA_TIME_FORMAT="%F %T.%f"
> RA_USEC_PRECISION=6
> RA_USERDATA_ENCODE=Ascii
> RA_SORT_ALGORITHMS="dpkts spkts saddr dport daddr sport"
> RA_UPDATE_INTERVAL=2
> RA_DELEGATED_IP="/usr/local/argus/delegated-ipv4-latest"
> RA_RELIABLE_CONNECT=yes
> RA_COLOR_SUPPORT="yes"
> RA_COLOR_CONFIG=/etc/racolor.conf
> RA_LOCAL=/usr/local/argus/local.addrs
> 
> Starting like this:
> 
> /usr/local/sbin/radium -f /etc/radium.conf
> /usr/local/bin/rastream -F /etc/ra.conf -M time 5m -B 10s -S 127.0.0.1 -w /srv/scratch/%Y-%m-%d_%H-%M-%S.argus -f /argus/process_argus_file.bash -d
> 
> Contents of process_argus_file.bash:
> 
> #!/bin/bash
> 
> #
> #  Argus Client Software.  Tools to read, analyze and manage Argus data.
> #  Copyright (C) 2000-2014 QoSient, LLC.
> #  All Rights Reserved
> #
> # Script called by rastream, to process files.
> #
> # Since this is being called from rastream(), it will have only a single
> # parameter, filename,
> #
> # Carter Bullard <carter at qosient.com>
> #
> 
> PATH="/usr/local/bin:$PATH"; export PATH
> package="argus-clients"
> version="3.0.8rc3"
> 
> OPTIONS="$*"
> FILE=
> while test $# != 0
> do
>    case "$1" in
>    -r) shift; FILE="$1"; break;;
>    esac
>    shift
> done
> 
> # Cluster ASN level data
> ASN_DIR=/net/nfdata/asn/`date +%Y/%m/%d`
> mkdir -p ${ASN_DIR}
> ASN_FILE=`basename ${FILE}`
> /usr/local/bin/racluster -m sas -r ${FILE} -w ${ASN_DIR}/${ASN_FILE}
> chgrp -R oititso ${ASN_DIR}
> chmod -R g+r ${ASN_DIR}
> 
> BASE=`basename ${FILE}`
> DIR=/net/nfdata/archive/`echo ${BASE}|sed -e 's/_.*//'|awk -F '-' '{print $1"/"$2"/"$3}'`
> mkdir -p $DIR
> /usr/local/bin/racluster -r ${FILE} -w ${DIR}/${BASE} && rm ${FILE}
> chgrp -R oititso ${DIR}
> find /net/nfdata/archive/ -type f -exec chmod -R 0640 {} \;
> find /net/nfdata/archive/ -type d -exec chmod -R 0750 {} \;
> 
>