rastream exits after multi day run

[Jesse Bowling]

Hi,

I’ve had an issue using rastream recently. What I’m seeing is that rastream will run successfully for a while (usually around 2 days or less) and then will simply exit. The only log messages I get are:

Oct 15 13:06:24 netf-test radium[33140]: 2014-10-15 13:06:24.908361 started
Oct 15 13:06:29 netf-test radium[33140]: 2014-10-15 13:06:29.599856 connect from localhost[127.0.0.1]
Oct 16 10:52:59 netf-test radium[33140]: 2014-10-16 10:52:59.537044 ArgusWriteOutSocket(0xb81269d0) client not processing: disconnecting

I tried running under gdb, but it quickly finishes/crashes...I’d can run it with -D, but unsure what level might catch what’s happening. Any recommendations? 

Any ideas of why this might be exiting? Is the bash script too heavy? Would it help to call a wrapper script that simply backgrounds the actual script I want to run?

Config information below.

Cheers,

Jesse

$ egrep -v '^#|^[ ]*$' /etc/radium.conf 
RADIUM_DAEMON="yes"
RADIUM_MONITOR_ID=10.138.3.168
RADIUM_MAR_STATUS_INTERVAL=60
RADIUM_ARGUS_SERVER=cisco://10.138.3.168:9995
RADIUM_ACCESS_PORT=561
RADIUM_BIND_IP=127.0.0.1
RADIUM_CLASSIFIER_FILE=/etc/ralabel.conf

$ egrep -v '^#|^[ ]*$' /etc/ra.conf
RA_SET_PID="no"
RA_PID_PATH="/var/run"
RA_RUN_TIME=0
RA_GENERATE_BIN_MAR_RECORDS=yes
RA_PRINT_LABELS=0
RA_FIELD_SPECIFIER="stime:25 flgs proto saddr sport sco sas dir daddr dport dco das spkts dpkts sbytes dbytes"
RA_FIELD_DELIMITER=''
RA_SEPARATE_ADDR_FROM_PORT_WITH_PERIOD="no"
RA_PRINT_NAMES='none'
RA_PRINT_RESPONSE_DATA=no
RA_PRINT_UNIX_TIME=no
RA_TIME_FORMAT="%F %T.%f"
RA_USEC_PRECISION=6
RA_USERDATA_ENCODE=Ascii
RA_SORT_ALGORITHMS="dpkts spkts saddr dport daddr sport"
RA_UPDATE_INTERVAL=2
RA_DELEGATED_IP="/usr/local/argus/delegated-ipv4-latest"
RA_RELIABLE_CONNECT=yes
RA_COLOR_SUPPORT="yes"
RA_COLOR_CONFIG=/etc/racolor.conf
RA_LOCAL=/usr/local/argus/local.addrs

Starting like this:

/usr/local/sbin/radium -f /etc/radium.conf
/usr/local/bin/rastream -F /etc/ra.conf -M time 5m -B 10s -S 127.0.0.1 -w /srv/scratch/%Y-%m-%d_%H-%M-%S.argus -f /argus/process_argus_file.bash -d

Contents of process_argus_file.bash:

#!/bin/bash

#
#  Argus Client Software.  Tools to read, analyze and manage Argus data.
#  Copyright (C) 2000-2014 QoSient, LLC.
#  All Rights Reserved
#
# Script called by rastream, to process files.
#
# Since this is being called from rastream(), it will have only a single
# parameter, filename,
#
# Carter Bullard <carter at qosient.com>
#

PATH="/usr/local/bin:$PATH"; export PATH
package="argus-clients"
version="3.0.8rc3"

OPTIONS="$*"
FILE=
while test $# != 0
do
    case "$1" in
    -r) shift; FILE="$1"; break;;
    esac
    shift
done

# Cluster ASN level data
ASN_DIR=/net/nfdata/asn/`date +%Y/%m/%d`
mkdir -p ${ASN_DIR}
ASN_FILE=`basename ${FILE}`
/usr/local/bin/racluster -m sas -r ${FILE} -w ${ASN_DIR}/${ASN_FILE}
chgrp -R oititso ${ASN_DIR}
chmod -R g+r ${ASN_DIR}

BASE=`basename ${FILE}`
DIR=/net/nfdata/archive/`echo ${BASE}|sed -e 's/_.*//'|awk -F '-' '{print $1"/"$2"/"$3}'`
mkdir -p $DIR
/usr/local/bin/racluster -r ${FILE} -w ${DIR}/${BASE} && rm ${FILE}
chgrp -R oititso ${DIR}
find /net/nfdata/archive/ -type f -exec chmod -R 0640 {} \;
find /net/nfdata/archive/ -type d -exec chmod -R 0750 {} \;