rasort fix

Thu May 31 10:30:51 EDT 2001

Hey Russ,
   Yes CMU changed the name of the mailing list, and I didn't
obviously make the right changes to the web site.  This is now
fixed, so thanks for the pointers!

   OK, rasort() is having a problem because of a basic bug.  I'm
rewriting it for the next release, but there is a quick fix to
getting past the current problems, and the patch is included.
Basically I have a RA_MAXQUEUESIZE that is defined.  We start
with a small queue array, and as we read in more records, we
allocate more and more memory to accommodate the number of records
until, of course we reach the MAX.  The bug is such that when
the number of records equals RA_MAXQUEUESIZE, we can't increase
the size of the queue, so, unfortunately, we keep looking for
an empty slot, that of course isn't there.

   Increasing RA_MAXQUEUESIZE to a particularly large number will
solve the problem for the moment ;o)

Sorry for the inconvenience!!

Carter

Carter Bullard
QoSient, LLC
300 E. 56th Street, Suite 18K
New York, New York  10022

carter at qosient.com
Phone +1 212 588-9133
Fax   +1 212 588-9134
http://qosient.com 

Index: rasort.c
===================================================================
RCS file: /usr/local/cvsroot/argus/clients/rasort.c,v
retrieving revision 1.28.4.4
diff -r1.28.4.4 rasort.c
687c687
< #define RA_MAXQUEUESIZE         1048576
---
> #define RA_MAXQUEUESIZE         67108864

-----Original Message-----
From: Russ Harvey [mailto:russ at cornucopia.ucr.edu] 
Sent: Wednesday, May 30, 2001 11:21 PM
To: Carter Bullard
Subject: RE: rasort doesn't finish

Hi Carter,
Thanks for the fast reply. Per your suggestion I did try racount (and
ramon), which seemed to work and generated output. At this late date
perhaps it only adds some small credence to Peter's findings that it is
a size problem. You may not need this now, but I put one of the files
that seems to break rasort in ftp.qosient.com/incoming
(argus.2001.05.29.14.00.00.tmp), just in case.

On the subscription list issue, I was just a little confused by the
different references. On the main Argus web page,
qosient.com/argus/index.htm, there are a couple of places to find info
on subscribing. The FAQ, qosient.com/argus/faq.htm, says `send
"subscribe Argus" in the body of a piece of mail to
majordomo at lists.andrew.cmu.edu'. The how-to,
qosient.com/argus/how-to.htm, says `Send "subscribe argus" in the body
of a piece of mail to majordomo at lists.andrew.cmu.edu'. These are the
same, of course, but unfortunately the listserver at cmu rejects the
subscribe requests as `list unknown'.

There is another place that talks about argus lists, under the `support'
header on the main page (also the `mailing lists' footer at the bottom
of each page), `qosient.com/argus/mailinglists.htm'. This says:

`To subscribe to the Argus Development Mailing list, send an email to
majordomo at lists.andrew.cmu.edu and make sure the word "subscribe" is in
the body of your message.'

The confusing part to me was that the above description doesn't include
a list name to subscribe to. I eventually sent mail to
lists.andrew.cmu.edu asking for a list of the lists, and the only
relevant one was argus-info, to which I subscribed (I may have also
somehow subscribed to argus, even though cmu rejected that list name as
unknown, since I get two copies of the mail from you).

Sorry this was so long-winded, I just wanted to give you an accurate
picture of my confusion.

Thanks very much for an excellent network tool. Your ability to respond
to all problem mail on the list is amazing. I am obviously an argus
newbie, but hope to become better at using it soon. I think it is an
excellent addition to our monitoring set of tools, in particular it will
be very helpful in tracking attacks.

Thanks,
--russ

P.S.
I think the raw argus output file I left in incoming may have been
truncated. The ftp transfer I did resulted in:

`Could not preserve times for argus.2001.05.29.14.00.00.tmp: UTIME
failed.'

and ncftp reported 136Mb transfered instead of the actual file size of
about 143Mb. Sorry.

On May 30, 12:49pm, "Carter Bullard" wrote:
} Subject: RE: rasort doesn't finish
} Hey Russ,
}    This is indeed the right mailing list!  I'm sorry
} for the confusion, if you have any suggestions for
} making it a bit easier, I'd love to hear them!!
} 
}    Must be a bug either in the file parsing or the output
} process.  Argus records are all TLV records,
} (type/length/value), so if the length value is zero, or
} some particularly bad value, for some particularly bad 
} reason, it will have infinite loop problems.  We fixed a
} number of these problems last year, but you never know
} about bugs.
} 
}    Do you have a file that causes rasort problems that
} you don't mind sharing?  If so, please deposit it in
} ftp://qosient.com/incoming and send me mail.  I'll
} pick it up and take a look.
} 
}    Do other ra* programs have problems with the same
} files, such as racount()?  They all share similar
} file and record parsing routines.
} 
} Sorry for the inconvenience.
} 
} Carter
} 
} Carter Bullard
} QoSient, LLC
} 300 E. 56th Street, Suite 18K
} New York, New York  10022
} 
} carter at qosient.com
} Phone +1 212 588-9133
} Fax   +1 212 588-9134
} http://qosient.com 
} 
} 
} -----Original Message-----
} From: owner-argus-info at lists.andrew.cmu.edu
} [mailto:owner-argus-info at lists.andrew.cmu.edu] On Behalf Of Russ
Harvey } Sent: Wednesday, May 30, 2001 12:21 PM } To:
argus-info at lists.andrew.cmu.edu } Subject: rasort doesn't finish } 
} 
} Hi,
} I am not sure this is the correct list (there are confusing references
} to argus, argus development, and argus-info in the documentation), I }
apologize if it is not. } 
} I am running argus-2.0.1 on FreeBSD 4.2 (dual 500MHz Compaq), and I
have } implemented argus monitoring my border traffic. I used the }
support/Archive script to archive my argus output files once an hour, }
but sometimes rasort does not finish running on the raw output files and
} I must kill them off manually. I let a couple run for 12 hours or so,
} but they still did not finish. } 
} In checking the archives there were references to loops in ra on
FreeBSD } machines (and also implications that the problem had been
fixed), as } well as discussions of rasort dumping core. My hourly raw
argus data } files are about 150Mb each, as our traffic is about 10K
packets/sec in } and out (~50Mbps out, ~25Mbps in) } 
} Is there a way I can get rasort to complete, or should I just not run
} any post-processing from the archiving cron script? If this is a
looping } problem, how can I figure out what in my data is causing
rasort to get } confused? } 
} Thanks,
} --russ
}-- End of excerpt from "Carter Bullard"