BUG - No immediate exit when pipeing ra output

[elof2 at sentor.se]

Carter, do you have any comments to this?
I believe this is a major bug.


Two examples using the old ra v2.x :

1)
time ra -Zb -nr argus.log - ip | head -1
04-27-11 23:54:00.925018  s        tcp             10.10.10.10.46382 
->   20.20.20.20.1503   5467  5550  967561   2559850     PA_PA

real    0m0.004s
user    0m0.000s
sys     0m0.000s


2)
time ra -Zb -nr argus-20110428.log - ip | grep -q "PA_PA$"

real    0m0.004s
user    0m0.000s
sys     0m0.000s


Both examples only take 0.004s to execute.
Good!



Now I try the same using ra v3 :

1)
time ra -Zb -nr argus.log - ip | head -1
04-27-11 23:54:00.925018  s        tcp             30.30.30.30.32451 
->   40.40.40.40.25    52  97  1151   12274     PA_PA

real    0m50.043s
user    0m48.171s
sys     0m1.892s


2)
time ra -Zb -nr argus-20110429.13.log - ip | grep -q "PA_PA$"

real    0m50.161s
user    0m48.103s
sys     0m2.068s


The commands take 50 seconds (!!!) to complete when they should have taken 
approx 0.004 seconds.

The argus.log file is 96 MB and hold 828 372 rows of data.
(The larger the file, the longer I have to wait until the command 
terminates. If I have an old argus.log holding 24 hours of data, it will 
take ~30 minutes to parse it (even longer if it is bzipped))


It takes 50 seconds just to parse through 100% of the file:
time ra -nr argus.log - ip > /dev/null

real    0m50.257s
user    0m48.599s
sys     0m1.596s


I think this is a major bug since all kind of scripted processing using 
'ra' will take forever to execute.

Say you have a cron-job that execute every minute, performing a test to 
check the validity of the sniffed traffic and the SPAN setup using a 
'grep -q' and its exit status. This test will take 50 seconds 
instead of 0.004. That is bad enough.
Now imagine you add a second simillar test. Now the script will take 1 
min 40 s to execute. It won't have finished executing when cron starts up 
a new job. :-/
(Naturally I can use lockfiles to prevent the new job from executing 
its tests if the previous one hasn't finished. Also I can use the option 
-N 1 to ra in order to have it terminate swiftly, but that's workarounds 
and don't solve the real problem.)

I've tested ra v2 and v3 on both Linux and FreeBSD.
v3 takes 50 seconds on both platforms.
v2 takes 0.004 seconds on both platforms.


Have a nice weekend!

/Elof



On Tue, 26 Apr 2011, elof2 at sentor.se wrote:

>
> Hi!
>
> Both you and Mike missed my point.
>
> It is not the output of one line I'm asking for, it is immediate termination 
> of 'ra' as soon as the pipe command terminates.
>
> Here's another example:
>
> ra -r huge.logfile | grep -q "CON"
>
>
>
> The 'grep' command will exit as soon as it encounters the string "CON".
> One of the first lines of ra output probably contain "CON" so I would expect 
> my script to end there, but no, even though grep has exited and ra no longer 
> has anything to output to, ra still continue to execute and chew through the 
> logfile(s) to absolutely no use.
>
> If the log file is several GB, or if you have *lots* of logfiles, or if the 
> files are located on a slow NFS-mount, ra will continue to execute for 
> several minutes for something that typically should have taken less that 1 
> ms.
>
> /Elof
>
>
>
>
> On Tue, 26 Apr 2011, Mark Poepping wrote:
>
>> Use the record count feature.
>> 	ra  -N 1
>> 
>> -----Original Message-----
>> From: argus-info-bounces+poepping=cmu.edu at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+poepping=cmu.edu at lists.andrew.cmu.edu] On Behalf
>> Of elof2 at sentor.se
>> Sent: Tuesday, April 26, 2011 10:37 AM
>> To: Argus Development
>> Subject: [ARGUS] Immediate exit
>> 
>> 
>> Question (and feature request):
>> 
>> Is it possible to have 'ra' terminate immediately in cases similar to
>> this:
>> 
>> ra -r huge.logfile | head -1
>> 
>> 
>> There's no need for 'ra' to spend minutes chewing through the rest of the
>> logfile(s). The pipe command has terminated and is not waiting for any more
>> data.
>> 
>> /Elof
>> 
>> 
>> 
>