BUG - No immediate exit when pipeing ra output

[Carter Bullard]

Hey Elof2,
Well, major bugs are core dumps, incorrect data, that type of thing.
This is a minor bug, only because it is an optimization issue.
But it is a bug, none-the-less, and I'll fix it.

There are several approaches to solving your problem, that you
should consider.  ra* could do your "exit on first match" thing, which
is not a bad option.  Would you want to recommend that?  What command
line method would you want to use to turn it on?

Wanting the client to exit when the last output is closed is a good thing,
and I'll do that as soon as I get the time.

Carter


On Apr 29, 2011, at 9:38 AM, elof2 at sentor.se wrote:

> 
> Carter, do you have any comments to this?
> I believe this is a major bug.
> 
> 
> Two examples using the old ra v2.x :
> 
> 1)
> time ra -Zb -nr argus.log - ip | head -1
> 04-27-11 23:54:00.925018  s        tcp             10.10.10.10.46382 ->   20.20.20.20.1503   5467  5550  967561   2559850     PA_PA
> 
> real    0m0.004s
> user    0m0.000s
> sys     0m0.000s
> 
> 
> 2)
> time ra -Zb -nr argus-20110428.log - ip | grep -q "PA_PA$"
> 
> real    0m0.004s
> user    0m0.000s
> sys     0m0.000s
> 
> 
> Both examples only take 0.004s to execute.
> Good!
> 
> 
> 
> Now I try the same using ra v3 :
> 
> 1)
> time ra -Zb -nr argus.log - ip | head -1
> 04-27-11 23:54:00.925018  s        tcp             30.30.30.30.32451 ->   40.40.40.40.25    52  97  1151   12274     PA_PA
> 
> real    0m50.043s
> user    0m48.171s
> sys     0m1.892s
> 
> 
> 2)
> time ra -Zb -nr argus-20110429.13.log - ip | grep -q "PA_PA$"
> 
> real    0m50.161s
> user    0m48.103s
> sys     0m2.068s
> 
> 
> The commands take 50 seconds (!!!) to complete when they should have taken approx 0.004 seconds.
> 
> The argus.log file is 96 MB and hold 828 372 rows of data.
> (The larger the file, the longer I have to wait until the command terminates. If I have an old argus.log holding 24 hours of data, it will take ~30 minutes to parse it (even longer if it is bzipped))
> 
> 
> It takes 50 seconds just to parse through 100% of the file:
> time ra -nr argus.log - ip > /dev/null
> 
> real    0m50.257s
> user    0m48.599s
> sys     0m1.596s
> 
> 
> I think this is a major bug since all kind of scripted processing using 'ra' will take forever to execute.
> 
> Say you have a cron-job that execute every minute, performing a test to check the validity of the sniffed traffic and the SPAN setup using a 'grep -q' and its exit status. This test will take 50 seconds instead of 0.004. That is bad enough.
> Now imagine you add a second simillar test. Now the script will take 1 min 40 s to execute. It won't have finished executing when cron starts up a new job. :-/
> (Naturally I can use lockfiles to prevent the new job from executing its tests if the previous one hasn't finished. Also I can use the option -N 1 to ra in order to have it terminate swiftly, but that's workarounds and don't solve the real problem.)
> 
> I've tested ra v2 and v3 on both Linux and FreeBSD.
> v3 takes 50 seconds on both platforms.
> v2 takes 0.004 seconds on both platforms.
> 
> 
> Have a nice weekend!
> 
> /Elof
> 
> 
> 
> On Tue, 26 Apr 2011, elof2 at sentor.se wrote:
> 
>> 
>> Hi!
>> 
>> Both you and Mike missed my point.
>> 
>> It is not the output of one line I'm asking for, it is immediate termination of 'ra' as soon as the pipe command terminates.
>> 
>> Here's another example:
>> 
>> ra -r huge.logfile | grep -q "CON"
>> 
>> 
>> 
>> The 'grep' command will exit as soon as it encounters the string "CON".
>> One of the first lines of ra output probably contain "CON" so I would expect my script to end there, but no, even though grep has exited and ra no longer has anything to output to, ra still continue to execute and chew through the logfile(s) to absolutely no use.
>> 
>> If the log file is several GB, or if you have *lots* of logfiles, or if the files are located on a slow NFS-mount, ra will continue to execute for several minutes for something that typically should have taken less that 1 ms.
>> 
>> /Elof
>> 
>> 
>> 
>> 
>> On Tue, 26 Apr 2011, Mark Poepping wrote:
>> 
>>> Use the record count feature.
>>> 	ra  -N 1
>>> -----Original Message-----
>>> From: argus-info-bounces+poepping=cmu.edu at lists.andrew.cmu.edu
>>> [mailto:argus-info-bounces+poepping=cmu.edu at lists.andrew.cmu.edu] On Behalf
>>> Of elof2 at sentor.se
>>> Sent: Tuesday, April 26, 2011 10:37 AM
>>> To: Argus Development
>>> Subject: [ARGUS] Immediate exit
>>> Question (and feature request):
>>> Is it possible to have 'ra' terminate immediately in cases similar to
>>> this:
>>> ra -r huge.logfile | head -1
>>> There's no need for 'ra' to spend minutes chewing through the rest of the
>>> logfile(s). The pipe command has terminated and is not waiting for any more
>>> data.
>>> /Elof
>> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110429/9ce1a764/attachment.bin>