ra reads argus file very slow

[elof2 at sentor.se]

Just as a comparison, I just ran this on a 1.0 GB argus file:

# time ra -nr argus-20131103.09.log > /dev/null
real    0m51.314s
user    0m47.367s
sys     0m2.101s

I now cat a new file (not in any file cache) that is also 1.0 GB in size:

# time (sync; cat argus-20131103.10.log > /dev/null; sync)
real    0m6.124s
user    0m0.008s
sys     0m0.799s


This is on a FreeBSD 9.1 amd64 on an ordinary intel server that is pretty 
loaded at the time of the test.

/Elof


On Wed, 30 Oct 2013, Zi Hu wrote:

> Hi, Carter,
>
> In my application, I need a simple tool to read what it is in the argus
> file, then output certain fields that I am interested in ascii format, such
> as srcip, dstip, sport, dport. protocol, ....
>
> I thought the command "ra" is what I need. However, I find it is very slow
> to read the argus data with "ra".  I did a small experiment: dump the same
> argus file (about 2G) with both "ra" and "cat".
> Using the "ra" command, it took me about 87 minutes to read the file, while
> it took only 40 seconds to dump it with "cat".  and also I notice that the
> memory keeps growing when I am running "ra".
>
> zihu at proton:~$ time cat
> 2013-09-01-0700/temp/20130831-223000-hWukIYC-lander4.argus > temp.dat
>
> real    0m39.490s
> user    0m0.027s
> sys     0m4.204s
> zihu at proton:~$ time ra -r
> 2013-09-01-0700/temp/20130831-223000-hWukIYC-lander4.argus -u > temp.dat
>
> real    87m40.973s
> user    86m42.397s
> sys     0m56.256s
> zihu at proton:~$
>
>
>
> So I guess "ra" does more than just reading the argus file, formatting and
> outputing the result.   Does "ra" keep track of flows in memory so that the
> memory keeps growing ?
>
> If "ra" is not the right choice for my application, then what's the right
> command for this simple application? Or if we don't have such a tool, I am
> thinking of writing one by myself. Could you point me where to start?  Any
> suggestions are welcomed.
>
>
> Thanks
> -Zi
>