Huge argus files and racluster
Marco
listaddr at gmail.com
Tue Feb 7 04:33:30 EST 2012
Hi, is there any limit on the size an argus file can have? I'm trying
to synthesize about 50GB of pcap data into argus data.
Doing a plain "argus -r hugefile.pcap -w argusdata.argus" works and
produces a 8GB argus file, but then if I try to run racluster on it to
reduce it further, it just seems to hang after a while.
I did some experiment, and I tried creating multiple argus files and
merge tem in successive iteration, but the final merge always fails.
When I say "fails", I mean that it looks like racluster is working up
to a point (ie, disk activity, strace shows it's reading data), but
from a certain point on, it just appears to be hung. Strace, shows
nothing, ltrace shows lots of stuff like
gettimeofday(0xcdd866b8, NULL)
= 0
pthread_mutex_lock(0x2303268, 1, 435251, 0, 0x2303268)
= 0
pthread_mutex_unlock(0x2303268, 1, 0xcdd866a0, 0xcdd866a0, 0x2303268)
= 0
pthread_mutex_lock(0x2303268, 0xcdd87830, 1, 0x4f2be088, 0)
= 0
pthread_mutex_unlock(0x2303268, 0xcdd87830, 1, 0xcdd866a0, 0x2303268)
= 0
gettimeofday(0xcdd87848, NULL)
= 0
pthread_mutex_lock(0x2303268, 1, 435628, 0, 0x2303268)
= 0
pthread_mutex_unlock(0x2303268, 1, 0xcdd87830, 0xcdd87830, 0x2303268)
= 0
pthread_mutex_lock(0x2303268, 0xcdd889c0, 1, 0x4f2be088, 0)
= 0
pthread_mutex_unlock(0x2303268, 0xcdd889c0, 1, 0xcdd87830, 0x2303268)
= 0
...
Currently, I've reduced the problem to the merging of two files:
$ ra -n -r 1_40.argus | wc -l
959586
$ ra -n -r 41_50.argus | wc -l
399927
Doing
$ racluster -r 1_40.argus -r 41_50.argus -w 1_50.argus
produces the behavior described above, and I have to kill racluster
with "kill -9".
Any help appreciated, thanks.
More information about the argus
mailing list