Huge argus files and racluster

[Carter Bullard]

Hey Marco,
No limit in size, other than a default racluster.1 will consume all your RAM and
start swapping, which is what looks to be the issue.  You are tracking too many flows
for the memory available on your machine.

You should take the huge files that you have and split them into more
manageable files using rasplit.1.

   rasplit -r 1_40.argus 41_50.argus -M time 5m -w split/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S

This will split the data into daily directories, with a file for each 5 minutes in the day.
Then have racluster.1 process these individual files.  The "-M ind" is really important,
without that, you not be doing anything different.

   racluster -R split -M ind replace

This will replace the files you have in the daily archive with aggregated files.
You can then merge the files back into a single file if you like:

   ra -R split -w all.the.argus.data

Carter



On Feb 7, 2012, at 4:33 AM, Marco wrote:

> Hi, is there any limit on the size an argus file can have? I'm trying
> to synthesize about 50GB of pcap data into argus data.
> Doing a plain "argus -r hugefile.pcap -w argusdata.argus" works and
> produces a 8GB argus file, but then if I try to run racluster on it to
> reduce it further, it just seems to hang after a while.
> 
> I did some experiment, and I tried creating multiple argus files and
> merge tem in successive iteration, but the final merge always fails.
> When I say "fails", I mean that it looks like racluster is working up
> to a point (ie, disk activity, strace shows it's reading data), but
> from a certain point on, it just appears to be hung. Strace, shows
> nothing, ltrace shows lots of stuff like
> 
> gettimeofday(0xcdd866b8, NULL)
>                                               = 0
> pthread_mutex_lock(0x2303268, 1, 435251, 0, 0x2303268)
>                                               = 0
> pthread_mutex_unlock(0x2303268, 1, 0xcdd866a0, 0xcdd866a0, 0x2303268)
>                                               = 0
> pthread_mutex_lock(0x2303268, 0xcdd87830, 1, 0x4f2be088, 0)
>                                               = 0
> pthread_mutex_unlock(0x2303268, 0xcdd87830, 1, 0xcdd866a0, 0x2303268)
>                                               = 0
> gettimeofday(0xcdd87848, NULL)
>                                               = 0
> pthread_mutex_lock(0x2303268, 1, 435628, 0, 0x2303268)
>                                               = 0
> pthread_mutex_unlock(0x2303268, 1, 0xcdd87830, 0xcdd87830, 0x2303268)
>                                               = 0
> pthread_mutex_lock(0x2303268, 0xcdd889c0, 1, 0x4f2be088, 0)
>                                               = 0
> pthread_mutex_unlock(0x2303268, 0xcdd889c0, 1, 0xcdd87830, 0x2303268)
>                                               = 0
> ...
> 
> Currently, I've reduced the problem to the merging of two files:
> 
> $ ra -n -r 1_40.argus | wc -l
> 959586
> $ ra -n -r 41_50.argus | wc -l
> 399927
> 
> Doing
> 
> $ racluster -r 1_40.argus -r 41_50.argus -w 1_50.argus
> 
> produces the behavior described above, and I have to kill racluster
> with "kill -9".
> 
> Any help appreciated, thanks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120207/2d0bd8b1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120207/2d0bd8b1/attachment.bin>