ranonymize too slow?

[Carter Bullard]

Hey Christos,
The primary demand in IP address anonymization is the number of IP addresses that need to be anonymized.   So how many addresses are in the file ??

  racount -M addr -r big.file

What version of clients are you using ??
Carter

> On Dec 1, 2014, at 1:14 AM, Christos Papadopoulos <christos at cs.colostate.edu> wrote:
> 
> Hi folks,
> 
> I am trying to use ranonymize for some large argus files. This is useful for us because we want to share some argus data with fellow researchers, but anonymize them to protect the innocent.
> 
> The file I am trying to anonymize is large, about 18GB compressed. As you can imagine, there are millions of flows in there.
> 
> I only want IP address anonymization, so I turned everything else off in the ranonymize.conf file.
> 
> Well, ranonymize has been running for almost 3 hours with about 1/20th of the file done. It is using 100% of a CPU, but only 4% of memory in a 32GB machine. Clearly it's not a memory or swap issue.
> 
> I can't figure out why it's taking so long. I thought it would be almost as fast as reading and writing the file plus some time to compress/decompress and some time for checking the hash for the anonymized addresses.
> 
> Any idea what's pounding the CPU and slowing it down? I can investigate further by profiling the code, but thought I throw the question out there first in case someone else has done it.
> 
> Thanks!
> 
> Christos.
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2443 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20141201/5710d523/attachment.bin>