Anonymization of argus flow data
Carter Bullard
carter at qosient.com
Tue Oct 22 11:40:00 EDT 2013
Hey Kaustubh,
Well, you have 100M+ ip addresses to anonymize, while trying to
preserve the address hierarchy, so I'm not surprised that its going
to take a little time to process. We don't pre-allocate the
anonymized space, we allocate it, as needed by grabbing a new
class address, of the same class as the address to be anonymized,
when we need it from a limited pool of addresses. When
that pool runs out, we need to create lists of class A,B and C
addresses that we can use to map, and then start allocating
address mappings as new addresses arrive. You're running into
that process of creating new blocks of addresses to use. That
isn't cheap, as we're trying to make sure that we don't get
collisions, etc...
Now the freebie anonymization code isn't designed for that scale
of operation, as you have found out.
To anonymize and preserve the network hierarchy for 100M+
addresses should require you to pre-allocate the mappings
before you start to process the records, then doing the
record address transformations is just a lot of
lookup,write … lookup,write.
If you want to work on the address allocation logic, I'll
add it to the code, if you like. If not, I can get to it
after 3.0.8 comes out.
Carter
On Oct 22, 2013, at 11:19 AM, Kaustubh Gadkari <kaustubh at cs.colostate.edu> wrote:
> Carter,
>
> These are the results of racount -M addr on my input file:
>
> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
> sum 660346226 22569184689 13942009249 8627175440 23277747475402 12592489900324 10685257575078
> Address Summary
> IPv4 Unicast src 6305958 dst 47659327
> IPv4 Unicast This Network src 3 dst 0
> IPv4 Unicast Private src 16325 dst 932
> IPv4 Unicast Reserved src 9169301 dst 68904256
> IPv6 LinkLocal src 742 dst 0
> IPv6 Multicast Link Local src 0 dst 737
>
> Kaustubh
>
> On Oct 15, 2013, at 1:27 PM, Kaustubh Gadkari <kaustubh at CS.ColoState.EDU> wrote:
>
>>
>> On Oct 15, 2013, at 1:24 PM, Carter Bullard <carter at qosient.com> wrote:
>>
>>> Hey Kaustubh,
>>> I haven't found anything that would generate obvious delays in the algorithms.
>>> How many IP addresses are we talking about??
>>>
>>> racount -M addr -r input.argus
>>>
>>
>> I am not sure. I have run this before and the process never finished (I let it run for about 2 hours before killing it), so I expect we have quite a large number of IP addresses. I can racount and let it run and I'll post results when it finishes.
>>
>> Kaustubh
>>
>>> Carter
>>>
>>>
>>> On Oct 8, 2013, at 2:11 PM, Kaustubh Gadkari <kaustubh at cs.colostate.edu> wrote:
>>>
>>>> Hey Carter,
>>>>
>>>> I don't have a .rarc file, and I am not setting RA_PRINT_NAMES explicitly anywhere. My invocation of ranonymize is as follows:
>>>>
>>>> ranonymize -f /path/to/configfile -r input.argus -w output.argus - <filter expression>
>>>>
>>>> The config file has the following entries:
>>>> RANON_PRESERVE_ETHERNET_VENDOR=yes
>>>> RANON_PRESERVE_BROADCAST_ADDRESS=yes
>>>> RANON_NET_ANONYMIZATION=sequential
>>>> RANON_HOST_ANONYMIZATION=sequential
>>>> RANON_PRESERVE_NET_ADDRESS_HIERARCHY=class
>>>>
>>>> Thanks,
>>>> Kaustubh
>>>>
>>>>
>>>> On Oct 8, 2013, at 10:04 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>
>>>>> Hey Kaustubh,
>>>>> There is a chance that if you run ranonymize() with the options to
>>>>> print hostnames, either in the .rarc file or using the -nn option
>>>>> on the command line, you will hurt ranonymize's performance by doing
>>>>> bind lookups on each address before the number is translated.
>>>>>
>>>>> Any chance that is going on here? What is the value of your RA_PRINT_NAMES
>>>>> variable in your .rarc, and/or how are you calling ranonymize() ?
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>> On Oct 8, 2013, at 8:45 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>
>>>>>> Hey Kaustubh,
>>>>>> I have not had a chance, but thanks for reminding me.
>>>>>> I'll look at it today !!!! Keep bugging me !!!
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>>> On Oct 7, 2013, at 12:39 PM, Kaustubh Gadkari <kaustubh at cs.colostate.edu> wrote:
>>>>>>>
>>>>>>> Hey Carter,
>>>>>>>
>>>>>>> I just wanted to check if you've found any reasons why ranonymize is taking so long to complete on my dataset?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kaustubh
>>>>>>>
>>>>>>>> On Sep 10, 2013, at 10:40 AM, Kaustubh Gadkari <kaustubh at CS.ColoState.EDU> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Sep 10, 2013, at 9:33 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>
>>>>>>>>> Well,
>>>>>>>>> On my system 80% of the cycles are being spent doing the address,
>>>>>>>>> port, mac, AS number mappings (managing allocation of a new object
>>>>>>>>> and caching the values), and a small amount on the lookups.
>>>>>>>>>
>>>>>>>>> I'll work on profiling the mapping logic to see if we've got
>>>>>>>>> something askew.
>>>>>>>>
>>>>>>>> Great. Thanks again for the help.
>>>>>>>>
>>>>>>>>> Hope all is most excellent,
>>>>>>>>
>>>>>>>> And with you too :)
>>>>>>>>
>>>>>>>> Kaustubh
>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Sep 10, 2013, at 12:22 PM, Kaustubh Gadkari <kaustubh at CS.ColoState.EDU> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Sep 10, 2013, at 8:40 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey Kaustubh,
>>>>>>>>>>> I've been profiling ranonymize() with a lot of data, and
>>>>>>>>>>> while I do see opportunities to improve performance, I don't
>>>>>>>>>>> see many massively inefficient parts of the code, when run
>>>>>>>>>>> against my data sets. There are still some things for
>>>>>>>>>>> me to look at, so I wanted you to know that I'm working on
>>>>>>>>>>> your problem.
>>>>>>>>>>
>>>>>>>>>> Thanks for looking at this, Carter.
>>>>>>>>>>
>>>>>>>>>>> Based on what you've seen me so far, you're machine is 85%
>>>>>>>>>>> idle, is ranonymize() using 100% of a single core, or is it
>>>>>>>>>>> sleeping a lot?
>>>>>>>>>>
>>>>>>>>>> top says ranonymize is using 100% of a single core.
>>>>>>>>>>
>>>>>>>>>>> What kind of machine are you running on?? Can you describe the
>>>>>>>>>>> machine a bit? CPUs, memory, disks, etc….
>>>>>>>>>>
>>>>>>>>>> I've been testing this on two machines. One is a Dell PowerEdge 2970, with 2 quad core AMD Opteron processors. The machine has 32GB RAM, a 130GB system disk and 16 8TB RAID5 partitions. The other machine is a Dell PowerEdge 2950. It has 2 quad core Intel Xeon X5450 CPUs, with 32GB RAM, a 140GB system disk and 3 8TB RAID5 partitions.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Kaustubh
>>>>>>>>>>
>>>>>>>>>>> Carter
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Sep 3, 2013, at 3:05 PM, Kaustubh Gadkari <kaustubh at cs.colostate.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 3, 2013 at 12:33 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>>>> Hmmmmmm, well, you're not using the machine much (85% idle)
>>>>>>>>>>>>> so I'm looking into whether we're making any calls to any
>>>>>>>>>>>>> routines that would add some wait states, like name lookups, or
>>>>>>>>>>>>> sleeping somewhere.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lets assume that there is a big problem, and I'll try to make
>>>>>>>>>>>>> some changes to improve your performance.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Carter.
>>>>>>>>>>>>
>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>
>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sep 3, 2013, at 1:57 PM, Kaustubh Gadkari <kaustubh at cs.colostate.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Carter,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sep 3, 2013, at 11:36 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Kaustubh,
>>>>>>>>>>>>>>> If its still writing records to the output file, its not in an infinite loop,
>>>>>>>>>>>>>>> although I'm sure that it feels like one. So, no need to print debug msgs
>>>>>>>>>>>>>>> or run under gdb().
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hmmmmmm, you must have a very large number of IP addresses. racount() isn't doing
>>>>>>>>>>>>>>> anything exotic with the "-M addr" mode. Its hashing and storing each unique
>>>>>>>>>>>>>>> IP address, so that we can report on how many and what types.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My guess is that you must be short on physical memory, and the programs are swapping,
>>>>>>>>>>>>>>> which means that everything on this machine will be going very slowly.
>>>>>>>>>>>>>>> Run " top " to see if one of our programs is eating all the memory, or
>>>>>>>>>>>>>>> use vmstat() or vm_stat() or whatever to see if there is any paging.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, the machine is not running out of memory. ranonymize is the largest memory user, and it is using 42.1% of a total of 32GB RAM. The swap usage is only 205MB, which is OK. vmstat shows me the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kaustubh at proton:~$ sudo vmstat -w
>>>>>>>>>>>>>> procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
>>>>>>>>>>>>>> r b swpd free buff cache si so bi bo in cs us sy id wa st
>>>>>>>>>>>>>> 1 0 205916 1638176 101636 16287400 0 0 527 342 1 1 14 0 85 1 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are no other memory intensive processes running on the box.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If it is a memory problem, then you will need to subdivide the data based
>>>>>>>>>>>>>>> on size, not on time, using rasplit(). And yes its easy to merge split files
>>>>>>>>>>>>>>> back to a single file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> UNFORTUNATELY, because the scope of anonymization is the file, anonymizing a
>>>>>>>>>>>>>>> single big file of records will generate different results compared to
>>>>>>>>>>>>>>> anonymizing a set of split files created from the big file. Address A will be
>>>>>>>>>>>>>>> anonymized potentially to a different address in each file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The configuration provides the means to get consistent results between files,
>>>>>>>>>>>>>>> but its a bit of work to do so.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you think you're running out of memory?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, I think I'm ok in terms of memory usage.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sep 3, 2013, at 1:11 PM, Kaustubh Gadkari <kaustubh.gadkari at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Sep 3, 2013 at 8:49 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>>>>>>>> Hmmm, if racount() takes 18min, I would think ranonymize() should take about 20min
>>>>>>>>>>>>>>>>> to complete. You can run " racount -M addr " to get racount() to printout address
>>>>>>>>>>>>>>>>> information, like how many addresses are in the file.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carter, I ran racount with -M addr, but the process hasn't finished
>>>>>>>>>>>>>>>> yet (it's been running for about 90 min now). I'll let it run for a
>>>>>>>>>>>>>>>> while longer and keep you updated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ranonymize() works on a single argus record at a time, reading a single record,
>>>>>>>>>>>>>>>>> anonymizing all the various data elements, and then writing the anonymized
>>>>>>>>>>>>>>>>> record out to the output file. If ranonymize() hasn't written out a record recently,
>>>>>>>>>>>>>>>>> then its possible that its in an infinite loop, especially if its running at 100%, and
>>>>>>>>>>>>>>>>> its been running for a month, and it seems to have stopped writing into the file.
>>>>>>>>>>>>>>>>> What was the last " modified " time on your output file ???
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It hasn't stopped writing to file .. the last modified time is right
>>>>>>>>>>>>>>>> now, since the process is still running.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If you've compiled debug support into your ra* programs, you can send a USR1
>>>>>>>>>>>>>>>>> signal to the running ranonymize() and it will start writing debug information out
>>>>>>>>>>>>>>>>> to stderr(). Send a USR2 to turn debug output off. Assuming that ranonymize()s
>>>>>>>>>>>>>>>>> process id is 35122, you can do this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> % kill -USR1 35122
>>>>>>>>>>>>>>>>> % kill -USR2 35122
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If you've compiled development support into your programs, you can attach
>>>>>>>>>>>>>>>>> to ranonymize() using gdb(), and then step through the program to see where
>>>>>>>>>>>>>>>>> it is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I haven't compiled my ra* programs with debug or development support.
>>>>>>>>>>>>>>>> If you can tell me what I need to change in the Makefiles, I can do so
>>>>>>>>>>>>>>>> and run ranonymize with gdb and see what's happening.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> % gdb ranonymize 35122
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This will attach to the program, and stop the acitve process. If this all seems
>>>>>>>>>>>>>>>>> unfamiliar, send more email, and I'll walk you through one of these strategies.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sep 3, 2013, at 9:56 AM, Kaustubh Gadkari <kaustubh.gadkari at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Sep 3, 2013 at 7:19 AM, Kaustubh Gadkari
>>>>>>>>>>>>>>>>>> <kaustubh.gadkari at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> On Tue, Sep 3, 2013 at 6:00 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>>>>>>>>>>> Hmmmm,
>>>>>>>>>>>>>>>>>>>> There shouldn't be any performance issues with anonymizing a file, if your
>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> anonymizing the IP addresses. How many addresses are in the file?
>>>>>>>>>>>>>>>>>>>> What does your ranonymize.conf file look like? How much memory is it
>>>>>>>>>>>>>>>>>>>> using?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am not quite sure how many IP addresses there are in the file. My
>>>>>>>>>>>>>>>>>>> ranonymize.conf looks like this:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> RANON_PRESERVE_ETHERNET_VENDOR=yes
>>>>>>>>>>>>>>>>>>> RANON_PRESERVE_BROADCAST_ADDRESS=yes
>>>>>>>>>>>>>>>>>>> RANON_NET_ANONYMIZATION=sequential
>>>>>>>>>>>>>>>>>>> RANON_HOST_ANONYMIZATION=sequential
>>>>>>>>>>>>>>>>>>> RANON_PRESERVE_NET_ADDRESS_HIERARCHY=class
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I took a look at how much memory ranonymize is using .. the usage is
>>>>>>>>>>>>>>>>>>> about 42% on a machine with 32GB RAM.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ranonymize() can be a little complex O(nLogN + C), but it should be
>>>>>>>>>>>>>>>>>>>> in the same time frame as racount(). How long does it take for racount()
>>>>>>>>>>>>>>>>>>>> to read the file?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am running racount right now .. I will post results once it finishes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> racount takes about 18min to run on the file:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> real 17m58.528s
>>>>>>>>>>>>>>>>>> user 17m12.413s
>>>>>>>>>>>>>>>>>> sys 2m0.332s
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Just a rule of thumb. If a ra* program doesn't complete in a few minutes,
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> should stop it and try to figure out if there is a memory problem or not.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks, I'll keep this in mind :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sep 2, 2013, at 2:20 PM, Kaustubh Gadkari <kaustubh.gadkari at gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have a set of argus flow data captured at our data capture vantage point,
>>>>>>>>>>>>>>>>>>>> and I want to anonymize the IP addresses (both source and destination) fully
>>>>>>>>>>>>>>>>>>>> i.e. I want to replace both the addresses, using a prefix preserving
>>>>>>>>>>>>>>>>>>>> technique. I have tried using ranonymize, but it is taking an extremely long
>>>>>>>>>>>>>>>>>>>> time to anonymize the file (I started the process a couple of months ago, on
>>>>>>>>>>>>>>>>>>>> a ~125GB file, and the output file size today is only ~30GB).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can anyone suggest the right way to go about anonymizing the data set I
>>>>>>>>>>>>>>>>>>>> have? Is ranonymize the right tool for the job?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Kaustubh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>>>>> kaustubh at cs.colostate.edu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Kaustubh Gadkari
>>>>>>>>>> kaustubh at cs.colostate.edu
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kaustubh Gadkari
>>>>>>>> kaustubh at cs.colostate.edu
>>>>>>>
>>>>>>> --
>>>>>>> Kaustubh Gadkari
>>>>>>> kaustubh at cs.colostate.edu
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Kaustubh Gadkari
>>>> kaustubh at cs.colostate.edu
>>>>
>>>
>>
>> --
>> Kaustubh Gadkari
>> kaustubh at cs.colostate.edu
>>
>
> --
> Kaustubh Gadkari
> kaustubh at cs.colostate.edu
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131022/af8ea8a6/attachment.bin>
More information about the argus
mailing list