A couple of labeling questions/issues...

Wed Mar 13 12:06:04 EDT 2013

Hey Craig,
Here is the deal with all the labeling and country code issues.

The problem is that we implemented country code labeling as a generic label, and there were assumptions regarding what the labels looked like.  This worked very well, until recently, when we extended labeling with a bunch more label sources and formats, to exploit the general implementation that we had done.

As a result, when you do IANA based labels and RIR based labels, both at the same time, because they are both address based labels (i.e. this CIDR, this range, this specific IP address gets this label) , we use the same Patricia Tree data structure to hold the data.  No problem, actually not a bad idea.   Bur currently, when we go to read the country code from the general label structure, we expect it to be at a specific place.

However, when you provide your own labels, this convention breaks, and our methods break.  I.e., we get your bug, where your country codes look like arbitrary 2 character snippets from the general label structure.

OK, now, the fix...... I'm going to continue to overload the address Patricia Tree, as you get some great side effects doing this.  But I have to add country codes as a specific object in the generic address structures.  I should have a working fix by this afternoon, and will upload as either a refreshed 3.0.7.6 or a new 3.0.7.7.  (I haven't really announced 6, so was hoping to just reuse it ).

Thanks for all the data, and the config files.  Makes fixing bugs actually possible !!!!!

Carter

On Mar 13, 2013, at 10:49 AM, Carter Bullard <carter at qosient.com> wrote:

> Hey Craig,
> Status.....
> 
> For this problem
>> I’ve tried using the –M label=”regex” option, but the regex never seems to match.
>>  
>> ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf-n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn’t match the following events:
> 
> The -M label="regex" test is done on input, but its ralabel() that is putting in the label, so the option
> given to ralabel() cannot work, even theoretically ;O).  Need to pipe the ralabel() output to
> something that can do the test.   This would work:
> 
>    ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -w - |  \
>           ra -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"
> 
> definitely works with the example files you provided.  Sorry I didn't see the problem immediately.
> With this type of strategy, using radium() to label the records make some sense.  That way you
> would just run:
> 
>           ra -S radium -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"
> 
> To get your blacklisted flows.  Still working on the country code problem.
> 
> Carter
> 
> 
> 
> On Mar 13, 2013, at 10:35 AM, Craig Merchant <cmerchant at responsys.com> wrote:
> 
>> Yeah, that covers it.  Or more precisely, when I use “-s +sco,+dco”, one of the country code fields is incorrect and the incorrect field seems to be populated with characters from the label.
>>  
>> I experimented early on with the argus-flow file, but once I realized how much easier it was to manipulate my data into the iana formatted files, I stopped playing around with it.  So, I don’t really have an argus-flow file to speak of.
>>  
>> Thanks!
>>  
>> Craig
>>  
>> From: Carter Bullard [mailto:carter at qosient.com] 
>> Sent: Wednesday, March 13, 2013 7:16 AM
>> To: Craig Merchant
>> Cc: Argus (argus-info at lists.andrew.cmu.edu)
>> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>>  
>> Hey Craig,
>> Yes I did get your Argus.zip file.  No email files showed up.
>>  
>> While your ralabel.conf has RALABEL_ARGUS_FLOW=no, you still refer to
>> an argus-flow file specification, but you didn't send that.  Can you provide?
>> I'll test to see if that causes an issue (directive = no, but conf provided ).
>>  
>> It will take me a little while to wade through this, but to keep focus on the issue,
>> your bug is that you are seeing incorrect country codes being reporred with -s +sco +dco ?
>> And you are not getting correct grep matches using the " -M label="regex" " ?
>>  
>> Does that cover it ?
>>  
>> Carter
>>  
>> On Mar 11, 2013, at 8:35 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>> 
>> 
>> I just posted Argus.zip to your FTP server.  It contains all of my config files, scripts called by rastream, our label file, and some flow data.  Let me know if you need anything else.
>>  
>> Thanks!
>>  
>> Craig
>>  
>> From: Carter Bullard [mailto:carter at qosient.com] 
>> Sent: Monday, March 11, 2013 11:24 AM
>> To: Craig Merchant
>> Cc: Argus (argus-info at lists.andrew.cmu.edu)
>> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>>  
>> Hey Craig,
>> Fractured pelvis is pretty tough.  New hip sounds traumatic, but they
>> get you going very early.  Not really sure how I'm able to do it, it does
>> seem like so early given what they did.
>>  
>> ftp the files to ftp://qosient.com/incoming
>> Its a blind directory.
>>  
>> Thanks!!!!!
>>  
>> Carter
>>  
>> On Mar 10, 2013, at 5:40 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>> 
>> 
>> 
>> Given the surgery you’re recovering from, your continued support has been quite admirable!  My doctor won’t even let me start on any kind of physical therapy for the fractured pelvis for at least another couple of weeks.  I’m going a bit stir crazy myself.
>>  
>> I’ll experiment with your suggestions this afternoon.  I’ll send you some flow records, my label file, and all of the scripts I’ve been using later today.  Do have any idea what the largest attachment is that your email server will accept?  I can also share that via Dropbox if you’d like.
>>  
>> Thanks again!
>>  
>> Craig
>>  
>> From: Carter Bullard [mailto:carter at qosient.com] 
>> Sent: Sunday, March 10, 2013 9:57 AM
>> To: Craig Merchant
>> Cc: Argus (argus-info at lists.andrew.cmu.edu)
>> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>>  
>> Hey Craig,
>> Sorry for the delayed response, therapy is a bit time consuming.
>> OK, well the you've described about 20 issues, so which one would you like to work on first?
>>  
>> So, the file format is the NIC registrar's RIR file format.  See this link for a description:
>>    https://www.apnic.net/publications/media-library/documents/resource-guidelines/rir-statistics-exchange-format
>>  
>> ZZ is used when looking up a country code results in no returned value.
>>  
>> Some of the values you're interested in are labels, but some are not.  Country codes are not labels, there is
>> a COCODE DSR that holds the source and destination 2 or three character country codes.  If the country codes
>> are screwed up, you may need to remove the errant data, in order to get new values added.
>>  
>> I'm worried that you're use of " -M dsrs='+..." could have inserted country code DSRs without initalized values.
>> So run this with your ralabel() command to see if you get something better:
>>  
>>    ralabel -M dsrs="-cocode" .............
>>  
>>  
>> Of course, I can't comment on your problems, until you provide the complete configurations, and argus data
>> that I can use to recreate the problem.  Then I can try to fix it ?
>>  
>>  
>> Carter
>>  
>> On Mar 9, 2013, at 6:14 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>> 
>> 
>> 
>> 
>> Hey, Carter…  I’ve been testing the labeling functionality and trying to investigate further the country code issue I posted about earlier in the week.
>>  
>> I’m currently using the delegated-ipv4-latest file that is included with the 3.0.7.5 client.  I’ve noticed that Argus isn’t finding a country code match for our public IP range.  I’d like to add those ranges to the delegated-ipv4-latest, but I’m not sure what the format is.  Is there any way to make the file recognize a subnet mask?  I notice that the entry for 10.0.0.0 isn’t causing the country codes for my internal networks to show up as “ZZ”.    What do the numbers in the second and third to last columns represent? 
>>  
>> delegated-afrinic-latest:afrinic|ZA|ipv4|41.73.32.0|8192|20100112|allocated
>>  
>> Is it possible to use the MaxMind GeoIP database for country codes instead of the iana file?  I compiled the clients with GeoIP support.  The ralabel.conf file says the GEOIP labels will be saved as scity,dcity,icity.  I’ve tried adding those fields to my searches and nothing shows up.  I looked through rarc.print.all.conf to see if I could find a field in there that looked related to GeoIP data, but nothing popped out at me.  I’m using the same GeoLiteCity.dat file that we use on our Splunk server.  The GeoIP part of my ralabel.conf is:
>>  
>> RALABEL_GEOIP_CITY="saddr,daddr,inode:cco,cname"
>> RALABEL_GEOIP_CITY_FILE="/usr/local/argus/GeoIPCity.dat"
>>  
>> I’ve tried using the –M label=”regex” option, but the regex never seems to match.
>>  
>> ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn’t match the following events:
>>  
>> 1362868911.000236, e s      ,tcp,199.7.204.127,39935, ->,82.98.86.167,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
>> 1362868911.000282, e s      ,tcp,199.7.204.127,56477, ->,82.98.86.161,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
>> 1362868912.000211, e s      ,tcp,199.7.202.248,44177, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
>> 1362868913.000465, e s      ,tcp,199.7.204.127,42282, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
>>  
>> The events above also show the issue with country codes.  I’m noticing a few things…  The erroneous country codes are always a pair of letters found somewhere in the label, but I don’t see any pattern as to which letters are used.  If I disable the iana label file in ralabel.conf, country codes display normally.  If a country code is found for both a source and destination, one of the country codes is always incorrect, but the country code is properly displayed in the label field:
>>  
>> 1362869594.000036, e        ,udp,12.130.136.11,53, ->,69.154.227.43,53,1,70,INT,pu,US,saddr=US,public,DC1,mta-external:daddr=US
>> 1362869594.000036, e        ,udp,12.130.136.12,53, ->,200.26.226.6,53,1,68,INT,pu,AN,saddr=US,public,DC1,mta-external:daddr=AN
>>  
>> In the first set of events, both country codes are incorrect (dest code not shown).  In the events above, the destination code is shown correctly.  I have noticed that if, as above, the 12.130.136.12 has its sco field displayed incorrectly, it will also have its dco field displayed incorrectly when it is the destination of a flow.
>>  
>> If there is no label for a source or destination, the country code for that host will be empty:
>>  
>> 1362866671.000438, e        ,udp,74.125.17.82,42492,<->,192.168.30.41,53,4,1260,CON,,dn,daddr=internal,DC4,SP1--Apache,dns,ext-DC4
>>  
>> I enabled label support and radium and ran the same query as above, except with ra.  The issue with the country codes is the same as when I run the searches with ralabel.  Radium has a fairly significant memory leak when labels are enabled. 
>>  
>> Let me know if you would like me to send you any binary flow files or my iana label file.
>>  
>> Thanks!
>>  
>> Craig
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130313/c0f0f0a4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130313/c0f0f0a4/attachment.bin>