A couple of labeling questions/issues...

Craig Merchant cmerchant at responsys.com
Fri Mar 15 20:30:16 EDT 2013


Hey, Carter...

I tried the latest version and everything looks great.  Ralabel properly formatted the sco and dco fields.  I also configured radium to do the labeling and they showed up properly there as well (and with no memory leak).

What is the proper "order" for the IANA file?  I tried to put the subnets at the top (descending in size) and hosts at the bottom, though I've gotten some spreadsheets from users with additional host labels they wanted included.  So those probably just got appended to the end of the list.

Thanks for all your help.  I hope you have a great weekend!

C

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, March 15, 2013 8:23 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
OK, so I found a set of bugs in the IANA and RIR file parsing logic.  You had a bunch of
network entries, in your files, that were " out of order ", and duplicates, which caused
some really serious address tree node insertion problems.

All fixed now, and working with the files that you provided, ... Thanks!!!
With this version, you should get your labels, and good country codes.

I have uploaded a new argus-clients-3.0.7.6.tar.gz, which fixes these problems, and
so you should grab it to test:

   http://qosient.com/argus/dev/argus-clients-3.0.7.6.tar.gz

Now there has been a copy of argus-clients-3.0.7.6.tar.gz on this server for a while,
but because I didn't announce it,  I feel OK with refreshing this copy.   If you grabbed
it before, grab it again.

Yell if there are any problems.

Carter

On Mar 13, 2013, at 12:11 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:


Thank you, Carter!  I'm glad my data and configs were useful to you.

I get what you're saying about the -M label="xyz" issue.  I know that I had it working at some point and thought I must be taking crazy pills, but it must have worked for me when I had radium applying the label file.

I appreciate all the support...

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com>]
Sent: Wednesday, March 13, 2013 9:06 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Here is the deal with all the labeling and country code issues.

The problem is that we implemented country code labeling as a generic label, and there were assumptions regarding what the labels looked like.  This worked very well, until recently, when we extended labeling with a bunch more label sources and formats, to exploit the general implementation that we had done.

As a result, when you do IANA based labels and RIR based labels, both at the same time, because they are both address based labels (i.e. this CIDR, this range, this specific IP address gets this label) , we use the same Patricia Tree data structure to hold the data.  No problem, actually not a bad idea.   Bur currently, when we go to read the country code from the general label structure, we expect it to be at a specific place.

However, when you provide your own labels, this convention breaks, and our methods break.  I.e., we get your bug, where your country codes look like arbitrary 2 character snippets from the general label structure.

OK, now, the fix...... I'm going to continue to overload the address Patricia Tree, as you get some great side effects doing this.  But I have to add country codes as a specific object in the generic address structures.  I should have a working fix by this afternoon, and will upload as either a refreshed 3.0.7.6 or a new 3.0.7.7.  (I haven't really announced 6, so was hoping to just reuse it ).

Thanks for all the data, and the config files.  Makes fixing bugs actually possible !!!!!

Carter


On Mar 13, 2013, at 10:49 AM, Carter Bullard <carter at qosient.com<mailto:carter at qosient.com>> wrote:



Hey Craig,
Status.....

For this problem
I've tried using the -M label="regex" option, but the regex never seems to match.

ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf-n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn't match the following events:

The -M label="regex" test is done on input, but its ralabel() that is putting in the label, so the option
given to ralabel() cannot work, even theoretically ;O).  Need to pipe the ralabel() output to
something that can do the test.   This would work:

   ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -w - |  \
          ra -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"

definitely works with the example files you provided.  Sorry I didn't see the problem immediately.
With this type of strategy, using radium() to label the records make some sense.  That way you
would just run:

          ra -S radium -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"

To get your blacklisted flows.  Still working on the country code problem.

Carter



On Mar 13, 2013, at 10:35 AM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:



Yeah, that covers it.  Or more precisely, when I use "-s +sco,+dco", one of the country code fields is incorrect and the incorrect field seems to be populated with characters from the label.

I experimented early on with the argus-flow file, but once I realized how much easier it was to manipulate my data into the iana formatted files, I stopped playing around with it.  So, I don't really have an argus-flow file to speak of.

Thanks!

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com/>]
Sent: Wednesday, March 13, 2013 7:16 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Yes I did get your Argus.zip file.  No email files showed up.

While your ralabel.conf has RALABEL_ARGUS_FLOW=no, you still refer to
an argus-flow file specification, but you didn't send that.  Can you provide?
I'll test to see if that causes an issue (directive = no, but conf provided ).

It will take me a little while to wade through this, but to keep focus on the issue,
your bug is that you are seeing incorrect country codes being reporred with -s +sco +dco ?
And you are not getting correct grep matches using the " -M label="regex" " ?

Does that cover it ?

Carter

On Mar 11, 2013, at 8:35 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:




I just posted Argus.zip to your FTP server.  It contains all of my config files, scripts called by rastream, our label file, and some flow data.  Let me know if you need anything else.

Thanks!

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com/>]
Sent: Monday, March 11, 2013 11:24 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Fractured pelvis is pretty tough.  New hip sounds traumatic, but they
get you going very early.  Not really sure how I'm able to do it, it does
seem like so early given what they did.

ftp the files to ftp://qosient.com/incoming
Its a blind directory.

Thanks!!!!!

Carter

On Mar 10, 2013, at 5:40 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:





Given the surgery you're recovering from, your continued support has been quite admirable!  My doctor won't even let me start on any kind of physical therapy for the fractured pelvis for at least another couple of weeks.  I'm going a bit stir crazy myself.

I'll experiment with your suggestions this afternoon.  I'll send you some flow records, my label file, and all of the scripts I've been using later today.  Do have any idea what the largest attachment is that your email server will accept?  I can also share that via Dropbox if you'd like.

Thanks again!

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com/>]
Sent: Sunday, March 10, 2013 9:57 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Sorry for the delayed response, therapy is a bit time consuming.
OK, well the you've described about 20 issues, so which one would you like to work on first?

So, the file format is the NIC registrar's RIR file format.  See this link for a description:
   https://www.apnic.net/publications/media-library/documents/resource-guidelines/rir-statistics-exchange-format

ZZ is used when looking up a country code results in no returned value.

Some of the values you're interested in are labels, but some are not.  Country codes are not labels, there is
a COCODE DSR that holds the source and destination 2 or three character country codes.  If the country codes
are screwed up, you may need to remove the errant data, in order to get new values added.

I'm worried that you're use of " -M dsrs='+..." could have inserted country code DSRs without initalized values.
So run this with your ralabel() command to see if you get something better:

   ralabel -M dsrs="-cocode" .............


Of course, I can't comment on your problems, until you provide the complete configurations, and argus data
that I can use to recreate the problem.  Then I can try to fix it ?


Carter

On Mar 9, 2013, at 6:14 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:






Hey, Carter...  I've been testing the labeling functionality and trying to investigate further the country code issue I posted about earlier in the week.

I'm currently using the delegated-ipv4-latest file that is included with the 3.0.7.5 client.  I've noticed that Argus isn't finding a country code match for our public IP range.  I'd like to add those ranges to the delegated-ipv4-latest, but I'm not sure what the format is.  Is there any way to make the file recognize a subnet mask?  I notice that the entry for 10.0.0.0 isn't causing the country codes for my internal networks to show up as "ZZ".    What do the numbers in the second and third to last columns represent?

delegated-afrinic-latest:afrinic|ZA|ipv4|41.73.32.0|8192|20100112|allocated

Is it possible to use the MaxMind GeoIP database for country codes instead of the iana file?  I compiled the clients with GeoIP support.  The ralabel.conf file says the GEOIP labels will be saved as scity,dcity,icity.  I've tried adding those fields to my searches and nothing shows up.  I looked through rarc.print.all.conf to see if I could find a field in there that looked related to GeoIP data, but nothing popped out at me.  I'm using the same GeoLiteCity.dat file that we use on our Splunk server.  The GeoIP part of my ralabel.conf is:

RALABEL_GEOIP_CITY="saddr,daddr,inode:cco,cname"
RALABEL_GEOIP_CITY_FILE="/usr/local/argus/GeoIPCity.dat"

I've tried using the -M label="regex" option, but the regex never seems to match.

ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn't match the following events:

1362868911.000236, e s      ,tcp,199.7.204.127,39935, ->,82.98.86.167,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868911.000282, e s      ,tcp,199.7.204.127,56477, ->,82.98.86.161,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868912.000211, e s      ,tcp,199.7.202.248,44177, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868913.000465, e s      ,tcp,199.7.204.127,42282, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted

The events above also show the issue with country codes.  I'm noticing a few things...  The erroneous country codes are always a pair of letters found somewhere in the label, but I don't see any pattern as to which letters are used.  If I disable the iana label file in ralabel.conf, country codes display normally.  If a country code is found for both a source and destination, one of the country codes is always incorrect, but the country code is properly displayed in the label field:

1362869594.000036, e        ,udp,12.130.136.11,53, ->,69.154.227.43,53,1,70,INT,pu,US,saddr=US,public,DC1,mta-external:daddr=US
1362869594.000036, e        ,udp,12.130.136.12,53, ->,200.26.226.6,53,1,68,INT,pu,AN,saddr=US,public,DC1,mta-external:daddr=AN

In the first set of events, both country codes are incorrect (dest code not shown).  In the events above, the destination code is shown correctly.  I have noticed that if, as above, the 12.130.136.12 has its sco field displayed incorrectly, it will also have its dco field displayed incorrectly when it is the destination of a flow.

If there is no label for a source or destination, the country code for that host will be empty:

1362866671.000438, e        ,udp,74.125.17.82,42492,<->,192.168.30.41,53,4,1260,CON,,dn,daddr=internal,DC4,SP1--Apache,dns,ext-DC4

I enabled label support and radium and ran the same query as above, except with ra.  The issue with the country codes is the same as when I run the searches with ralabel.  Radium has a fairly significant memory leak when labels are enabled.

Let me know if you would like me to send you any binary flow files or my iana label file.

Thanks!

Craig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130316/787ec5af/attachment.html>


More information about the argus mailing list