A couple of labeling questions/issues...

Wed Mar 13 10:35:24 EDT 2013

Yeah, that covers it.  Or more precisely, when I use "-s +sco,+dco", one of the country code fields is incorrect and the incorrect field seems to be populated with characters from the label.

I experimented early on with the argus-flow file, but once I realized how much easier it was to manipulate my data into the iana formatted files, I stopped playing around with it.  So, I don't really have an argus-flow file to speak of.

Thanks!

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Wednesday, March 13, 2013 7:16 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Yes I did get your Argus.zip file.  No email files showed up.

While your ralabel.conf has RALABEL_ARGUS_FLOW=no, you still refer to
an argus-flow file specification, but you didn't send that.  Can you provide?
I'll test to see if that causes an issue (directive = no, but conf provided ).

It will take me a little while to wade through this, but to keep focus on the issue,
your bug is that you are seeing incorrect country codes being reporred with -s +sco +dco ?
And you are not getting correct grep matches using the " -M label="regex" " ?

Does that cover it ?

Carter

On Mar 11, 2013, at 8:35 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

I just posted Argus.zip to your FTP server.  It contains all of my config files, scripts called by rastream, our label file, and some flow data.  Let me know if you need anything else.

Thanks!

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com>]
Sent: Monday, March 11, 2013 11:24 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Fractured pelvis is pretty tough.  New hip sounds traumatic, but they
get you going very early.  Not really sure how I'm able to do it, it does
seem like so early given what they did.

ftp the files to ftp://qosient.com/incoming
Its a blind directory.

Thanks!!!!!

Carter

On Mar 10, 2013, at 5:40 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

Given the surgery you're recovering from, your continued support has been quite admirable!  My doctor won't even let me start on any kind of physical therapy for the fractured pelvis for at least another couple of weeks.  I'm going a bit stir crazy myself.

I'll experiment with your suggestions this afternoon.  I'll send you some flow records, my label file, and all of the scripts I've been using later today.  Do have any idea what the largest attachment is that your email server will accept?  I can also share that via Dropbox if you'd like.

Thanks again!

Craig

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com/>]
Sent: Sunday, March 10, 2013 9:57 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] A couple of labeling questions/issues...

Hey Craig,
Sorry for the delayed response, therapy is a bit time consuming.
OK, well the you've described about 20 issues, so which one would you like to work on first?

So, the file format is the NIC registrar's RIR file format.  See this link for a description:
   https://www.apnic.net/publications/media-library/documents/resource-guidelines/rir-statistics-exchange-format

ZZ is used when looking up a country code results in no returned value.

Some of the values you're interested in are labels, but some are not.  Country codes are not labels, there is
a COCODE DSR that holds the source and destination 2 or three character country codes.  If the country codes
are screwed up, you may need to remove the errant data, in order to get new values added.

I'm worried that you're use of " -M dsrs='+..." could have inserted country code DSRs without initalized values.
So run this with your ralabel() command to see if you get something better:

   ralabel -M dsrs="-cocode" .............

Of course, I can't comment on your problems, until you provide the complete configurations, and argus data
that I can use to recreate the problem.  Then I can try to fix it ?

Carter

On Mar 9, 2013, at 6:14 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

Hey, Carter...  I've been testing the labeling functionality and trying to investigate further the country code issue I posted about earlier in the week.

I'm currently using the delegated-ipv4-latest file that is included with the 3.0.7.5 client.  I've noticed that Argus isn't finding a country code match for our public IP range.  I'd like to add those ranges to the delegated-ipv4-latest, but I'm not sure what the format is.  Is there any way to make the file recognize a subnet mask?  I notice that the entry for 10.0.0.0 isn't causing the country codes for my internal networks to show up as "ZZ".    What do the numbers in the second and third to last columns represent?

delegated-afrinic-latest:afrinic|ZA|ipv4|41.73.32.0|8192|20100112|allocated

Is it possible to use the MaxMind GeoIP database for country codes instead of the iana file?  I compiled the clients with GeoIP support.  The ralabel.conf file says the GEOIP labels will be saved as scity,dcity,icity.  I've tried adding those fields to my searches and nothing shows up.  I looked through rarc.print.all.conf to see if I could find a field in there that looked related to GeoIP data, but nothing popped out at me.  I'm using the same GeoLiteCity.dat file that we use on our Splunk server.  The GeoIP part of my ralabel.conf is:

RALABEL_GEOIP_CITY="saddr,daddr,inode:cco,cname"
RALABEL_GEOIP_CITY_FILE="/usr/local/argus/GeoIPCity.dat"

I've tried using the -M label="regex" option, but the regex never seems to match.

ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn't match the following events:

1362868911.000236, e s      ,tcp,199.7.204.127,39935, ->,82.98.86.167,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868911.000282, e s      ,tcp,199.7.204.127,56477, ->,82.98.86.161,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868912.000211, e s      ,tcp,199.7.202.248,44177, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
1362868913.000465, e s      ,tcp,199.7.204.127,42282, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted

The events above also show the issue with country codes.  I'm noticing a few things...  The erroneous country codes are always a pair of letters found somewhere in the label, but I don't see any pattern as to which letters are used.  If I disable the iana label file in ralabel.conf, country codes display normally.  If a country code is found for both a source and destination, one of the country codes is always incorrect, but the country code is properly displayed in the label field:

1362869594.000036, e        ,udp,12.130.136.11,53, ->,69.154.227.43,53,1,70,INT,pu,US,saddr=US,public,DC1,mta-external:daddr=US
1362869594.000036, e        ,udp,12.130.136.12,53, ->,200.26.226.6,53,1,68,INT,pu,AN,saddr=US,public,DC1,mta-external:daddr=AN

In the first set of events, both country codes are incorrect (dest code not shown).  In the events above, the destination code is shown correctly.  I have noticed that if, as above, the 12.130.136.12 has its sco field displayed incorrectly, it will also have its dco field displayed incorrectly when it is the destination of a flow.

If there is no label for a source or destination, the country code for that host will be empty:

1362866671.000438, e        ,udp,74.125.17.82,42492,<->,192.168.30.41,53,4,1260,CON,,dn,daddr=internal,DC4,SP1--Apache,dns,ext-DC4

I enabled label support and radium and ran the same query as above, except with ra.  The issue with the country codes is the same as when I run the searches with ralabel.  Radium has a fairly significant memory leak when labels are enabled.

Let me know if you would like me to send you any binary flow files or my iana label file.

Thanks!

Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130313/460c5448/attachment.html>