MaxMind GeoIP support in 3.0.8.2
David Edelman
dedelman at iname.com
Sat Sep 29 15:47:16 EDT 2018
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
It looks like we are almost there, you need to indicate which addresses need to have the labels associated
Please change this in your configuration file
RALABEL_GEOIP_CITY="saddr,daddr:city "
This information might help Looking at the code, the netmask is not functional
#
# Data for city relevant data is enabled through enabling and configuring
# the city database support. The types of data available are:
# country_code, country_code3, country_name, region, city, postal_code,
# latitude, longitude, metro_code, area_code and continent_code.
# time_offset is also available.
#
# The concept is that you should be able to add semantics for any
# IP address that is in the argus record. Support addresses are:
#
# saddr, daddr, inode
#
# The labels provided will be tagged as:
# scity, dcity, icity
#
# To configure what you want to have placed in the label, use the list of
# objects, in whatever order you like, as the RALABEL_GEOPIP_CITY string
# using these keywords:
# cco - country_code
# cco3 - country_code3
# cname - country_name
# reg - region
# city - city
# pcode - postal_code
# lat - latitude
# long - longitude
# metro - metro_code
# area - area_code
# cont - continent_code
# netmask - netmask value
#
# Working examples could be:
# RALABEL_GEOIP_CITY="saddr,daddr:lat/lon"
# RALABEL_GEOIP_CITY="*:city,region,cname,lat,lon"
#
#RALABEL_GEOIP_CITY="saddr,daddr,inode:off,cont,lat,lon"
#RALABEL_GEOIP_CITY_FILE="/usr/local/share/GeoIP/GeoIP.dat"
- --Dave
From: Kevin Branch <kevin at branchnetconsulting.com>
Sent: Saturday, September 29, 2018 11:57 AM
To: Carter Bullard <carter at qosient.com>
Cc: David Edelman <dedelman at iname.com>; Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] MaxMind GeoIP support in 3.0.8.2
Hi Carter and David,
I just tried the following with the same ralabel.conf as before:
ra -r /argus/today/unt-01.arg -w - | ralabel -f /etc/ralabel.conf -r - -s label:200 -N20
and saw this. Looks like I was using the wrong syntax which I am glad to know better now, but the GeoIP lookups are still not happening:
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
"scity=:dcity="
I know the data file is good because it works with this:
# geoiplookup -f /usr/local/share/GeoIP/GeoIPCity.dat 207.48.48.19
GeoIP City Edition, Rev 1: US, MO, Missouri, Chesterfield, 63017, 38.650002, -90.533401, 609, 314
In case it helps, I rebuilt argus-clients with the .debug flag file present and then ran this command limited to only a single argus record that I independently confirmed has a srcip value known to geoiplookup
# ra -r /argus/today/unt-01.arg -N1 -w - | ralabel -f /etc/ralabel.conf -r - -s label:50 -N200 -D3
ralabel[44570.40b753ff8e7f0000]: 15:52:57.950161 ArgusNewLabeler (0x7f8eff3f7010, 0) returning 0x1b057c0
ralabel[44570.40b753ff8e7f0000]: 15:52:57.959656 RaLabelParseResourceFile (/etc/ralabel.conf) returning 0
ralabel[44570.40b753ff8e7f0000]: 15:52:57.959730 ArgusReadConnection() read 16 bytes
ralabel[44570.40b753ff8e7f0000]: 15:52:57.959748 ArgusReadConnection() read 112 bytes
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961133 ArgusInitAddrtoname (0x7f8eff3f7010, 0xa9fe6700, 0xffffff00)
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961145 ArgusParseInit(0x7f8eff3f7010 0x7f8eff386010
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961155 ArgusReadConnection(0xff386010, 1) returning 1
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961329 ArgusAddToRecordLabel (0x7f8eff3f7010, 0x1b188c0, scity=:dcity=) returning 0
Label
scity=:dcity=
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961385 ArgusCloseInput(0xff386010) closing
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961408 ArgusCloseInput(0xff386010) done
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961415 main: ArgusReadFileStream (-) done
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961426 main: reading files completed
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961432 ArgusShutDown (0)
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961439 RaParseComplete (0) returning
ralabel[44570.40b753ff8e7f0000]: 15:52:57.961444 RaParseComplete(caught signal 0)
root at nsm.wycliffe.org:~/argus-clients-3.0.8# <mailto:root at nsm.wycliffe.org:~/argus-clients-3.0.8#> ra -r /argus/today/unt-01.arg -N1 -w - | ralabel -f /etc/ralabel.conf -r - -s label:50 -N200 -D8
ralabel[4462.40472429a37f0000]: 15:53:24.374433 ArgusFree (0x26b2250)
ralabel[4462.40472429a37f0000]: 15:53:24.374461 ArgusFree (0x26b22b0)
ralabel[4462.40472429a37f0000]: 15:53:24.374469 ArgusFree (0x26b2310)
ralabel[4462.40472429a37f0000]: 15:53:24.374474 ArgusFree (0x26b2370)
ralabel[4462.40472429a37f0000]: 15:53:24.374479 ArgusFree (0x26b23d0)
ralabel[4462.40472429a37f0000]: 15:53:24.374484 ArgusFree (0x26b2430)
ralabel[4462.40472429a37f0000]: 15:53:24.374489 ArgusFree (0x26b2490)
ralabel[4462.40472429a37f0000]: 15:53:24.374494 ArgusFree (0x26b24f0)
ralabel[4462.40472429a37f0000]: 15:53:24.374499 ArgusFree (0x26b2550)
ralabel[4462.40472429a37f0000]: 15:53:24.374504 ArgusFree (0x26b25b0)
ralabel[4462.40472429a37f0000]: 15:53:24.374509 ArgusFree (0x26b2610)
ralabel[4462.40472429a37f0000]: 15:53:24.374533 ArgusCalloc (1, 80) returning 0x26b2610
ralabel[4462.40472429a37f0000]: 15:53:24.374544 ArgusCalloc (1, 296) returning 0x26b27c0
ralabel[4462.40472429a37f0000]: 15:53:24.374566 ArgusCalloc (1, 112) returning 0x26b2950
ralabel[4462.40472429a37f0000]: 15:53:24.374577 ArgusCalloc (1, 80) returning 0x26b25b0
ralabel[4462.40472429a37f0000]: 15:53:24.374595 ArgusNewQueue () returning 0x26b25b0
ralabel[4462.40472429a37f0000]: 15:53:24.374613 ArgusCalloc (65536, 8) returning 0x2849a010
ralabel[4462.40472429a37f0000]: 15:53:24.374619 ArgusNewLabeler (0x7fa329100010, 0) returning 0x26b27c0
ralabel[4462.40472429a37f0000]: 15:53:24.384352 RaLabelParseResourceFile (/etc/ralabel.conf) returning 0
ralabel[4462.40472429a37f0000]: 15:53:24.384387 ArgusCalloc (1, 560) returning 0x26b2250
ralabel[4462.40472429a37f0000]: 15:53:24.384411 ArgusCalloc (1, 112) returning 0x26b2e80
ralabel[4462.40472429a37f0000]: 15:53:24.384418 ArgusCalloc (1, 80) returning 0x26b2f00
ralabel[4462.40472429a37f0000]: 15:53:24.384424 ArgusNewQueue () returning 0x26b2f00
ralabel[4462.40472429a37f0000]: 15:53:24.384430 ArgusCalloc (1, 56) returning 0x26b2f60
ralabel[4462.40472429a37f0000]: 15:53:24.384454 ArgusCalloc (65536, 8) returning 0x27358010
ralabel[4462.40472429a37f0000]: 15:53:24.384461 ArgusNewHashTable (65536) returning 0x26b2f60
ralabel[4462.40472429a37f0000]: 15:53:24.384483 ArgusReadConnection() read 16 bytes
ralabel[4462.40472429a37f0000]: 15:53:24.384498 ArgusReadConnection() read 112 bytes
ralabel[4462.40472429a37f0000]: 15:53:24.384514 ArgusCalloc (1, 4194304) returning 0x26f57010
ralabel[4462.40472429a37f0000]: 15:53:24.384523 ArgusCalloc (1, 262144) returning 0x26f16010
ralabel[4462.40472429a37f0000]: 15:53:24.385934 ArgusInitAddrtoname (0x7fa329100010, 0xa9fe6700, 0xffffff00)
ralabel[4462.40472429a37f0000]: 15:53:24.385947 ArgusParseInit(0x7fa329100010 0x7fa32908f010
ralabel[4462.40472429a37f0000]: 15:53:24.385956 ArgusReadConnection(0x2908f010, 1) returning 1
ralabel[4462.40472429a37f0000]: 15:53:24.385969 ArgusReadFileStream() starting
ralabel[4462.40472429a37f0000]: 15:53:24.385981 ArgusReadStreamSocket (0x7fa32908f010) read 380 bytes
ralabel[4462.40472429a37f0000]: 15:53:24.386015 ArgusCalloc (1, 384) returning 0x26c58c0
ralabel[4462.40472429a37f0000]: 15:53:24.386025 ArgusCalloc (1, 12) returning 0x26c5a50
ralabel[4462.40472429a37f0000]: 15:53:24.386031 ArgusCalloc (1, 80) returning 0x26c5a70
ralabel[4462.40472429a37f0000]: 15:53:24.386037 ArgusCalloc (1, 36) returning 0x26c5ad0
ralabel[4462.40472429a37f0000]: 15:53:24.386042 ArgusCalloc (1, 52) returning 0x26c5b00
ralabel[4462.40472429a37f0000]: 15:53:24.386047 ArgusCalloc (1, 80) returning 0x26c5b40
ralabel[4462.40472429a37f0000]: 15:53:24.386053 ArgusCalloc (1, 120) returning 0x26c5ba0
ralabel[4462.40472429a37f0000]: 15:53:24.386058 ArgusCalloc (1, 8) returning 0x26c5c20
ralabel[4462.40472429a37f0000]: 15:53:24.386063 ArgusCalloc (1, 164) returning 0x26c5c40
ralabel[4462.40472429a37f0000]: 15:53:24.386070 ArgusCalloc (1, 20) returning 0x26c5cf0
ralabel[4462.40472429a37f0000]: 15:53:24.386075 ArgusCalloc (1, 20) returning 0x26c5d10
ralabel[4462.40472429a37f0000]: 15:53:24.386080 ArgusCalloc (1, 12) returning 0x26c5d30
ralabel[4462.40472429a37f0000]: 15:53:24.386092 ArgusCalloc (1, 16) returning 0x26c5d50
ralabel[4462.40472429a37f0000]: 15:53:24.386131 ArgusCalloc (1, 12) returning 0x26c5e80
ralabel[4462.40472429a37f0000]: 15:53:24.386234 ArgusAddToRecordLabel (0x7fa329100010, 0x26c58c0, scity=:dcity=) returning 0
Label
scity=:dcity=
ralabel[4462.40472429a37f0000]: 15:53:24.386280 ArgusFree (0x26c5a50)
ralabel[4462.40472429a37f0000]: 15:53:24.386287 ArgusFree (0x26c5a70)
ralabel[4462.40472429a37f0000]: 15:53:24.386292 ArgusFree (0x26c5ad0)
ralabel[4462.40472429a37f0000]: 15:53:24.386297 ArgusFree (0x26c5b00)
ralabel[4462.40472429a37f0000]: 15:53:24.386301 ArgusFree (0x26c5b40)
ralabel[4462.40472429a37f0000]: 15:53:24.386304 ArgusFree (0x26c5ba0)
ralabel[4462.40472429a37f0000]: 15:53:24.386308 ArgusFree (0x26c5c20)
ralabel[4462.40472429a37f0000]: 15:53:24.386312 ArgusFree (0x26c5c40)
ralabel[4462.40472429a37f0000]: 15:53:24.386316 ArgusFree (0x26c5cf0)
ralabel[4462.40472429a37f0000]: 15:53:24.386322 ArgusFree (0x26c5d10)
ralabel[4462.40472429a37f0000]: 15:53:24.386359 ArgusFree (0x26c5d30)
ralabel[4462.40472429a37f0000]: 15:53:24.386369 ArgusFree (0x26c5e80)
ralabel[4462.40472429a37f0000]: 15:53:24.386377 ArgusFree (0x26c5d50)
ralabel[4462.40472429a37f0000]: 15:53:24.386384 ArgusFree (0x26c58c0)
ralabel[4462.40472429a37f0000]: 15:53:24.386392 RaProcessRecord (0x2908f630) returning
ralabel[4462.40472429a37f0000]: 15:53:24.386400 RaScheduleRecord (0x7fa329100010, 0x7fa32908f630) scheduled
ralabel[4462.40472429a37f0000]: 15:53:24.386408 ArgusHandleRecord (0x7fa326f57010, 0x7fa329221800) returning 380
ralabel[4462.40472429a37f0000]: 15:53:24.386417 ArgusReadStreamSocket (0x7fa32908f010) returning 0
ralabel[4462.40472429a37f0000]: 15:53:24.386432 ArgusReadStreamSocket (0x7fa32908f010) read 0 bytes
ralabel[4462.40472429a37f0000]: 15:53:24.386439 ArgusReadStreamSocket (0x7fa32908f010) returning 1
ralabel[4462.40472429a37f0000]: 15:53:24.386447 ArgusCloseInput(0x2908f010) closing
ralabel[4462.40472429a37f0000]: 15:53:24.386463 ArgusFree (0x7fa326f57010)
ralabel[4462.40472429a37f0000]: 15:53:24.386473 ArgusFree (0x7fa326f16010)
ralabel[4462.40472429a37f0000]: 15:53:24.386492 ArgusCloseInput(0x2908f010) done
ralabel[4462.40472429a37f0000]: 15:53:24.386500 ArgusReadFileStream() returning
ralabel[4462.40472429a37f0000]: 15:53:24.386508 main: ArgusReadFileStream (-) done
ralabel[4462.40472429a37f0000]: 15:53:24.386522 ArgusFree (0x7fa32908f010)
ralabel[4462.40472429a37f0000]: 15:53:24.386529 main: reading files completed
ralabel[4462.40472429a37f0000]: 15:53:24.386536 ArgusShutDown (0)
ralabel[4462.40472429a37f0000]: 15:53:24.386546 ArgusFree (0x26b2190)
ralabel[4462.40472429a37f0000]: 15:53:24.386553 ArgusDeleteQueue (0x26b2190) returning
ralabel[4462.40472429a37f0000]: 15:53:24.386561 ArgusFree (0x26b21f0)
ralabel[4462.40472429a37f0000]: 15:53:24.386569 ArgusDeleteQueue (0x26b21f0) returning
ralabel[4462.40472429a37f0000]: 15:53:24.386599 RaParseComplete (0) returning
ralabel[4462.40472429a37f0000]: 15:53:24.386608 RaParseComplete(caught signal 0)
Thanks for looking at this,
Kevin
On Sat, Sep 29, 2018 at 11:18 AM Carter Bullard <carter at qosient.com <mailto:carter at qosient.com> > wrote:
Hey Kevin,
Any success ???
Carter
<http://qosient.com/>
<http://qosient.com/>
Carter Bullard • CTO <http://qosient.com/>
150 E 57th Street Suite 12D <http://qosient.com/>
New York, New York 10022-2795 <http://qosient.com/>
Phone +1.212.588.9133 • Mobile +1.917.497.9494 <http://qosient.com/>
On Sep 28, 2018, at 9:40 PM, David Edelman <dedelman at iname.com> wrote: <http://qosient.com/>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1 <http://qosient.com/>
Please try this: <http://qosient.com/>
# ra -r /argus/today/unt-01.arg -w - | ralabel -f /etc/ralabel.conf -r - -s label:200 -N20 <http://qosient.com/>
<http://qosient.com/>
From: Argus-info <argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu> On Behalf Of Kevin Branch
Sent: Friday, September 28, 2018 5:25 PM
To: Carter Bullard <carter at qosient.com>
Cc: Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] MaxMind GeoIP support in 3.0.8.2 <http://qosient.com/>
<http://qosient.com/>
Looks like GeoIP is linked in already: <http://qosient.com/>
<http://qosient.com/>
# ldd `which ralabel`
linux-vdso.so.1 => (0x00007ffddaf8a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5fe67c9000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5fe65ab000)
libGeoIP.so.1 => /usr/lib/x86_64-linux-gnu/libGeoIP.so.1 (0x00007f5fe637c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fe5fb3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5fe6acf000) <http://qosient.com/>
<http://qosient.com/>
I switched to just "lat,lon" like you suggested in ralabel.conf: <http://qosient.com/>
<http://qosient.com/>
# cat /etc/ralabel.conf <http://qosient.com/>
RALABEL_ARIN_COUNTRY_CODES=no <http://qosient.com/>
<http://qosient.com/>
RALABEL_GEOIP_ASN=yes <http://qosient.com/>
RALABEL_GEOIP_ASN_FILE="/usr/local/share/GeoIP/GeoIPASNum.dat" <http://qosient.com/>
<http://qosient.com/>
RALABEL_GEOIP_CITY="lat,lon" <http://qosient.com/>
RALABEL_GEOIP_CITY_FILE="/usr/local/share/GeoIP/GeoIPCity.dat" <http://qosient.com/>
<http://qosient.com/>
But I don't see anything but ASNs getting added in: <http://qosient.com/>
<http://qosient.com/>
# ra -r /argus/today/unt-01.arg -w - | ralabel -f /etc/ralabel.conf -r - -s sas,das,scity,dcity,icity | head -n20 <http://qosient.com/>
sAS dAS <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
22927 4323 <http://qosient.com/>
6582 4323 <http://qosient.com/>
7018 4323 <http://qosient.com/>
4323 3 <http://qosient.com/>
4323 <http://qosient.com/>
4323 15169 <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
4323 15169 <http://qosient.com/>
4323 21928 <http://qosient.com/>
2828 4323 <http://qosient.com/>
<http://qosient.com/>
Thoughts? <http://qosient.com/>
Kevin <http://qosient.com/>
<http://qosient.com/>
On Fri, Sep 28, 2018 at 2:36 PM Carter Bullard <carter at qosient.com <mailto:carter at qosient.com> > wrote: <http://qosient.com/>
Hey Kevin, <http://qosient.com/>
If you do a ./configure —help you’ll see all the options that ./configure supports. If you specify —with-GeoIP it wants the directory where the GeoIP library is. <http://qosient.com/>
—with-GeoIP=DIR. compile with GeoIP in <dir> <http://qosient.com/>
<http://qosient.com/>
You can check if the library is bound to the client binaries with ldd … <http://qosient.com/>
ldd `which ralabel` <http://qosient.com/>
<http://qosient.com/>
If you’re already bound, the city stuff should work …. I’m not familiar with “reg,cco” as GeoIP City objects off the top of my head. I’m familiar with “off,cont,lat,lon,region,city,cname”. Maybe try lat,lon as a test, as I use that all the time .. <http://qosient.com/>
<http://qosient.com/>
Carter <http://qosient.com/>
<http://qosient.com/>
<http://qosient.com/>
On Sep 28, 2018, at 1:59 PM, Kevin Branch <kevin at branchnetconsulting.com <mailto:kevin at branchnetconsulting.com> > wrote: <http://qosient.com/>
<http://qosient.com/>
Hi Carter, <http://qosient.com/>
<http://qosient.com/>
I was just trying today to make ralabel on argus 3.0.8.2 do GeoIP labeling with GeoIPCity.dat, but even though the file referred to in ralabel.conf for that purpose exists, no GeoIP labeling takes place. ASN lookups work great, but ralabel never adds any scity or dcity fields. <http://qosient.com/>
<http://qosient.com/>
My ralabel.conf: <http://qosient.com/>
<http://qosient.com/>
RALABEL_ARIN_COUNTRY_CODES=no <http://qosient.com/>
<http://qosient.com/>
RALABEL_GEOIP_ASN=yes <http://qosient.com/>
RALABEL_GEOIP_ASN_FILE="/usr/local/share/GeoIP/GeoIPASNum.dat" <http://qosient.com/>
<http://qosient.com/>
RALABEL_GEOIP_CITY="reg,cco" <http://qosient.com/>
RALABEL_GEOIP_CITY_FILE="/usr/local/share/GeoIP/GeoIPCity.dat" <http://qosient.com/>
<http://qosient.com/>
No errors are thrown by ralabel. I thought maybe I needed to specially compile in GeoIP support like I read about here: https://qosient.com/argus/geolocation.shtml <http://qosient.com/>
but when I run "./configure --with-GeoIP=yes" it throws this error: <http://qosient.com/>
configure: WARNING: unrecognized options: --with-GeoIP <http://qosient.com/>
However, I presume the fact that ASN lookups work means that MaxMind GeoIP support libraries are already installed, and presumably installed by default in the latest argus now. <http://qosient.com/>
<http://qosient.com/>
# ra -r /argus/today/unt-01.arg -w - | ralabel -f /etc/ralabel.conf -r - -s sas,das,scity,dcity,icity | head -n20 <http://qosient.com/>
sAS dAS <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
22927 4323 <http://qosient.com/>
6582 4323 <http://qosient.com/>
7018 4323 <http://qosient.com/>
4323 3 <http://qosient.com/>
4323 <http://qosient.com/>
4323 15169 <http://qosient.com/>
4323 <http://qosient.com/>
4323 <http://qosient.com/>
4323 15169 <http://qosient.com/>
4323 21928 <http://qosient.com/>
2828 4323 <http://qosient.com/>
<http://qosient.com/>
Any thoughts on what I am missing or how I might further debug this issue? <http://qosient.com/>
<http://qosient.com/>
Thanks! <http://qosient.com/>
Kevin <http://qosient.com/>
-----BEGIN PGP SIGNATURE----- <http://qosient.com/>
iF0EARECAB0WIQQP+UHquEepll566aqXCCyZOY1FIQUCW67X6gAKCRCXCCyZOY1F
IdLwAKDv6tPpe7OwLBEEfM1YTnAD8MxZCgCeOlGEEuEIUvqlyIMh8CPBWctmBd0=
=p/tg
-----END PGP SIGNATURE----- <http://qosient.com/>
-----BEGIN PGP SIGNATURE-----
iF0EARECAB0WIQQP+UHquEepll566aqXCCyZOY1FIQUCW6/WuAAKCRCXCCyZOY1F
IYGaAKDk/9QFqcgbEWN6bN9eMNitGAXU7ACfduIQKuGR7lsDTCkdRogye0u8TxM=
=eoBH
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20180929/efcb2153/attachment.html>
More information about the argus
mailing list