Unusual label behavior

[Carter Bullard]

As I asked before, what is the flow of data here... is it 
   argus -> radium(labeler) -> rastream() -> racluster() -> Splunk ?

Where in this chain does the label get screwed up?

A few things that are not quite right.  Your " -s printFields..." option for racluster()
appears really complex, overly complex.  The "+" says to add these fields to an
existing field definition.  And with the first being "+0...." you're actually changing
the order of that unspecified printing list with this appending list.  Opportunity for
serious confusion.  Why not specify the output fields in a rarc formatted file
specific for the rastream script?  Not a huge problem but there are better ways
of doing this.

Does the " -d " option for rastream() not work for you ?  Works for me.

Your labeling the country codes twice... in the RALABEL_ARIN_COUNTRY_CODES
directive and in the RALABEL_GEOIP_CITY directive.  Not sure what that may do,
but not necessarily the problems you're reporting.

Carter 

On Jul 23, 2013, at 10:30 PM, Craig Merchant <cmerchant at responsys.com> wrote:

> Ralabel.conf:
>  
> RALABEL_IANA_ADDRESS=yes
> RALABEL_IANA_ADDRESS_FILE="/usr/local/argus/responsys-iana-file"
> #RALABEL_IANA_ADDRESS_FILE="/usr/local/argus/iana-address-file.full"
> RALABEL_ARIN_COUNTRY_CODES=yes
> #RALABEL_BIND_NAME="all"
> RALABEL_IANA_PORT=no
> #RALABEL_IANA_PORT_FILE="/usr/local/argus/iana-port-numbers"
> RALABEL_ARGUS_FLOW=no
> RALABEL_ARGUS_FLOW_FILE="/usr/local/argus/label-file"
> #RALABEL_GEOIP_ASN=yes
> #RALABEL_GEOIP_ASN_FILE="/usr/local/share/GeoIP/GeoIPASNum.dat"
> #       RALABEL_GEOIP_CITY="saddr,daddr:lat/lon"
> #       RALABEL_GEOIP_CITY="*:city,region,cname,lat,lon"
> RALABEL_GEOIP_CITY="saddr,daddr,inode:cco,cname"
> RALABEL_GEOIP_CITY_FILE="/usr/local/argus/GeoIPCity.dat"
>  
> I grab my data with rastream and  then use the scripting feature to do some post-processing:
>  
> /usr/local/bin/rastream -S 10.10.10.10:561 -M time 5m -B 10s -w /ssd/argus/%s.argus -f /usr/local/argus/rastream.sh &
>  
> Rastream.sh looks like:
>  
> racluster -r $FILES -M correct -m proto saddr daddr dport -c "@" -p 3 -u -Z b -n –s "+0ltime,+trans,+dur,+runtime,+mean,+stddev,+sum,+sco,+dco,+spkts                 ,+dpkts,+sbytes,+dbytes,+load,+sload,+dload,+loss,+sloss,+dloss,+ploss,+sploss,+dploss,+rate,+srate,+drate,+appbytes,+sappbytes,+dappbytes,+label:200,+abr" - src net 10.0.0.0/8 and dst net 10.0.0.0/8 | sed -e 's/,/#/g' | sed -e 's/@/,/g' > /ssd/argus/splunk/internal.csv
>  
> I found that variable length label fields that were comma separated confused Splunk when I imported the CSV file.  So, I run it through sed to change the label field’s commas to # and then change the @ delimiter back to a comma.
>  
> Thx.
>  
> Craig
>  
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Tuesday, July 23, 2013 7:17 PM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] Unusual label behavior
>  
> Labels are strings that are assembled in big global buffers, and we need to work on concurrent processing, especially when the label configurations have "cont", where multiple appended strings are applied to the same flow, which can push us  to overflow, which can cause string collision, which looks to be the problem your reporting.
>  
> So need to understand how the processing goes.  argus -> radium(labeler) -> ra ?
>  
> So need to see your ralabel.conf file to see what could be wrong.
> 
> Carter
> 
> On Jul 23, 2013, at 5:24 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> 
> Hey, Carter…
>  
> Since we upgraded to the latest version, I’m getting some really strange behavior with labels.  Some events have massive incorrect labels.  Other events that should have labels don’t.
>  
> For example, one event had the following in the label field:
>  
> saddr=internal,DC2,I5-PSO,app,dev-tserv2,app,dev-tserv2,app,qa-tserv2,app,qa-tserv2,app,ri5-tserv2,app,ri5-tserv2,app,ri5-tserv2,app,ri5-tserv2,app,ri5-tserv2,app,ri5-tserv2,app,sp1-tserv2,app,stage3-tserv2,app,stage3-tserv2,etldev,tserv2,Cpn01:tserv2=etlserv,tserv2,etlserv,tserv2,sa,tserv2,sa,tserv2,sa,tserv2,sa,tserv2,sa,tserv2,database,database,database,database,database,database,database,database,DME,DC2,ETL,Dev,DME,DC2,ETL,Dev,DME,DB:daddr=internal,DC2,SysOps-Admin,bi-etl,dc2,dbadmin,dc2,dbmonitor,dc2,dns-master,dc2,etltest,dc2,fly,uber,fly,uber,ldap-master,dc2,ldap-master,dc2,mailscan,uber,mon,dc2,mondb,dc2,mta,uber,mta,uber,prodops:dc2=prodopsdb,dc2,rac,01n01-dc2,rac,01n02-dc2,rac,02n01-dc2,rac,02n02-dc2,sa,dc2,sa,dc2,sa,dc2,zabbix,uber,zabbixdb,uber,database,database,database,database,database,database"
>  
> The correct label should have been:
>  
> saddr=internal,DC2,I5-PSO,app,ri5-tserv2:daddr=internal,DC2,SysOps-Admin,fly,uber
>  
> I checked through the label file for some of the hosts that were missing labels and there were definitely records that should have had labels that didn’t.
>  
> If you still have that tcpdump I sent you, I can forward you the latest version of the label file and see if you can replicate the results.
>  
> Thanks.
>  
> Craig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130724/a19a7d64/attachment.bin>