Question about Filtering Argus Data
Carter Bullard
carter at qosient.com
Mon Jun 16 21:49:41 EDT 2014
It seems that you have a problem in your collection strategy, as this type of filter problem just
doesn’t seem to exist in any version that I’ve tested.
Can you describe your setup. Argus version, are you using radium, its version,
and what clients are you using where the filter isn’t working, and its versions.
Also, I need to know what hardware platforms you are using, compilers and
whether the machines are 16, 32 or 64 bit machines.
Carter
On Jun 16, 2014, at 9:42 PM, Chungen Li <jiafei427 at gmail.com> wrote:
> Hey, Carter.
>
> I tried the version 3.0.8.rc.1, But not helpful.
>
> So I want to know that is it matter if I use one machine for both Argus-DataServer and Argus-Client?
>
> May it cause that kind of problems?
>
> and what does that mean the "local" option in the filtering part?
>
> Thanks.
>
> Best Regards!
>
>
>
> On Tue, Jun 17, 2014 at 10:30 AM, Carter Bullard <carter at qosient.com> wrote:
> Oooops a type. The link is:
>
> http://qosient.com/argus/dev/argus-latest.tar.gz
>
> Sorry about that,
> Carter
>
> On Jun 16, 2014, at 9:01 PM, Chungen Li <jiafei427 at gmail.com> wrote:
>
>> Hey, Carter,
>>
>> The Link shows not available.
>>
>> Do you have other links?
>>
>> Thanks.
>>
>> Best Regards.
>>
>> <image.png>
>>
>>
>> On Mon, Jun 16, 2014 at 7:24 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Chungen Li,
>> Try argus-3.0.8.rc.1 from here:
>> http://qosient.com/argus/dev/argus-latest-tar.gz
>>
>> Carter
>>
>> On Jun 15, 2014, at 10:07 PM, Chungen Li <jiafei427 at gmail.com> wrote:
>>
>>> Hi, Carter,
>>>
>>> Finally, my problem got a hit for your hint.
>>>
>>> the "ra -S remoteDataSource - local src bytes gt 100" prints out the right data flow now!
>>>
>>> you said that I'll need to upgrade my argus data source, but my argus server side is 3.0.6.1 which shows it's latest version from argus official site.
>>>
>>> How do I "UPGRADE" my argus data source?
>>>
>>>
>>> FYI,
>>>
>>> My machine type is 64-bit Cent OS.
>>>
>>> and "ra -b - src bytes gt 100" shows the result:
>>>
>>> (000) ldll dsr[3][12]
>>> (001) jgt #0xa jt 2 jf 3
>>> (002) ret #150
>>> (003) ret #0
>>>
>>>
>>>
>>> Thank you so much!
>>>
>>>
>>> On Fri, Jun 13, 2014 at 11:18 PM, Carter Bullard <carter at qosient.com> wrote:
>>> Hey Chungen Li,
>>> Yes, that is what I meant. The direction support in the
>>> clients can change the assignment of source and destination.
>>> If you send a filter to the data source that says “ src metric “
>>> and you change the assignment of src and dst, and then
>>> apply the same filter, you may end up rejecting all the flows.
>>>
>>> I tested this problem quite a bit late last night, and I am
>>> not seeing any issues with filtering on bytes, either from
>>> a file or from realtime streaming on my systems here.
>>>
>>> Did you test your filter works when reading argus data files ??
>>>
>>> If you could use the -b option in your client, that will
>>> print out the compiler pseudo code that the client uses
>>> to filter. My clients generate these filters:
>>>
>>> thoth:~ carter$ ra -b - src bytes gt 100
>>> (000) ldll dsr[3][12]
>>> (001) jgt #0x64 jt 2 jf 3
>>> (002) ret #150
>>> (003) ret #0
>>>
>>> thoth:~ carter$ ra -b - not src bytes lt 100
>>> (000) ldll dsr[3][12]
>>> (001) jge #0x64 jt 2 jf 3
>>> (002) ret #150
>>> (003) ret #0
>>>
>>>
>>> What type of machine is the source and client ??? 32-bit, 64-bit, 8-bit Atari ?? ;O)
>>>
>>> There are some issues with filter compatibility with legacy
>>> argus data sources, like argus and radium, because the syntax
>>> and semantics have evolved over the years.
>>>
>>> If you suspect that your argus data source is the problem,
>>> use “local” in front of your filter. That will cause the
>>> client to NOT send the filter to the source, but do all
>>> the filtering in the client.
>>>
>>> ra -S remoteDataSource - local src bytes gt 100
>>>
>>> If this behaves differently, then you will need to upgrade your
>>> argus data source.
>>>
>>> Hope this is helpful,
>>>
>>> Carter
>>>
>>>
>>>
>>>
>>> On Jun 13, 2014, at 2:13 AM, Chungen Li <jiafei427 at gmail.com> wrote:
>>>
>>> > Since I don't know what you exactly meant by "the direction commands in your .rarc file"
>>> >
>>> > I commented out the all things with direction in rarc, and it turns out with nothing.
>>> >
>>> > Following is the part from rarc file that I changed.
>>> >
>>> > ------------------------------------------------------------------------------------------------------------------
>>> > # Many ra* clients process flow records based on source and destination
>>> > # properties. TCP and UDP ports values can be used to assign direction,
>>> > # and are best used for well-known ports (< 1024), values that
>>> > # are in the /etc/services defintions, and the reserved ports (> 1023, < 49151).
>>> > #
>>> > # The syntax is:
>>> > # RA_PORT_DIRECTION="services"
>>> > # RA_PORT_DIRECTION="services,wellknown"
>>> > # RA_PORT_DIRECTION="services,wellknown,registered"
>>> > #
>>> > # We recommend the wellknown and services options, as they are a bit more
>>> > # discriminating. If there are ports that you know are services that are in
>>> > # the registered port range, we suggest that you add them to your /etc/services
>>> > # file rather than include the registered port range; only because the
>>> > # registered range is so large. However, this option is applied only to
>>> > # flow in which the direction is ambiguous, and as such, corrections based
>>> > # on the logic should have minimum effect on analytics.
>>> >
>>> >
>>> > #EDIT HERE
>>> > #RA_PORT_DIRECTION="services,wellknown"
>>> >
>>> >
>>> > # Sites use locality for a number of features, such as access control,
>>> > # and this support is intended to support visualization, and analytics.
>>> > #
>>> > # Currently, you can identify a collection of IP addresses that represent RA_LOCAL,
>>> > # and are specified using an iana-address-file formatted file. (See ralabel.conf)
>>> >
>>> > #RA_LOCAL=/usr/local/argus/local.addrs
>>> >
>>> > # When locality information is available, programs like ra(), and
>>> > # as the assignement of source when there is ambiguity in the
>>> > # flow record as to who is the actual initiator or receiver of the flow.
>>> > #
>>> > # When locality information is available, programs like ra(), and
>>> > # ratop() can use that information to make display decisions, such
>>> > #
>>> > # RA_LOCAL_DIRECTION provides the logic for using the locality
>>> > # information to assign flow direction. You can force the local
>>> > # address to be either the source (src) or the destination (dst).
>>> > #
>>> > # The syntax is:
>>> > # RA_LOCAL_DIRECTION="local:src"
>>> > # RA_LOCAL_DIRECTION="local:dst"
>>> > #
>>> >
>>> > #RA_LOCAL_DIRECTION="suggest:src"
>>> > #EDIT HERE
>>> > #RA_LOCAL_DIRECTION="force:src"
>>> > ------------------------------------------------------------------------------------------------------------------
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Jun 13, 2014 at 2:06 PM, Carter Bullard <carter at qosient.com> wrote:
>>> > Are you using any of the direction commands in your .rarc file ???
>>> > If you are, turn them off, to see if things get better.
>>> >
>>> > Because you only have unidirectional data, there maybe issues with
>>> > src and dst operations ???
>>> >
>>> > Carter
>>> >
>>> > On Jun 13, 2014, at 12:29 AM, Chungen Li <jiafei427 at gmail.com> wrote:
>>> >
>>> >> Hi, Dear Carter,
>>> >>
>>> >> I just downloaded the latest version from your link which is argus-client-3.0.7.34 and I think the problems are not solved.
>>> >>
>>> >> First, I just added sbytes and dbytes to check whether there are argus-data that fits the condition "bytes gt 100"
>>> >>
>>> >> Following are the result from ra:
>>> >>
>>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434
>>> >> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State SrcBytes DstBytes
>>> >> 13:10:01.726715 * udp 147.46.66.83.26283 -> 203.218.216.240.27754 2 152 INT 152 0
>>> >> 13:10:01.726943 * tcp 164.125.53.178.sais -> 74.125.128.94.http 4 1275 CON 1275 0
>>> >> 13:10:01.726981 * tcp 203.250.10.234.54653 -> 173.194.127.230.http 1 58 CON 58 0
>>> >> 13:10:01.727066 * udp 143.248.30.28.20576 -> 223.205.207.168.16471 1 66 REQ 66 0
>>> >> 13:10:01.727144 * udp 203.253.206.11.6881 -> 183.179.233.176.9043 1 368 REQ 368 0
>>> >> 13:10:01.727443 * udp 210.119.13.116.52472 -> 171.6.102.51.24302 2 132 REQ 132 0
>>> >> 13:10:01.727603 * tcp 164.125.53.178.infor* -> 74.125.128.94.http 5 1158 CON 1158 0
>>> >> 13:10:01.727619 * tcp 147.47.241.31.63705 ?> 173.194.117.217.http 1 58 CON 58 0
>>> >> 13:10:01.727730 * tcp 147.46.17.88.iee-q* -> 173.194.72.125.xmpp-* 1 59 CON 59 0
>>> >> 13:10:01.727882 * tcp 147.46.6.168.58296 -> 173.194.117.193.https 2 140 RST 140 0
>>> >> 13:10:01.727984 * tcp 164.125.53.178.mloadd -> 74.125.128.94.http 5 1613 CON 1613 0
>>> >> 13:10:01.728311 * tcp 164.125.129.188.57351 -> 74.125.128.136.https 13 1910 CON 1910 0
>>> >> 13:10:01.728810 * tcp 143.248.60.99.49702 -> 74.125.128.157.http 2 128 CON 128 0
>>> >> 13:10:01.728838 * udp 203.250.1.11.51731 -> 192.221.77.23.domain 1 105 INT 105 0
>>> >> 13:10:01.728919 * tcp 163.152.3.57.27812 -> 173.194.117.250.http 13 2110 FIN 2110 0
>>> >>
>>> >> As you can see there are many of them that SrcBytes greater than 100.
>>> >>
>>> >> So I ran the following command,
>>> >>
>>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434 - src bytes gt 100
>>> >>
>>> >> And it just stuck there without any result.
>>> >>
>>> >> So I checked the argus server side to see the log, and it came up with nothing.
>>> >>
>>> >> But, here's the weirdest thing.
>>> >>
>>> >> When I use the option "bytes lt 100" it just works like a charm.
>>> >>
>>> >> Here are some result from the command:
>>> >>
>>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434 - src bytes lt 100
>>> >> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State SrcBytes DstBytes
>>> >> 13:26:43.539575 * tcp 210.107.236.96.6901 -> 173.194.127.129.https 1 59 CON 59 0
>>> >> 13:26:43.539988 * tcp 164.125.150.36.50026 -> 74.125.128.19.https 1 58 CON 58 0
>>> >> 13:26:43.540055 * tcp 147.46.238.80.63363 -> 74.125.128.17.https 1 58 CON 58 0
>>> >> 13:26:43.540231 * tcp 164.125.141.132.icpv2 -> 74.125.128.113.https 1 59 CON 59 0
>>> >> 13:26:43.541930 * tcp 143.248.53.100.53629 -> 74.125.128.148.http 1 59 CON 59 0
>>> >> 13:26:43.542199 * tcp 143.248.15.107.60584 -> 74.125.31.104.https 1 59 CON 59 0
>>> >> 13:26:43.543277 * tcp 210.218.201.19.58711 -> 173.194.127.188.http 1 59 CON 59 0
>>> >> 13:26:43.543386 * tcp 147.46.131.85.56737 -> 173.194.117.199.http 1 59 CON 59 0
>>> >> 13:26:43.543612 * tcp 210.125.122.102.51756 ?> 173.194.126.168.https 1 58 FIN 58 0
>>> >> 13:26:43.543727 * tcp 210.125.122.102.51755 ?> 173.194.126.168.https 1 58 FIN 58 0
>>> >> 13:26:43.544334 * tcp 147.46.60.47.icp -> 74.125.203.94.https 1 59 CON 59 0
>>> >> 13:26:43.544637 * udp 147.46.61.186.62348 -> 42.3.104.118.ansof* 1 66 REQ 66 0
>>> >> 13:26:43.544638 * tcp 168.188.15.212.53359 ?> 74.125.235.150.https 1 58 RST 58 0
>>> >> 13:26:43.544851 * icmp 150.183.95.135.0x0008 -> 130.126.57.173.0x0db0 1 62 ECO 62 0
>>> >> 13:26:43.545332 * tcp 143.248.150.125.50392 -> 74.125.31.125.xmpp-* 1 58 CON 58 0
>>> >> 13:26:43.545535 * udp 155.230.24.235.menta* -> 74.125.23.127.19302 1 66 REQ 66 0
>>> >> 13:26:43.545731 * tcp 14.44.126.17.60481 ?> 74.125.31.95.https 1 58 CON 58 0
>>> >> 13:26:43.547221 * tcp 203.237.41.198.56537 -> 74.125.128.113.https 1 59 CON 59 0
>>> >> 13:26:43.547258 * udp 134.75.151.8.domain -> 74.125.186.215.33670 1 95 INT 95 0
>>> >> 13:26:43.547381 * tcp 209.85.215.36.61758 -> 155.230.11.8.pop3 218 276543 FIN 0 276543
>>> >>
>>> >>
>>> >> What can I do for the next?
>>> >>
>>> >> Thanks for your help.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Jun 13, 2014 at 12:09 PM, Carter Bullard <carter at qosient.com> wrote:
>>> >> Hey Chungen Li,
>>> >> The filter “ bytes gt 10 “ , like all metrics, is decomposed to “ src bytes gt 10 or dst bytes gt 10”.
>>> >> You should print the sbytes and dbytes fields to see that their values are either in or out of
>>> >> the filter range.
>>> >>
>>> >> If your filter is not working, what version of argus-clients are you running ???
>>> >> The best version to use, currently, is argus-clients-3.0.7.34, which is about
>>> >> to be released as argus-clients-3.0.8. You can get the latest version here.
>>> >>
>>> >> http://qosient.com/argus/dev/argus-clients-latest.tar.gz
>>> >>
>>> >> When you connect to a remote argus data source, your client sends its filter to the remote
>>> >> data server where the filtering takes place. As a result, filter failure may be at the argus data
>>> >> source. argus and argus-clients are designed to detect when there are filter problems
>>> >> locally or remotely, but it is still a good exercise to check the logs of the remote
>>> >> argus data source, to see if it reports filter issues.
>>> >>
>>> >> Be sure and upgrade to the latest code, and if you are still having problems,
>>> >> send more email.
>>> >>
>>> >>
>>> >> Carter
>>> >>
>>> >> On Jun 12, 2014, at 10:11 PM, Chungen Li <jiafei427 at gmail.com> wrote:
>>> >>
>>> >>> Hello,
>>> >>>
>>> >>> Now I'm developing a system using the ARGUS Framework and I got trouble when I try to use the filtering option in the RA (Argus Client).
>>> >>>
>>> >>> Following are some results from RA.
>>> >>>
>>> >>> $ ../../argus/argus-clients-3.0.6.2/bin/ra -S 127.0.0.1:3434
>>> >>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>>> >>> 11:01:56.368794 * tcp 147.46.112.192.ici -> 173.194.72.84.https 1 59 CON
>>> >>> 11:01:56.369780 * s tcp 147.46.208.185.57788 -> 61.239.168.101.16057 6 440 FIN
>>> >>> 11:01:56.370350 * tcp 210.98.16.36.15890 -> 173.194.127.133.https 1 59 CON
>>> >>> 11:01:56.370355 * tcp 163.152.69.238.hpvro* -> 203.84.208.52.https 1 59 CON
>>> >>> 11:01:56.370415 * tcp 223.195.2.196.59281 -> 173.194.127.73.http 6 593 CON
>>> >>> 11:01:56.370575 * tcp 210.98.50.5.46496 -> 173.194.127.85.https 5 2511 CON
>>> >>> 11:01:56.370666 * tcp 163.152.34.179.54862 -> 74.125.128.139.https 1 59 CON
>>> >>> 11:01:56.370982 * tcp 147.46.188.242.49245 -> 74.125.204.188.hpvro* 1 59 CON
>>> >>> 11:01:56.371008 * udp 147.46.81.23.33782 -> 219.79.58.17.osmos* 3 267 REQ
>>> >>> 11:01:56.371503 * tcp 117.16.196.145.40337 -> 173.194.38.70.http 3 1619 CON
>>> >>> 11:01:56.372168 * tcp 147.46.39.53.qsm-p* -> 199.59.148.139.https 4 232 CON
>>> >>> 11:01:56.372620 * udp 203.241.84.210.48613 -> 216.239.32.10.domain 1 93 INT
>>> >>> 11:01:56.373037 * tcp 163.152.105.227.43465 -> 74.125.203.95.http 1 70 CON
>>> >>> 11:01:56.373097 * tcp 164.125.174.109.5194 -> 74.125.128.101.https 9 1312 CON
>>> >>> 11:01:56.374016 * tcp 203.247.182.211.35832 -> 173.194.38.71.https 1 59 CON
>>> >>>
>>> >>> Here I just want the Argus record with the packet-bytes greater than 100, So I set the filtering option at the end of the command like this
>>> >>>
>>> >>> $ ../../argus/argus-clients-3.0.6.2/bin/ra -S 127.0.0.1:3434 - bytes gt 10
>>> >>>
>>> >>> But this never returns me any results, and I don't know why.
>>> >>>
>>> >>> Plz tell me where did I get wrong with that filtering option.
>>> >>>
>>> >>> Thanks.
>>> >>>
>>> >>>
>>> >>> Chungen, Li
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards
>>> >> Li ChunGen , 李 春根, 리 춘근
>>> >> Department of Computer Science, POSTECH
>>> >> PIRL 323 Mobile : +82-10-7522-5977
>>> >> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>>> >> Pohang 790-784, Republic of Korea
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards
>>> > Li ChunGen , 李 春根, 리 춘근
>>> > Department of Computer Science, POSTECH
>>> > PIRL 323 Mobile : +82-10-7522-5977
>>> > San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>>> > Pohang 790-784, Republic of Korea
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> Li ChunGen , 李 春根, 리 춘근
>>> Department of Computer Science, POSTECH
>>> PIRL 323 Mobile : +82-10-7522-5977
>>> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>>> Pohang 790-784, Republic of Korea
>>
>>
>>
>> --
>> Best Regards
>> Li ChunGen , 李 春根, 리 춘근
>> Department of Computer Science, POSTECH
>> PIRL 323 Mobile : +82-10-7522-5977
>> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>> Pohang 790-784, Republic of Korea
>
>
>
>
> --
> Best Regards
> Li ChunGen , 李 春根, 리 춘근
> Department of Computer Science, POSTECH
> PIRL 323 Mobile : +82-10-7522-5977
> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
> Pohang 790-784, Republic of Korea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140616/c823be9f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140616/c823be9f/attachment.bin>
More information about the argus
mailing list