Question about Filtering Argus Data
Carter Bullard
carter at qosient.com
Mon Jun 16 21:30:28 EDT 2014
Oooops a type. The link is:
http://qosient.com/argus/dev/argus-latest.tar.gz
Sorry about that,
Carter
On Jun 16, 2014, at 9:01 PM, Chungen Li <jiafei427 at gmail.com> wrote:
> Hey, Carter,
>
> The Link shows not available.
>
> Do you have other links?
>
> Thanks.
>
> Best Regards.
>
> <image.png>
>
>
> On Mon, Jun 16, 2014 at 7:24 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Chungen Li,
> Try argus-3.0.8.rc.1 from here:
> http://qosient.com/argus/dev/argus-latest-tar.gz
>
> Carter
>
> On Jun 15, 2014, at 10:07 PM, Chungen Li <jiafei427 at gmail.com> wrote:
>
>> Hi, Carter,
>>
>> Finally, my problem got a hit for your hint.
>>
>> the "ra -S remoteDataSource - local src bytes gt 100" prints out the right data flow now!
>>
>> you said that I'll need to upgrade my argus data source, but my argus server side is 3.0.6.1 which shows it's latest version from argus official site.
>>
>> How do I "UPGRADE" my argus data source?
>>
>>
>> FYI,
>>
>> My machine type is 64-bit Cent OS.
>>
>> and "ra -b - src bytes gt 100" shows the result:
>>
>> (000) ldll dsr[3][12]
>> (001) jgt #0xa jt 2 jf 3
>> (002) ret #150
>> (003) ret #0
>>
>>
>>
>> Thank you so much!
>>
>>
>> On Fri, Jun 13, 2014 at 11:18 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Chungen Li,
>> Yes, that is what I meant. The direction support in the
>> clients can change the assignment of source and destination.
>> If you send a filter to the data source that says “ src metric “
>> and you change the assignment of src and dst, and then
>> apply the same filter, you may end up rejecting all the flows.
>>
>> I tested this problem quite a bit late last night, and I am
>> not seeing any issues with filtering on bytes, either from
>> a file or from realtime streaming on my systems here.
>>
>> Did you test your filter works when reading argus data files ??
>>
>> If you could use the -b option in your client, that will
>> print out the compiler pseudo code that the client uses
>> to filter. My clients generate these filters:
>>
>> thoth:~ carter$ ra -b - src bytes gt 100
>> (000) ldll dsr[3][12]
>> (001) jgt #0x64 jt 2 jf 3
>> (002) ret #150
>> (003) ret #0
>>
>> thoth:~ carter$ ra -b - not src bytes lt 100
>> (000) ldll dsr[3][12]
>> (001) jge #0x64 jt 2 jf 3
>> (002) ret #150
>> (003) ret #0
>>
>>
>> What type of machine is the source and client ??? 32-bit, 64-bit, 8-bit Atari ?? ;O)
>>
>> There are some issues with filter compatibility with legacy
>> argus data sources, like argus and radium, because the syntax
>> and semantics have evolved over the years.
>>
>> If you suspect that your argus data source is the problem,
>> use “local” in front of your filter. That will cause the
>> client to NOT send the filter to the source, but do all
>> the filtering in the client.
>>
>> ra -S remoteDataSource - local src bytes gt 100
>>
>> If this behaves differently, then you will need to upgrade your
>> argus data source.
>>
>> Hope this is helpful,
>>
>> Carter
>>
>>
>>
>>
>> On Jun 13, 2014, at 2:13 AM, Chungen Li <jiafei427 at gmail.com> wrote:
>>
>> > Since I don't know what you exactly meant by "the direction commands in your .rarc file"
>> >
>> > I commented out the all things with direction in rarc, and it turns out with nothing.
>> >
>> > Following is the part from rarc file that I changed.
>> >
>> > ------------------------------------------------------------------------------------------------------------------
>> > # Many ra* clients process flow records based on source and destination
>> > # properties. TCP and UDP ports values can be used to assign direction,
>> > # and are best used for well-known ports (< 1024), values that
>> > # are in the /etc/services defintions, and the reserved ports (> 1023, < 49151).
>> > #
>> > # The syntax is:
>> > # RA_PORT_DIRECTION="services"
>> > # RA_PORT_DIRECTION="services,wellknown"
>> > # RA_PORT_DIRECTION="services,wellknown,registered"
>> > #
>> > # We recommend the wellknown and services options, as they are a bit more
>> > # discriminating. If there are ports that you know are services that are in
>> > # the registered port range, we suggest that you add them to your /etc/services
>> > # file rather than include the registered port range; only because the
>> > # registered range is so large. However, this option is applied only to
>> > # flow in which the direction is ambiguous, and as such, corrections based
>> > # on the logic should have minimum effect on analytics.
>> >
>> >
>> > #EDIT HERE
>> > #RA_PORT_DIRECTION="services,wellknown"
>> >
>> >
>> > # Sites use locality for a number of features, such as access control,
>> > # and this support is intended to support visualization, and analytics.
>> > #
>> > # Currently, you can identify a collection of IP addresses that represent RA_LOCAL,
>> > # and are specified using an iana-address-file formatted file. (See ralabel.conf)
>> >
>> > #RA_LOCAL=/usr/local/argus/local.addrs
>> >
>> > # When locality information is available, programs like ra(), and
>> > # as the assignement of source when there is ambiguity in the
>> > # flow record as to who is the actual initiator or receiver of the flow.
>> > #
>> > # When locality information is available, programs like ra(), and
>> > # ratop() can use that information to make display decisions, such
>> > #
>> > # RA_LOCAL_DIRECTION provides the logic for using the locality
>> > # information to assign flow direction. You can force the local
>> > # address to be either the source (src) or the destination (dst).
>> > #
>> > # The syntax is:
>> > # RA_LOCAL_DIRECTION="local:src"
>> > # RA_LOCAL_DIRECTION="local:dst"
>> > #
>> >
>> > #RA_LOCAL_DIRECTION="suggest:src"
>> > #EDIT HERE
>> > #RA_LOCAL_DIRECTION="force:src"
>> > ------------------------------------------------------------------------------------------------------------------
>> >
>> >
>> >
>> >
>> > On Fri, Jun 13, 2014 at 2:06 PM, Carter Bullard <carter at qosient.com> wrote:
>> > Are you using any of the direction commands in your .rarc file ???
>> > If you are, turn them off, to see if things get better.
>> >
>> > Because you only have unidirectional data, there maybe issues with
>> > src and dst operations ???
>> >
>> > Carter
>> >
>> > On Jun 13, 2014, at 12:29 AM, Chungen Li <jiafei427 at gmail.com> wrote:
>> >
>> >> Hi, Dear Carter,
>> >>
>> >> I just downloaded the latest version from your link which is argus-client-3.0.7.34 and I think the problems are not solved.
>> >>
>> >> First, I just added sbytes and dbytes to check whether there are argus-data that fits the condition "bytes gt 100"
>> >>
>> >> Following are the result from ra:
>> >>
>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434
>> >> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State SrcBytes DstBytes
>> >> 13:10:01.726715 * udp 147.46.66.83.26283 -> 203.218.216.240.27754 2 152 INT 152 0
>> >> 13:10:01.726943 * tcp 164.125.53.178.sais -> 74.125.128.94.http 4 1275 CON 1275 0
>> >> 13:10:01.726981 * tcp 203.250.10.234.54653 -> 173.194.127.230.http 1 58 CON 58 0
>> >> 13:10:01.727066 * udp 143.248.30.28.20576 -> 223.205.207.168.16471 1 66 REQ 66 0
>> >> 13:10:01.727144 * udp 203.253.206.11.6881 -> 183.179.233.176.9043 1 368 REQ 368 0
>> >> 13:10:01.727443 * udp 210.119.13.116.52472 -> 171.6.102.51.24302 2 132 REQ 132 0
>> >> 13:10:01.727603 * tcp 164.125.53.178.infor* -> 74.125.128.94.http 5 1158 CON 1158 0
>> >> 13:10:01.727619 * tcp 147.47.241.31.63705 ?> 173.194.117.217.http 1 58 CON 58 0
>> >> 13:10:01.727730 * tcp 147.46.17.88.iee-q* -> 173.194.72.125.xmpp-* 1 59 CON 59 0
>> >> 13:10:01.727882 * tcp 147.46.6.168.58296 -> 173.194.117.193.https 2 140 RST 140 0
>> >> 13:10:01.727984 * tcp 164.125.53.178.mloadd -> 74.125.128.94.http 5 1613 CON 1613 0
>> >> 13:10:01.728311 * tcp 164.125.129.188.57351 -> 74.125.128.136.https 13 1910 CON 1910 0
>> >> 13:10:01.728810 * tcp 143.248.60.99.49702 -> 74.125.128.157.http 2 128 CON 128 0
>> >> 13:10:01.728838 * udp 203.250.1.11.51731 -> 192.221.77.23.domain 1 105 INT 105 0
>> >> 13:10:01.728919 * tcp 163.152.3.57.27812 -> 173.194.117.250.http 13 2110 FIN 2110 0
>> >>
>> >> As you can see there are many of them that SrcBytes greater than 100.
>> >>
>> >> So I ran the following command,
>> >>
>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434 - src bytes gt 100
>> >>
>> >> And it just stuck there without any result.
>> >>
>> >> So I checked the argus server side to see the log, and it came up with nothing.
>> >>
>> >> But, here's the weirdest thing.
>> >>
>> >> When I use the option "bytes lt 100" it just works like a charm.
>> >>
>> >> Here are some result from the command:
>> >>
>> >> $ ra -s +sbytes +dbytes -S 127.0.0.1:3434 - src bytes lt 100
>> >> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State SrcBytes DstBytes
>> >> 13:26:43.539575 * tcp 210.107.236.96.6901 -> 173.194.127.129.https 1 59 CON 59 0
>> >> 13:26:43.539988 * tcp 164.125.150.36.50026 -> 74.125.128.19.https 1 58 CON 58 0
>> >> 13:26:43.540055 * tcp 147.46.238.80.63363 -> 74.125.128.17.https 1 58 CON 58 0
>> >> 13:26:43.540231 * tcp 164.125.141.132.icpv2 -> 74.125.128.113.https 1 59 CON 59 0
>> >> 13:26:43.541930 * tcp 143.248.53.100.53629 -> 74.125.128.148.http 1 59 CON 59 0
>> >> 13:26:43.542199 * tcp 143.248.15.107.60584 -> 74.125.31.104.https 1 59 CON 59 0
>> >> 13:26:43.543277 * tcp 210.218.201.19.58711 -> 173.194.127.188.http 1 59 CON 59 0
>> >> 13:26:43.543386 * tcp 147.46.131.85.56737 -> 173.194.117.199.http 1 59 CON 59 0
>> >> 13:26:43.543612 * tcp 210.125.122.102.51756 ?> 173.194.126.168.https 1 58 FIN 58 0
>> >> 13:26:43.543727 * tcp 210.125.122.102.51755 ?> 173.194.126.168.https 1 58 FIN 58 0
>> >> 13:26:43.544334 * tcp 147.46.60.47.icp -> 74.125.203.94.https 1 59 CON 59 0
>> >> 13:26:43.544637 * udp 147.46.61.186.62348 -> 42.3.104.118.ansof* 1 66 REQ 66 0
>> >> 13:26:43.544638 * tcp 168.188.15.212.53359 ?> 74.125.235.150.https 1 58 RST 58 0
>> >> 13:26:43.544851 * icmp 150.183.95.135.0x0008 -> 130.126.57.173.0x0db0 1 62 ECO 62 0
>> >> 13:26:43.545332 * tcp 143.248.150.125.50392 -> 74.125.31.125.xmpp-* 1 58 CON 58 0
>> >> 13:26:43.545535 * udp 155.230.24.235.menta* -> 74.125.23.127.19302 1 66 REQ 66 0
>> >> 13:26:43.545731 * tcp 14.44.126.17.60481 ?> 74.125.31.95.https 1 58 CON 58 0
>> >> 13:26:43.547221 * tcp 203.237.41.198.56537 -> 74.125.128.113.https 1 59 CON 59 0
>> >> 13:26:43.547258 * udp 134.75.151.8.domain -> 74.125.186.215.33670 1 95 INT 95 0
>> >> 13:26:43.547381 * tcp 209.85.215.36.61758 -> 155.230.11.8.pop3 218 276543 FIN 0 276543
>> >>
>> >>
>> >> What can I do for the next?
>> >>
>> >> Thanks for your help.
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Jun 13, 2014 at 12:09 PM, Carter Bullard <carter at qosient.com> wrote:
>> >> Hey Chungen Li,
>> >> The filter “ bytes gt 10 “ , like all metrics, is decomposed to “ src bytes gt 10 or dst bytes gt 10”.
>> >> You should print the sbytes and dbytes fields to see that their values are either in or out of
>> >> the filter range.
>> >>
>> >> If your filter is not working, what version of argus-clients are you running ???
>> >> The best version to use, currently, is argus-clients-3.0.7.34, which is about
>> >> to be released as argus-clients-3.0.8. You can get the latest version here.
>> >>
>> >> http://qosient.com/argus/dev/argus-clients-latest.tar.gz
>> >>
>> >> When you connect to a remote argus data source, your client sends its filter to the remote
>> >> data server where the filtering takes place. As a result, filter failure may be at the argus data
>> >> source. argus and argus-clients are designed to detect when there are filter problems
>> >> locally or remotely, but it is still a good exercise to check the logs of the remote
>> >> argus data source, to see if it reports filter issues.
>> >>
>> >> Be sure and upgrade to the latest code, and if you are still having problems,
>> >> send more email.
>> >>
>> >>
>> >> Carter
>> >>
>> >> On Jun 12, 2014, at 10:11 PM, Chungen Li <jiafei427 at gmail.com> wrote:
>> >>
>> >>> Hello,
>> >>>
>> >>> Now I'm developing a system using the ARGUS Framework and I got trouble when I try to use the filtering option in the RA (Argus Client).
>> >>>
>> >>> Following are some results from RA.
>> >>>
>> >>> $ ../../argus/argus-clients-3.0.6.2/bin/ra -S 127.0.0.1:3434
>> >>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> >>> 11:01:56.368794 * tcp 147.46.112.192.ici -> 173.194.72.84.https 1 59 CON
>> >>> 11:01:56.369780 * s tcp 147.46.208.185.57788 -> 61.239.168.101.16057 6 440 FIN
>> >>> 11:01:56.370350 * tcp 210.98.16.36.15890 -> 173.194.127.133.https 1 59 CON
>> >>> 11:01:56.370355 * tcp 163.152.69.238.hpvro* -> 203.84.208.52.https 1 59 CON
>> >>> 11:01:56.370415 * tcp 223.195.2.196.59281 -> 173.194.127.73.http 6 593 CON
>> >>> 11:01:56.370575 * tcp 210.98.50.5.46496 -> 173.194.127.85.https 5 2511 CON
>> >>> 11:01:56.370666 * tcp 163.152.34.179.54862 -> 74.125.128.139.https 1 59 CON
>> >>> 11:01:56.370982 * tcp 147.46.188.242.49245 -> 74.125.204.188.hpvro* 1 59 CON
>> >>> 11:01:56.371008 * udp 147.46.81.23.33782 -> 219.79.58.17.osmos* 3 267 REQ
>> >>> 11:01:56.371503 * tcp 117.16.196.145.40337 -> 173.194.38.70.http 3 1619 CON
>> >>> 11:01:56.372168 * tcp 147.46.39.53.qsm-p* -> 199.59.148.139.https 4 232 CON
>> >>> 11:01:56.372620 * udp 203.241.84.210.48613 -> 216.239.32.10.domain 1 93 INT
>> >>> 11:01:56.373037 * tcp 163.152.105.227.43465 -> 74.125.203.95.http 1 70 CON
>> >>> 11:01:56.373097 * tcp 164.125.174.109.5194 -> 74.125.128.101.https 9 1312 CON
>> >>> 11:01:56.374016 * tcp 203.247.182.211.35832 -> 173.194.38.71.https 1 59 CON
>> >>>
>> >>> Here I just want the Argus record with the packet-bytes greater than 100, So I set the filtering option at the end of the command like this
>> >>>
>> >>> $ ../../argus/argus-clients-3.0.6.2/bin/ra -S 127.0.0.1:3434 - bytes gt 10
>> >>>
>> >>> But this never returns me any results, and I don't know why.
>> >>>
>> >>> Plz tell me where did I get wrong with that filtering option.
>> >>>
>> >>> Thanks.
>> >>>
>> >>>
>> >>> Chungen, Li
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards
>> >> Li ChunGen , 李 春根, 리 춘근
>> >> Department of Computer Science, POSTECH
>> >> PIRL 323 Mobile : +82-10-7522-5977
>> >> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>> >> Pohang 790-784, Republic of Korea
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards
>> > Li ChunGen , 李 春根, 리 춘근
>> > Department of Computer Science, POSTECH
>> > PIRL 323 Mobile : +82-10-7522-5977
>> > San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>> > Pohang 790-784, Republic of Korea
>>
>>
>>
>>
>> --
>> Best Regards
>> Li ChunGen , 李 春根, 리 춘근
>> Department of Computer Science, POSTECH
>> PIRL 323 Mobile : +82-10-7522-5977
>> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
>> Pohang 790-784, Republic of Korea
>
>
>
> --
> Best Regards
> Li ChunGen , 李 春根, 리 춘근
> Department of Computer Science, POSTECH
> PIRL 323 Mobile : +82-10-7522-5977
> San 31, Hyoja-dong, Nam-gu Email : jiafei427 at postech.ac.kr
> Pohang 790-784, Republic of Korea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140616/b4663dcd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140616/b4663dcd/attachment.bin>
More information about the argus
mailing list