Need some help writing "simple" query
Carter Bullard
carter at qosient.com
Tue Oct 2 07:58:43 EDT 2012
Hey Slacker T,
You need to specify the fields that you want printed using either your
~/.rarc file or the command line. You are printing out the default fields here.
Copy the ./support/Config/rarc to your home directory as ~/.rarc. This will help.
Try this:
rasort -r addrs.file -m sbytes -s stime dur flgs trans saddr:15 sbytes dbytes sappbytes dappbytes
This will sort based on total bytes transmitted (because of the original racluster() ),
and will print the ' starting time ', the ' dur 'ation, some flags, the number of records use to generate
the aggregate data, your IP address of interest, and then some metrics, like bytes transmitted
and received ( sbytes, dbytes ), and the bytes loaded ( sappbytes, dappbytes ).
Argus generates a lot of metrics, but some are not collected by default. Your appbytes
metrics maybe zero, if you have not configured argus to generate them.
Use the /etc/argus.conf file to turn those and other metrics on.
Printing out the "daddr" will provide useful data, but you need to know how to interpret it.
In your ~/.rarc file, turn on printing CIDR addresses.
RA_CIDR_ADDRESS_FORMAT="yes"
and you'll see that the addresses in the resulting daddr field are really aggregates. When
you see an address like 10.208.0.0/16, or 10.0.0.0/8, that will indicate that your ' saddr '
talked to a lot of destinations, and the set belong in these CIDR addresses.
When the address is a complete /32 address, then your ' saddr ' only talked to 1 other
address.
The daddr field has data in it, because you are using the default aggregation mode:
RACLUSTER_PRESERVE_FIELDS=yes
If you set this to no in a racluster.conf file, all of those daddr values would be 0.0.0.0.
Hope this works for you,
Carter
On Oct 2, 2012, at 1:06 AM, Slacker T <slackert at gmail.com> wrote:
> Carter,
>
> Thanks for your reply!
>
> Ok, so I finally got around to creating some packages for 3.0.6 and
> upgrading. All is good there. In the examples below 11.140.20.185 was
> downloading several large isos from 149.20.4.71 over http for about a
> 20 minute window. I used the following cluster command
>
> racluster -r argus.23.1* argus.23.2* -M rmon -m saddr -w addrs.file - ip
>
> Of the other commands you gave to try, only 'sappbytes' and 'sappbytes
> dappbytes' seem to give me info needed as they show the source and
> destination ip involved. Other commands show 0.0.0.0 or at least the
> last few octets as 0.0.
>
> # rasort -r addrs.file -m sappbytes -No4
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 23:12:55.523633 M d tcp 149.20.4.71.http ->
> 11.140.20.185.* 6152105 6447411233 FIN
> 23:10:00.000000 M F ip 11.140.0.147 <->
> 0.0.0.0 199132 139434535 CON
> 23:10:00.000000 M d tcp 11.142.14.190.https ->
> 11.140.0.215.* 51234 53191694 FIN
> 23:10:00.000000 MU F ip 12.15.205.13 <->
> 0.0.0.0 237873 63545090 CON
>
> # rasort -r addrs.file -m dappbytes -No4
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 23:10:00.311628 MU ip 11.140.20.185 <->
> 0.0.0.0 6157256 6447914791 CON
> 23:10:00.000000 M ip 11.140.0.215 <->
> 10.208.0.0 61391 62223854 CON
> 23:10:00.053295 M ip 13.16.180.200 <->
> 10.0.0.0 285917 69964870 CON
> 23:10:00.064677 M ip 13.16.180.210 <->
> 10.0.0.0 285845 69900877 CON
>
> # rasort -r addrs.file -m sappbytes dappbytes -No4
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 23:12:55.523633 M d tcp 149.20.4.71.http ->
> 11.140.20.185.* 6152105 6447411233 FIN
> 23:10:00.000000 M F ip 11.140.0.147 <->
> 0.0.0.0 199132 139434535 CON
> 23:10:00.000000 M d tcp 11.142.14.190.https ->
> 11.140.0.215.* 51234 53191694 FIN
> 23:10:00.000000 MU F ip 12.15.205.13 <->
> 0.0.0.0 237873 63545090 CON
>
> # rasort -r addrs.file -m appbytes -No4
> StartTime Flgs Proto SrcAddr Sport Dir
> DstAddr Dport TotPkts TotBytes State
> 23:10:00.311628 MU ip 11.140.20.185 <->
> 0.0.0.0 6157256 6447914791 CON
> 23:12:55.523633 M d tcp 149.20.4.71.http ->
> 11.140.20.185.* 6152105 6447411233 FIN
> 23:10:00.000000 M F ip 11.140.0.147 <->
> 0.0.0.0 199132 139434535 CON
> 23:10:00.000000 M ip 11.140.0.215 <->
> 10.208.0.0 61391 62223854 CON
>
> So except for the 0.0.0.0 and *.*.0.0 entries, this is starting to
> give me the info I'm looking for. What would be your next step in
> getting some of the L4 info?
>
>
> On Tue, Sep 25, 2012 at 7:01 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Slacker T,
>> Your set of queries are not simple, and so understand that you're starting
>> with an advanced set of questions to solve. It will take a bit to get the
>> right final answer but the approach should become logical after you go
>> through a few steps toward building your result.
>>
>> First you should upgrade to argus-3.0.6. That will fix a number of bugs
>> and will be a good starting point. Your argus-3.0.4 data is fine, but the
>> upgrade will solve some problems.
>>
>> Rasplit() is generating your basic chunks, so we'll work with these files to get going.
>> When you decide that there's a reason to look at a specific chunk, we'll process that
>> whole 5 minute chunk. Doing other granularities is easy, but will add complexity so lets
>> start with your 5 minute files.
>>
>> You're ultimately looking for individual top talkers, not top flows or top matrix, so
>> we'll need to process the data looking for individual addresses. This is a complicated
>> idea, because you have data that has pairs of addresses in it, but you're interested in
>> single addresses. Not hard, but just complicated at first. The secret is the " -M rmon "
>> option.
>>
>> Uploaded and downloaded generally refer to the data that is transported, minus the
>> transport headers, we can do that no problem, so to look at real data loaded, we'll
>> look at the " appbytes " metric.
>>
>> With all of this, lets try to get a list of single address oriented data, and try to rank them
>> by the data they send, the data they receive, both sent and received, and then total
>> bytes, regardless of protocol, and regardless of who initiated the connection.
>>
>> Hopefully this will be helpful.
>>
>> With any of your 5 minute files, try this:
>>
>> racluster -r 5.min.file -w addrs.file -M rmon -m saddr - ip
>>
>> This will generate a set of argus data, that is oriented around the
>> saddr field as the key object. By using the " -M rmon " option,
>> racluster() will move all the addresses into the saddr field, so that
>> simple aggregation will generate a good set of data.
>>
>> Then with this addrs.file, read the file to generate your top talkers. Lets do
>> N=20.
>>
>> rasort -r addrs.file -m sappbytes -No 20
>> rasort -r addrs.file -m dappbytes -No 20
>> rasort -r addrs.file -m sappbytes dappbytes -No 20
>> rasort -r addrs.file -m appbytes -No 20
>>
>> The -No option, specifies the number of records to output.
>>
>> The source metrics report on the data that the object sourced (sent),
>> and the dst metrics are the data where the obj was the destination,
>> so its the data that it received. If this corresponds to your upload/download,
>> then success ?
>>
>> The object field that is useful is the "saddr" field, because that was what we
>> aggregated on (" -m saddr "), as as result, it has the actual address your
>> interested in. In the above commands, to specify the fields you want to
>> print, use the -s option.
>>
>> -s stime dur saddr spkts dpkts sbytes dbytes sappbytes dappbytes
>>
>> You can be a bit more clever, and aggregate on the CIDR saddr, so that you
>> get aggregate data for subnets, rather than actual addresses:
>>
>> racluster -r 5.min.file -w cidrs.file -M rmon -m saddr/24 - ip
>>
>> Now, the address data is pretty straight foward. However, to couple that with
>> L4 data is a bit more interesting. This is complicated enough to wait for another
>> email.
>>
>> Give this a try, and if it seems like its working, great, we can move onto how
>> to take a list of IP addresses and try to do your port reporting. Perl programs
>> like raports() will really help here.
>>
>>
>> Carter
>>
>> On Sep 25, 2012, at 6:27 PM, Slacker T <slackert at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to tie argus into our alerting system. The goal is to take
>>> a snapshot of current usage when certain firewall/switch interfaces
>>> exceed a threshold. I'm having difficulty writing a proper command to
>>> get the information I'm looking for. I'd like to be able to show the
>>> top talkers for a given period of time based first on total amount
>>> downloaded, uploaded, both, or a total of both. Then from that list
>>> I'd like to take the IP and show similar information broken up across
>>> L4 ports (total amount downloaded, uploaded, both, or a total of
>>> both).
>>>
>>> I've gone through the man pages and nsmwiki and I'm a bit overwhelmed.
>>> I'm excited by the granularity of detail argus can provide, and how
>>> easy it is to keep a historical record of traffic, but the learning
>>> curve seems quite steep. I'm thinking that racluster and rasort is
>>> where I need to focus on, but I'm having trouble getting the info I'm
>>> looking for.
>>>
>>> Somewhat relevant info:
>>> I'm collecting argus data from a SPAN port with the following command:
>>> argus -mAJZRU 1024 -i eth3 -P561 -B 127.0.0.1 -w /argus/argus.log -d
>>>
>>> I use rasplit to give me 5 minute chunks to work with (I've got a
>>> script that does some log rotation, but this is the core of it):
>>> /usr/bin/rasplit -r $DATADIR/argus.log-to_be_archived -M time 5m -w
>>> $DATADIR'/archive/%Y/%m/%d/argus.%H.%M.%S.arg3'
>>>
>>> # argus --version
>>> Argus Version 3.0.4
>>>
>>> This is about as far as I have got, if I understand correctly this
>>> uses racluster to total the bytes used by the source addresses in the
>>> argus.14.00.00.arg3 source, then filter out only the IP records and
>>> pipe them to rasort. Rasort sorts on bytes and prints the source
>>> address and byte count..then head shows the top 5.
>>>
>>> racluster -m saddr -r argus.14.00.00.arg3 -w - - ip | rasort -m bytes
>>> -s saddr bytes | head -n 5
>>> 10.210.20.214 50509905
>>> 10.210.21.33 49000815
>>> 10.210.20.110 48677815
>>> 10.210.21.4 37675239
>>> 192.168.100.159 33097697
>>> ...
>>>
>>> If I'm on the right path, great, if not, what am I doing wrong? How do
>>> I get the byte count to show per port and per destination IP?
>>>
>>> Thanks!
>>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20121002/4fe97583/attachment.bin>
More information about the argus
mailing list