pcr and
Frank
argus-mailinglist-1524134246 at f-block.org
Tue Apr 24 15:05:27 EDT 2018
Hi Carter,
thx again for your response.
I decreased my status interval to 5 mins, but it doesn't solve my core
problem. your cmd line works great with connections not using the same 5
tuple, but as soon as there is the same 5 tuple (although there is a
different sequence number for each transmission), it doesn't work anymore
before aggregation:
-----------------------
ra -r /tmp/ntp -p0 -s +mean +sum +trans +seq -s -flgs -s -dir -s -bytes
-s -pkts -s -state -- host 10.10.10.10 | head
StartTime Proto SrcAddr Sport DstAddr
Dport Mean Sum Trans Seq
06:20:44 udp 10.10.10.10.ntp
10.10.10.20.ntp 284 284 1 484094
06:26:35 udp 10.10.10.10.ntp
10.10.10.20.ntp 268 268 1 484454
06:32:07 udp 10.10.10.10.ntp
10.10.10.20.ntp 260 260 1 485046
06:37:31 udp 10.10.10.10.ntp
10.10.10.20.ntp 296 296 1 485046
06:43:37 udp 10.10.10.10.ntp
10.10.10.20.ntp 265 265 1 485727
06:49:08 udp 10.10.10.10.ntp
10.10.10.20.ntp 282 282 1 485992
06:54:56 udp 10.10.10.10.ntp
10.10.10.20.ntp 271 271 1 486350
07:00:37 udp 10.10.10.10.ntp
10.10.10.20.ntp 259 259 1 486780
07:06:00 udp 10.10.10.10.ntp
10.10.10.20.ntp 261 261 1 486780
after aggregation:
-------------------------
racluster -r /tmp/ntp -w - | racluster -m matrix -M dsrs="-agr" -w - |
rasort -m mean -p0 -s +mean +sum +trans +seq -s -flgs -s -dir -s -bytes
-s -pkts -s -state -- host 10.10.10.10
StartTime Proto SrcAddr Sport DstAddr
Dport Mean Sum Trans Seq
06:21:03 udp 10.10.10.20.ntp 10.10.10.10.ntp
137769 137769 1 484094
i'm guessing i could get rid of at least the ntp issue by decreasing the
status interval below 1 second, but thats sth i don't wanna do, and the
problem would reoccur with another system/service that uses a specific
(set of) source port(s) for outgoing connections, where each connection
lasts for a few seconds/minutes...
But i'm guessing there is no other way around this?!
Is there a plan to support the aggregation of flows based on the
sequence number in the future? So instead of doing the first racluster
like this:
racluster -r argus.out -w - | ...
being able to do sth like this:
racluster -r argus.out -m seq -w - | ...
ps: filtering on 'mean gt 1800' doesn't seem to be supported (anymore?):
filter syntax error: 'mean gt 1800'
regards,
Frank
On 04/20/2018 06:48 PM, Carter Bullard wrote:
> Hey Frank,
> Argus is much different that other flow systems. The status interval specifies how often a flow will be report. We do not use state to determine when to generate a flow record, this has been used to keep a flow from being reported, which is not good …
>
> Pick a status interval that works for you, but the longer you hold a flow in cache, the more memory argus requires (can’t garbage collect a flow until the status interval) and more likely the 5-tuple will collide. Its easy and fast to run racluster against your flow data files, before you analyze them. Maybe 5 min ???
>
> So, collect 1 min status interval flow records into a file, racluster it, then look for long durations …
>
> racluster -r argus.out -w - | rafilteraddr -f /tmp/filter -w - | racluster -m matrix -M dsrs=“-agr” - mean gt 1800
>
> Something like that….
>
> Carter
>
>
>> On Apr 20, 2018, at 11:19 AM, Frank <argus-mailinglist-1524134246 at f-block.org> wrote:
>>
>> Hi Carter,
>>
>> thx for your quick answer.
>>
>> 1. That did the trick, thx!
>>
>>
>> 2. Yes, i've set ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
>>
>> The reasons for my big ARGUS_FLOW_STATUS_INTERVAL value:
>> - i don't need instant live updates about active flows
>> - i don't want those updates filling up my log files ( we have many long
>> connections going on, and when there is a status update entry for each
>> of them every second, it gets unnecessarily big)
>> - As far as i understand, flow information are written as soon as the
>> flow is "done" (timeout reached, connection closed,...), so this value
>> shouldn't affect those, right?
>>
>> Ok, what i want in the end is a list of saddr daddr pairs that have long
>> connections with each other, based on the mean field.
>> I have for example those records:
>>
>> rafilteraddr -r argus.out -f /tmp/filter -p0 -s -state -s -flgs -s -dir
>> -s +mean +seq
>> StartTime Proto SrcAddr Sport DstAddr Dport
>> TotPkts TotBytes Mean Seq
>> 21:37:19 tcp 10.10.10.10.42972
>> 10.20.20.20.https 32536 26592499 1790 30487007
>> 22:07:19 tcp 10.10.10.10.42972
>> 10.20.20.20.https 4961 3942584 1795 30487007
>> 22:37:21 tcp 10.10.10.10.42972
>> 10.20.20.20.https 1125 272903 1791 30487007
>> 23:07:21 tcp 10.10.10.10.42972
>> 10.20.20.20.https 494 138085 780 30487007
>> [...]
>> 06:19:14 tcp 10.10.10.10.23391
>> 10.20.20.20.https 730 72320 1793 31292145
>> 06:49:16 tcp 10.10.10.10.23391
>> 10.20.20.20.https 647 57559 1796 31292145
>> 07:19:22 tcp 10.10.10.10.23391
>> 10.20.20.20.https 701 70449 1793 31292145
>> 07:49:25 tcp 10.10.10.10.23391
>> 10.20.20.20.https 685 60024 1794 31292145
>> 08:19:26 tcp 10.10.10.10.23391
>> 10.20.20.20.https 727 72165 1799 31292145
>> 08:49:35 tcp 10.10.10.10.23391
>> 10.20.20.20.https 672 59209 1793 31292145
>>
>>
>> There are two connections (the first lines are from the day before )
>> As i can't aggregate on the sequence number, i use the following command
>> as my first aggregation step:
>>
>> rafilteraddr -r argus.out -f /tmp/filter -w - | racluster -m proto saddr
>> sport daddr dport -M norep -p0 -s -state -s -flgs -s -dir -s +mean +seq
>> +trans +sum
>> StartTime Proto SrcAddr Sport DstAddr Dport
>> TotPkts TotBytes Mean Seq Trans Sum
>> 06:19:14 tcp 10.10.10.10.23391
>> 10.20.20.20.https 8231 776687 0 31292145
>> 0 0
>> 21:37:19 tcp 10.10.10.10.42972
>> 10.20.20.20.https 39116 30946071 0 30487007
>> 0 0
>>
>>
>> I use the '-M norep' option, as without it, i get this, which gives me
>> the mean duration time over all records, but not for connections:
>> StartTime Proto SrcAddr Sport DstAddr Dport
>> TotPkts TotBytes Mean Seq Trans Sum
>> 06:19:14 tcp 10.10.10.10.23391
>> 10.20.20.20.https 8231 776687 1789 31292145
>> 12 21468
>> 21:37:19 tcp 10.10.10.10.42972
>> 10.20.20.20.https 39116 30946071 1539 30487007
>> 4 6156
>>
>>
>> And in my last step, i aggregate with saddr daddr:
>>
>> rafilteraddr -r argus.out -f /tmp/filter -w - | racluster -m proto saddr
>> sport daddr dport -M norep -w - | racluster -m saddr daddr -p0 -s
>> -state -s -flgs -s -dir -s +mean +seq +trans +sum
>> StartTime Proto SrcAddr Sport DstAddr Dport
>> TotPkts TotBytes Mean Seq Trans Sum
>> 21:37:19 tcp 10.10.10.10
>> 10.20.20.20.https 47347 31722758 13877 31292145
>> 2 27753
>>
>> For this scenario, it works as expected: there were 2 connections
>> between those two ip addresses, with a total duration of 27753 and mean
>> 13877.
>>
>> But as i said, the remaining problem are connections that use the same
>> 5-tuple for multiple connections/flows as with ntp.
>> Did i understand/do somehting wrong here?
>>
>>
>> 3. Ah, okay, understood.
>>
>> Thx and Regards,
>> Frank
>>
>>
>> On 04/20/2018 03:38 PM, Carter Bullard wrote:
>>> Hey Frank,
>>> The PCR requires that you have ARGUS_GENERATE_APPBYTE_METRIC set in argus. The PCR is based on the values of sappbytes and dappbytes, so as a sanity check, print these fields using ra to see if the metric is being generated.
>>>
>>> So not sure what you’re wanting to achieve in 2. The 5-tuple aggregation is the default, as that is the “basic” way of forming flows. Its always better if you include the command line that you are using, so we can see whats up.
>>>
>>> But, why use such a large ARGUS_FLOW_STATUS_INTERVAL, I use either 5 or 1, and never go much higher. This is the definition of your “quantum" of observation. With a 3 hour interval, man, do you have to wait a long time before anything comes out if your monitoring live traffic. Biggest problem is that your going to see a lot of port reuse in 3 hours, so flows will collide, and you’ll get large flow duration times. If you use default aggregation, then the mean, stddev, max and min are calculated using the ‘due’ field of each record. So it looks like it is working.
>>>
>>> Use a small ARGUS_FLOW_STATUS_INTERVAL, and then use aggregation to build up the long lived flows.
>>>
>>> So, #3 is a ‘gotcha’ of sorts. “tcp” and “udp” are really shortcut aliases, tcp means “ip proto tcp”, so you're generating filter errors by accident ??? Because tcp and udp and icmp are used so often, they are aliased, seems that all the other protocols (i.e udplite) need the ip proto.
>>>
>>> So this works…
>>>
>>> ra -b - not udp
>>> ra -b - not \(ip proto 17\)
>>>
>>> So quoting them makes them a string, rather than the keyword, and we then lookup them up, rather than assign the value. Sorry for that inconvenience.
>>>
>>> Carter
>>>
>>>> On Apr 20, 2018, at 6:39 AM, Frank <argus-mailinglist-1524134246 at f-block.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> i'm new to the list and to argus, so i hope i don't repeat already asked
>>>> questions.
>>>> First of all: i really like the workflow of argus and its functionality.
>>>> But i encountered some obstacles that however might also be caused by my
>>>> misunderstanding.
>>>>
>>>> Argus server and client version: 3.0.8.2
>>>>
>>>>
>>>> 1. Is it possible that the PCR functionality is broken? I always get a
>>>> PCR of -0
>>>>
>>>> ra -r argus.out -s sbytes dbytes pcr | head
>>>> SrcBytes DstBytes PCRatio
>>>> 83 126 -0.000000
>>>> 1644 4213 -0.000000
>>>> 83 126 -0.000000
>>>> 194 0 -0.000000
>>>> 194 0 -0.000000
>>>> 73 89 -0.000000
>>>> 73 101 -0.000000
>>>> 74 60 -0.000000
>>>> 3351 4025 -0.000000
>>>>
>>>>
>>>> 2. Did i see right that it is not possible to aggregate on the argus
>>>> sequence number, or did i overlook something? The main reason i'd
>>>> want/need this is for an accurate mean duration time. The best i was
>>>> able to accomplish was an aggregation (via racluster) over proto src/dst
>>>> addr/port, but this does e.g. not work with ntp queries.
>>>> I've set ARGUS_FLOW_STATUS_INTERVAL=1800 , and when i look for long
>>>> connections and aggregate with '-m proto saddr sport daddr dport' , i
>>>> get a maximum mean time of 1800, as each status message seems to be
>>>> treated as a separate flow (at least it looks that way to me). When i
>>>> also use -M norep, it works for all connections where the selected
>>>> features are unique for the connection, but, as mentioned before, does
>>>> not work with ntp connections, or for chatty hosts that use the same
>>>> source port for a destination host twice or more.
>>>>
>>>>
>>>> 3. Another issue that i encounter is a weird filtering behaviour.
>>>> For example:
>>>>
>>>> ra -r argus.out -- 'not (ip proto udp)'
>>>> ra[26153]: 1524219408.776398 filter syntax error: 'not (ip proto udp)'
>>>>
>>>> ra -r argus.out -- 'not (ip proto tcp)'
>>>> ra[26219]: 12:19:04.157946 filter syntax error: 'not (ip proto tcp)'
>>>>
>>>> ra -r argus.out -- 'not (ip proto udp or tcp)'
>>>> ra[26233]: 12:19:31.355346 filter syntax error: 'not (ip proto udp or ...'
>>>>
>>>> But this works:
>>>> ra -r argus.out -- 'not (ip proto "udp")'
>>>> ra -r argus.out -- 'not (ip proto 17 or tcp)'
>>>> ra -r argus.out -- 'not (ip proto "udp" or tcp)'
>>>> ra -r argus.out -- 'not udp and not tcp'
>>>>
>>>>
>>>> Thanks in advance and regards,
>>>> Frank
>>>>
>>>>
>>>>
>>
More information about the argus
mailing list