[ARGUS] Question about merging flows (racluster) and effect of srcid
Carter Bullard
carter at qosient.com
Thu Feb 11 07:23:32 EST 2021
Hey Patrick,
The default aggreagation flow model is:
srcid saddr daddr proto sport dport
This keeps flows from different srcids from merging together. In your example, you are removing the srcid
saddr daddr proto sport dport
and you will combine the flows as if they had come from the same argus monitoring a single wire.
Put the srcid back into your flow models(in your racluster.conf file) and see if things don't start to make a bit more sense. When you want to combine them. then take the srcid out. This may be as simple as using 2 rounds of aggregation ... one to aggregate with the srcid, followed by one without. Correction will try to find a flow to match and merge, and the resulting algorithm is "first flow wins" thinking you figured it out ... there are exceptions as it is clever with TCP state, so correction may be generating artifacts, especially if the 2 sensors are not sych'd in time ...
When you are being scanned, the source doesn't use an actual TCP stack yo generare the challenge packet, so you can get all sorrs of non-conforming states which can confyse tge direction ... if you suspect a scan ... don't use correction, or don't pay attention to the direction ... they are trying to confuse your firewall to get into your space ... that process will confuse your analysis ...
I'm replying from my phone so I haven't studied your email ... will check it out on the big screen later today ...
Carter
Carter Bullard • CEO
250 E 53rd Street Suite 501
New York, New York 10022
Phone +1.212.588.9133 • Mobile +1.917.497.9494
> On Feb 10, 2021, at 12:33 PM, Patrick Forsberg <fors at chalmers.se> wrote:
>
> I am yet again having trouble merging my flows.
>
> We have two border routers in active-active configuration and we have
> two sensors connected to these routers. This means sensor data have
> different srcid.
>
> Each sensor collects data to file and the files are rotated every five
> minutes.
> We send the sensor files to an aggregation server where all analysis is
> done.
>
> Files are aggregated with
>
> racluster -f racluster.conf -r sensor1.ra sensor2.ra -w output ra.
>
> With racluster.conf:
> RACLUSTER_AUTO_CORRECTION=yes
> filter="" model="saddr daddr proto sport dport
>
> For some reason, running racluster -X -M correct -m saddr daddr proto
> sport dport will not give the same result!
>
> If a full flow passes through both sensors flow direction will sometimes
> be messed up and mostly look unidirectional. Is there a way to ignore
> srcid when aggregating?
>
> I've anonymized my example data so that 1.0.0.0/24 is an external
> network and 100.0.0.0/16 is my internal network.
> What we see is an external IP (1.0.0.6) probing an internal IP (100.0.0.27)
>
> Sensor1 data
>
> ra -X -r sensor1.ra -s+trans,srcid
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State
> Trans SrcId
> 09:57:29.270837 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65430 2 120 RST 1 100.0.1.33
> 09:57:43.038193 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65082 2 120 RST 1 100.0.1.33
> 09:58:02.606511 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65353 2 120 RST 1 100.0.1.33
>
> Sensor2 data
> ra -X -r sensor2.ra -s+trans,srcid
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State
> Trans SrcId
> 09:57:29.270906 e tcp 100.0.0.27.65430 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
> 09:57:43.038263 e tcp 100.0.0.27.65082 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
> 09:58:02.606602 e tcp 100.0.0.27.65353 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
>
> If we read both sensor1 and sensor2 with rasort, we clearly see that
> incoming traffic comes through sensor1 and outgoing traffic through sensor2
>
> rasort -X -m stime -s+trans,srcid -r sensor1.ra sensor2.ra
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State
> Trans SrcId
> 09:57:29.270837 e tcp 1.0.0.6.47546 ->
> 100.0.0.7.65430 2 120 RST 1 100.0.1.33
> 09:57:29.270906 e tcp 100.0.0.7.65430 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
> 09:57:43.038193 e tcp 1.0.0.6.47546 ->
> 100.0.0.7.65082 2 120 RST 1 100.0.1.33
> 09:57:43.038263 e tcp 100.0.0.7.65082 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
> 09:58:02.606511 e tcp 1.0.0.6.47546 ->
> 100.0.0.7.65353 2 120 RST 1 100.0.1.33
> 09:58:02.606602 e tcp 100.0.0.7.65353 ?>
> 1.0.0.6.47546 1 60 RST 1 100.0.1.34
>
> Now, running racluster it's evident that the order in which the sensor
> files are read affects the flow direction. In this case we need to read
> sensor1 before sensor2.
>
> racluster -X -f /tmp/raclu.conf -r sensor1.ra sensor2.ra -s+trans
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State Trans
> 09:57:29.270837 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65430 3 180 RST 2
> 09:57:43.038193 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65082 3 180 RST 2
> 09:58:02.606511 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65353 3 180 RST 2
>
> racluster -X -f /tmp/raclu.conf -r sensor2.ra sensor1.ra -s+trans
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State Trans
> 09:57:29.270837 e tcp 1.0.0.6.47546 <-
> 100.0.0.27.65430 3 180 RST 2
> 09:57:43.038193 e tcp 1.0.0.6.47546 <-
> 100.0.0.27.65082 3 180 RST 2
> 09:58:02.606511 e tcp 1.0.0.6.47546 <-
> 100.0.0.27.65353 3 180 RST 2
>
> I will get a somewhat better result i read the data with rasort and then
> pipe it to racluster. It still looks as if traffic only flows one way
> though, and I gather this is because of the srcid not being the same.
> rasort -X -r sensor1.ra sensor2.ra -w - |racluster -X -f /tmp/raclu.conf
> -s+trans
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State Trans
> 09:57:29.270837 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65430 3 180 RST 2
> 09:57:43.038193 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65082 3 180 RST 2
> 09:58:02.606511 e tcp 1.0.0.6.47546 ->
> 100.0.0.27.65353 3 180 RST 2
>
> The result that I expected is more like the one below.
> StartTime Flgs Proto SrcAddr Sport
> Dir DstAddr Dport TotPkts TotBytes State Trans
> 09:57:29.270837 e tcp 1.0.0.6.47546 <->
> 100.0.0.27.65430 3 180 RST 2
> 09:57:43.038193 e tcp 1.0.0.6.47546 <->
> 100.0.0.27.65082 3 180 RST 2
> 09:58:02.606511 e tcp 1.0.0.6.47546 <->
> 100.0.0.27.65353 3 180 RST 2
>
> Does anyone know how to solve this?
>
> Regards,
>
> Patrick Forsberg
> Chalmers IRT
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20210211/18e1a91d/attachment-0001.html>
More information about the argus
mailing list