[ARGUS] Question about merging flows (racluster) and effect of srcid

[Patrick Forsberg]

I am yet again having trouble merging my flows.

We have two border routers in active-active configuration and we have
two sensors connected to these routers. This means sensor data have
different srcid.

Each sensor collects data to file and the files are rotated every five
minutes.
We send the sensor files to an aggregation server where all analysis is
done.

Files are aggregated with

racluster -f racluster.conf -r sensor1.ra sensor2.ra -w output ra.

With racluster.conf:
RACLUSTER_AUTO_CORRECTION=yes
filter=""              model="saddr daddr proto sport dport

For some reason, running racluster -X -M correct -m saddr daddr proto
sport dport will not give the same result!

If a full flow passes through both sensors flow direction will sometimes
be messed up and mostly look unidirectional. Is there a way to ignore
srcid when aggregating?

I've anonymized my example data so that 1.0.0.0/24 is an external
network and 100.0.0.0/16 is my internal network.
What we see is an external IP (1.0.0.6) probing an internal IP (100.0.0.27)

Sensor1 data

ra -X -r sensor1.ra -s+trans,srcid
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State 
Trans              SrcId
   09:57:29.270837  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65430         2        120   RST      1        100.0.1.33
   09:57:43.038193  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65082         2        120   RST      1        100.0.1.33
   09:58:02.606511  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65353         2        120   RST      1        100.0.1.33

Sensor2 data
ra -X -r sensor2.ra -s+trans,srcid
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State 
Trans              SrcId
   09:57:29.270906  e           tcp       100.0.0.27.65430     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34
   09:57:43.038263  e           tcp       100.0.0.27.65082     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34
   09:58:02.606602  e           tcp       100.0.0.27.65353     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34

If we read both sensor1 and sensor2 with rasort, we clearly see that
incoming traffic comes through sensor1 and outgoing traffic through sensor2

rasort -X -m stime -s+trans,srcid -r sensor1.ra sensor2.ra
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State 
Trans              SrcId
   09:57:29.270837  e           tcp      1.0.0.6.47546     ->      
100.0.0.7.65430         2        120   RST      1        100.0.1.33
   09:57:29.270906  e           tcp       100.0.0.7.65430     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34
   09:57:43.038193  e           tcp      1.0.0.6.47546     ->      
100.0.0.7.65082         2        120   RST      1        100.0.1.33
   09:57:43.038263  e           tcp       100.0.0.7.65082     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34
   09:58:02.606511  e           tcp      1.0.0.6.47546     ->      
100.0.0.7.65353         2        120   RST      1        100.0.1.33
   09:58:02.606602  e           tcp       100.0.0.7.65353     ?>     
1.0.0.6.47546         1         60   RST      1        100.0.1.34

Now, running racluster it's evident that the order in which the sensor
files are read affects the flow direction. In this case we need to read
sensor1 before sensor2.

racluster -X -f /tmp/raclu.conf -r sensor1.ra sensor2.ra -s+trans
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State  Trans
   09:57:29.270837  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65430         3        180   RST      2
   09:57:43.038193  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65082         3        180   RST      2
   09:58:02.606511  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65353         3        180   RST      2

racluster -X -f /tmp/raclu.conf -r sensor2.ra sensor1.ra -s+trans
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State  Trans
   09:57:29.270837  e           tcp      1.0.0.6.47546    <-       
100.0.0.27.65430         3        180   RST      2
   09:57:43.038193  e           tcp      1.0.0.6.47546    <-       
100.0.0.27.65082         3        180   RST      2
   09:58:02.606511  e           tcp      1.0.0.6.47546    <-       
100.0.0.27.65353         3        180   RST      2

I will get a somewhat better result i read the data with rasort and then
pipe it to racluster. It still looks as if traffic only flows one way
though, and I gather this is because of the srcid not being the same.
rasort -X -r sensor1.ra sensor2.ra -w - |racluster -X -f /tmp/raclu.conf
-s+trans
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State  Trans
   09:57:29.270837  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65430         3        180   RST      2
   09:57:43.038193  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65082         3        180   RST      2
   09:58:02.606511  e           tcp      1.0.0.6.47546     ->      
100.0.0.27.65353         3        180   RST      2

The result that I expected is more like the one below.
         StartTime      Flgs  Proto            SrcAddr  Sport  
Dir            DstAddr  Dport  TotPkts   TotBytes State  Trans
   09:57:29.270837  e           tcp      1.0.0.6.47546     <->      
100.0.0.27.65430         3        180   RST      2
   09:57:43.038193  e           tcp      1.0.0.6.47546     <->      
100.0.0.27.65082         3        180   RST      2
   09:58:02.606511  e           tcp      1.0.0.6.47546     <->      
100.0.0.27.65353         3        180   RST      2

Does anyone know how to solve this?

Regards,

Patrick Forsberg
Chalmers IRT