Errors in gap detection
elof2 at sentor.se
elof2 at sentor.se
Mon Oct 27 09:41:30 EDT 2014
On Sun, 26 Oct 2014, Carter Bullard wrote:
> Idle field is not something you will want to use, in this case.
> That is intended for tools like ratop.1 so you can see how
> the idle time is growing as nothing is coming in .
Ok. You asked "So what are the flow idle timeout values in your
sensor(s)?", apparently you were looking for something else. :)
I use this config:
# cat /etc/argus.conf
ARGUS_MONITOR_ID=1.2.3.4
ARGUS_INTERFACE=mon0
ARGUS_OUTPUT_FILE=/usr/foobar/log/out.log
ARGUS_DAEMON=yes
ARGUS_ACCESS_PORT=0
ARGUS_GENERATE_MAC_DATA=yes
ARGUS_CAPTURE_DATA_LEN=120
ARGUS_FILTER=""
Everything else is using the defaults.
> Since both flows are reporting the same number, my guess is that its algorithmic,
> but there is more to this. So those flows are not the same set of packets, as
> the stime is off and the base sequence numbers are not the same ????
Yes, they are the same.
'stime' differ since I replayed the pcap using tcpreplay to my test/lab
sensor.
Why the sequence numbers differ I don't know. That is probably the main
issue here, and the reason for the crazy gap numbers.
> Can you print the duration ??? Print out the ‘trans’ field.
On the live sensor that logged the crazy gap numbers (argus has been
running for days and it has previously seen flows for these same IPs and
ports, so there could possibly be garbage still present in memory when
the following flow gets logged):
ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap
trans dur stcpb dtcpb -nr gaps.argus - tcp and port 1087
StartTime Flgs Sport Dir Dport SrcPkts DstPkts
State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
13:42:43.511666 e g 1087 -> 443 56 101
FSPA_FSPA 11274535 1767532* 1 1.599528 1640464994 1368438167
13:42:43.511743 e g 1087 -> 443 56 101
FSPA_FSPA 11274535 1767532* 1 1.599467 1640464994 1368438167
Same command on the new logfile where I replayed the pcap (freshly
started argus daemon, so no garbage in memory):
ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap
trans dur stcpb dtcpb -nr out_tcpreplay_gaps_after_kill.log - tcp and port 1087
StartTime Flgs Sport Dir Dport SrcPkts DstPkts
State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
17:41:56.413382 e 1087 -> 443 56 101
FSPA_FSPA 0 0 1 1.631145 1651739529 1545191373
17:41:56.413462 e 1087 -> 443 56 101
FSPA_FSPA 0 0 1 1.631081 1651739529 1545191373
So, again, stime differ because this was a tcp-replay I did in the
evening.
The diff in duration could either be because of some old garbage numbers
in argus memory, or more likely, it is simply the minor differences in
time when the pcap was written to hdd plus minor differences in time when
it was replayed on the network.
The diff in sequence numbers however can't be explained. The problem must
be here.
My guess is that when argus see reused IP+ports some garbage remain. (see
the logs where I could see that ports are reused day after day)
OR
When argus see two almost identical flows happening at the same time,
something is screwed up and tracked to the wrong flow.
Like when the SPAN is setup to mirror both the external link AND the
DMZ-vlan.
Client x.x.x.x do a 3way handshake towards the NAT-IP y.y.y.y:443.
The NAT-box now do a 3way handshake of its own towards 10.10.10.10:443
(only dst-NAT, the src remain x.x.x.x).
The client send a request (PA) which is NATed into a new PA-packet on the
DMZ.
Argus see both these PA-packets:
x.x.x.x:1087 <-> y.y.y.y:443
and
x.x.x.x:1087 <-> 10.10.10.10:443
The server ACKs this and sends data back, and so on.
The two sessions send data and they finally terminate with FIN.
Looking at these two flows in the fresh argus daemon log, everything looks
just fine.
Could something be left in memory after the flows terminate, so the next
time they reuse the port numbers, we get crazy numbers?
I start a new argus daemon and replay the pcap with a single tcp flow a
few times (at normal speed and sequentially):
The pcap contain a single tcp connection. 56 packets from x.x.x.x and 101
response packets back from 10.10.10.10. (See the entire pcap below)
# tcpreplay --preload-pcap --stats=1 -i mon0 no_gaps.pcap
I waited a few seconds and then ran it again, and again, and again...
ra show no crazy gap numbers but it DOES show different sequence numbers
the first run compared to the rest:
# ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap
trans dur stcpb dtcpb -nr out.log -
StartTime Flgs Sport Dir Dport SrcPkts DstPkts
State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
12:02:15.837014 e * 1087 -> 443 112 202
FSPA_FSPA 0 0 1 4.726716 1651739529 1545191373
12:02:21.147487 e * 1087 -> 443 166 302
FSPA_FSPA 0 0 1 4.717483 1651741098 1545322196
12:02:26.874580 e * 1087 -> 443 135 274
FSPA_FSPA 0 0 1 4.996352 1651741098 1545322196
12:02:31.928752 e * 1087 -> 443 94 134
FSPA_FSPA 0 0 1 5.069161 1651741098 1545322196
12:02:37.055272 e * 1087 -> 443 52 98
FPA_FPA 0 0 1 1.440231 1651741098 1545322196
12:03:18.478272 e * 1087 -> 443 174 307
FSPA_FSPA 0 0 1 4.948444 1651741098 1545322196
12:03:23.556139 e * 1087 -> 443 202 399
FSPA_FSPA 0 0 1 4.989495 1651741098 1545322196
12:03:28.595580 e * 1087 -> 443 182 303
FSPA_FSPA 0 0 1 4.829063 1651741098 1545322196
12:03:34.434253 e * 1087 -> 443 176 308
FSPA_FSPA 0 0 1 4.988512 1651741098 1545322196
...
So the first flow show different sequence numbers compared to the second
one. This is strange since I replay the exact same tcp flow.
pcap seq: 1651739529 and 1545191373
So the first flow gets logged correctly while all the others are
incorrect.
Also, all packet counters look strange! When the tcp connection in the
replayed pcap is finished (FIN), argus don't seem to purge the flow from
memory and start treating the new replayed flow as a new one. This next
replay gets accumulated into the previous flow's counters.
All flows should have "56 101" packets, not "112 202", "166
302", "135 274", etc.
> What is your ARGUS_MAR_STAUS_INTERVAL ???
Default (which I think is 5 min). See conf above.
> So are any of the packets out of order ???
Wireshark doesn't complain and I've even manually checked all 56 and 101
packets in the flows. No out of order. Everything looks fine.
> And you aren’t aggregating any of these records, that could be cause some the problems ???
Nope. I'm just running argus with the above config file and looking at the
out-file with 'ra'.
Both daemon and client is 3.0.8.
> No way you can share a tcpdump of just one flows that has the gaps problem ???
I don't have a pcap that generate the problem. The pcap I have generates
correct values when replayed to a fresh argus daemon.
The only way to get a pcap that generate crazy numbers *might* be to
capture *lots* of flows, perhaps over a day. Enough traffic to trigger the
fault in argus... Then checking with ra if crazy numbers show up and
terminate the tcpdump. This pcap would probaly be waaay bigger than my
entire HDD, so I guess it' too hard to do. Plus the fact that this is
possibly sensitive customer-data that I would need to anonymize before
putting it on a FTP-server or simmilar.
Sorry.
(But see below for a txt version of my test-pcap with just one single
flow)
> So this email has too many issues in it. We can talk about each independently.
> Argus should not be holding onto flows, it is not state based. And killing argus
> doesn’t cause argus to flush, it causes the kernel to flush its buffers to
> disk, because we close the file. We don’t flush the output socket to disk, as its very
> expensive.
Ah, ok.
The replayed file no_gaps.pcap contain these 56+101 packets. No more. All
of them look sane. No out of order. The session is terminated with FIN:s:
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [S], seq 1651739529, win 14600,
options [mss 1380,sackOK,TS val 28452891 ecr 0,nop,wscale 6], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [S.], seq 1545191373, ack
1651739530, win 28960, options [mss 1460,sackOK,TS val 584372731 ecr
28452891,nop,wscale 7], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 1, win 229, options
[nop,nop,TS val 28452897 ecr 584372731], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 1:439, ack 1, win 229,
options [nop,nop,TS val 28452897 ecr 584372731], length 438
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 439, win 235, options
[nop,nop,TS val 584372746 ecr 28452897], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 1:159, ack 439, win
235, options [nop,nop,TS val 584372746 ecr 28452897], length 158
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 159, win 245, options
[nop,nop,TS val 28452903 ecr 584372746], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 439:567, ack 159, win
245, options [nop,nop,TS val 28452903 ecr 584372746], length 128
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 567:1569, ack 159, win
245, options [nop,nop,TS val 28452903 ecr 584372746], length 1002
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 1569, win 259, options
[nop,nop,TS val 584372768 ecr 28452903], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 159:1527, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 1527:2895, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 2895:4263, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 4263:5631, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 5631:6999, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 6999:8367, ack 1569,
win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 8367:9735, ack 1569, win
259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 9735:11103, ack 1569,
win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 11103:12471, ack 1569,
win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 12471:13839, ack 1569,
win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 2895, win 336, options
[nop,nop,TS val 28452920 ecr 584372782], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 5631, win 426, options
[nop,nop,TS val 28452920 ecr 584372782], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 13839:15207, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 15207:16575, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 16575:17943, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 17943:19311, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 19311:20679, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 20679:22047, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 22047:23415, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 23415:24783, ack 1569,
win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 8367, win 517, options
[nop,nop,TS val 28452920 ecr 584372782], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 11103, win 607, options
[nop,nop,TS val 28452920 ecr 584372782], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 13839, win 698, options
[nop,nop,TS val 28452920 ecr 584372782], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 24783:26151, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 26151:27519, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 27519:28887, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 28887:30255, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 30255:31623, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 31623:32991, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 32991:34359, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 34359:35727, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 35727:37095, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 37095:38463, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 38463:39831, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 39831:41199, ack 1569,
win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 41199:42567, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 42567:43935, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 43935:45303, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 16575, win 788, options
[nop,nop,TS val 28452927 ecr 584372800], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 19311, win 879, options
[nop,nop,TS val 28452927 ecr 584372800], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 45303:46671, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 46671:48039, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 48039:49407, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 49407:50775, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 50775:52143, ack 1569,
win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 22047, win 969, options
[nop,nop,TS val 28452927 ecr 584372800], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 24783, win 1060, options
[nop,nop,TS val 28452927 ecr 584372800], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 27519, win 1150, options
[nop,nop,TS val 28452927 ecr 584372806], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 30255, win 1241, options
[nop,nop,TS val 28452927 ecr 584372806], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 32991, win 1331, options
[nop,nop,TS val 28452927 ecr 584372806], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 52143:53511, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 53511:54879, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 54879:56247, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 56247:57615, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 57615:58983, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 58983:60351, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 60351:61719, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 61719:63087, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 63087:64455, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 64455:65823, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 65823:67191, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 67191:68559, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 68559:69927, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 69927:71295, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 71295:72663, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 72663:74031, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 74031:75399, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 75399:76767, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 76767:78135, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 78135:79503, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 35727, win 1422, options
[nop,nop,TS val 28452928 ecr 584372806], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 38463, win 1512, options
[nop,nop,TS val 28452928 ecr 584372806], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 41199, win 1603, options
[nop,nop,TS val 28452928 ecr 584372806], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 79503:80871, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 80871:82239, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 82239:83607, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 83607:84975, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 84975:86343, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 86343:87711, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 87711:89079, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 89079:90447, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 90447:91815, ack 1569,
win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 43935, win 1693, options
[nop,nop,TS val 28452934 ecr 584372820], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 46671, win 1784, options
[nop,nop,TS val 28452934 ecr 584372820], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 91815:93183, ack 1569,
win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 93183:94551, ack 1569,
win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 94551:95919, ack 1569,
win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 95919:97287, ack 1569,
win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 49407, win 1874, options
[nop,nop,TS val 28452934 ecr 584372820], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 52143, win 1965, options
[nop,nop,TS val 28452934 ecr 584372820], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 54879, win 2055, options
[nop,nop,TS val 28452935 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 57615, win 2146, options
[nop,nop,TS val 28452935 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 60351, win 2236, options
[nop,nop,TS val 28452935 ecr 584372826], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 97287:98655, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 98655:100023, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 100023:101391, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 101391:102759, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 102759:104127, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 104127:105495, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 105495:106863, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 106863:108231, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 108231:109599, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 109599:110967, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 63087, win 2327, options
[nop,nop,TS val 28452936 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 65823, win 2417, options
[nop,nop,TS val 28452936 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 68559, win 2508, options
[nop,nop,TS val 28452936 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 71295, win 2598, options
[nop,nop,TS val 28452936 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 74031, win 2689, options
[nop,nop,TS val 28452938 ecr 584372826], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 110967:112335, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 112335:113703, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 113703:115071, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 115071:116439, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 116439:117807, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 117807:119175, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 119175:120543, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 120543:121911, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 121911:123279, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 123279:124647, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 76767, win 2779, options
[nop,nop,TS val 28452938 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 79503, win 2870, options
[nop,nop,TS val 28452938 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 82239, win 2960, options
[nop,nop,TS val 28452938 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 84975, win 3051, options
[nop,nop,TS val 28452939 ecr 584372826], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 124647:126015, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 126015:127383, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 127383:128751, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 128751:130119, ack 1569,
win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 130119:130823, ack
1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 704
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 87711, win 3141, options
[nop,nop,TS val 28452939 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 90447, win 3232, options
[nop,nop,TS val 28452939 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 93183, win 3322, options
[nop,nop,TS val 28452941 ecr 584372826], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 95919, win 3413, options
[nop,nop,TS val 28452942 ecr 584372840], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 98655, win 3503, options
[nop,nop,TS val 28452943 ecr 584372840], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 101391, win 3594,
options [nop,nop,TS val 28452944 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 104127, win 3684,
options [nop,nop,TS val 28452944 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 106863, win 3775,
options [nop,nop,TS val 28452945 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 109599, win 3865,
options [nop,nop,TS val 28452946 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 112335, win 3956,
options [nop,nop,TS val 28452946 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 115071, win 4046,
options [nop,nop,TS val 28452946 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 117807, win 4137,
options [nop,nop,TS val 28452947 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 120543, win 4227,
options [nop,nop,TS val 28452947 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 123279, win 4318,
options [nop,nop,TS val 28452948 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 126015, win 4408,
options [nop,nop,TS val 28452948 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 128751, win 4499,
options [nop,nop,TS val 28452948 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130823, win 4589,
options [nop,nop,TS val 28452948 ecr 584372846], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [F.], seq 1569, ack 130823, win
4589, options [nop,nop,TS val 28453048 ecr 584372846], length 0
IP 10.10.10.10.443 > x.x.x.x.1087: Flags [F.], seq 130823, ack 1570, win
259, options [nop,nop,TS val 584373120 ecr 28453048], length 0
IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130824, win 4589,
options [nop,nop,TS val 28453053 ecr 584373120], length 0
/Elof
>> On Oct 23, 2014, at 12:16 PM, elof2 at sentor.se wrote:
>>
>>
>> Hi!
>>
>> The crazy flow look like this with idle, stcpb and dtcpb:
>>
>> ra -Zb -s stime:9 flgs saddr sport dir:3 daddr dport spkts dpkts state:13 sgap dgap idle:14 stcpb dtcpb -nr gaps.argus - tcp
>> 13:42:43.511666 e g x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
>> 13:42:43.511743 e g x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
>>
>> Apparently the network SPAN is setup to both mirror the outside vlan (before NAT:ing y.y.y.y to 10.10.10.10) and the inside vlan. So argus see both flows on its sniffer interface.
>> Both flows were caught in my pcap. There are no gaps in either of them. I see how x.x.x.x do a threeway handshake towards y.y.y.y. Then I see how the inside if the NAT-fw send its SYN to 10.10.10.10. The two sessions are happening at the same time, not sequentially.
>> Ra show the crazy gap values for both flows.
>>
>>
>>
>> The same two flows looks like this when I replayed the full pcap (20 seconds of traffic):
>>
>> 17:41:56.413382 e x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
>> 17:41:56.413462 e x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
>>
>>
>> I restarted argus and replayed a filtered pcap which only contained one of the flows (all 157 packets).
>> out.log is only 128 bytes and growing 128 bytes every minute due to the MAR-status events.
>> I've waited more than 5 minutes and the single flow is still not flushed to the file.
>> I don't know if this is due to the fact that no more data at all is coming in on the sniffer interface, or if argus didn't realize that the connection has finished (the FIN packets are sent).
>> Anyhow, I kill the argus daemon to force it to flush the data into out.log.
>>
>> 17:52:04.561207 e x.x.x.x.1087 -> 10.10.10.10.443 55 101 FSPA_FSPA 0 0 1414080256.00* 1651739529 1545191373
>>
>>
>> So in both pcap cases, there are zero gaps reported, and the base sequence numbers are the same.
>> However, the sequence numbers do NOT match the ones from the long-running argus daemon.
>>
>>
>>
>>
>> Side-step:
>> Regarding the non-flushing of the single flow to out.log. Is that a bug or work as intended?
>>
>> /Elof
>>
>>
>>
>> On Thu, 23 Oct 2014, Carter Bullard wrote:
>>
>>> So what are the flow idle timeout values in your sensor(s) ???
>>> Could it be really long, leaving older base sequence numbers
>>> around, and we’re getting port reuse ??? Or it could be we’re
>>> seeing sequence number rollover ??? What are the stcpb and
>>> dtcpb for the flows that have crazy numbers ???
>>>
>>> Carter
>>>
>>> On Oct 23, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>>>
>>>>
>>>> Hi Carter!
>>>>
>>>> Sorry, but I can't give you a pcap or argus-logfile since they contain sensitive data.
>>>>
>>>>
>>>> I'm also sorry to say that this will probably be hard to debug.
>>>>
>>>> 'cause when I replayed the pcap on another sensor, ra showed "0 0"
>>>> gaps for this flow instead of "11274535 1767532*". This is correct, since I found no gaps in the flow in the pcap.
>>>>
>>>> (the other flow I analysed show the correct "0 1367" since one packet is really missing in this flow in the pcap)
>>>>
>>>> So a freshly started argus daemon seem to log correct values.
>>>>
>>>>
>>>> I found another sensor with lots of crazy numbers.
>>>> See attached logfile.
>>>>
>>>> In there you can see that 1.2.3.4 is running a continous web-spider towards a wiki on 2.2.2.2:80.
>>>> All the GET requests always generates flows with approximately 11 packes in each direction.
>>>> There are zero gaps for days, and then suddenly there is a burst of crazy numbers during a period of 40 seconds. Then everything is good for two hours and then another 40 second burst.
>>>>
>>>>
>>>> netstat -B show zero drops for the argus daemon.
>>>> My graphs for the cpu usage, swapping, memory usage, packets per second, etc show nothing out of the ordinary. The machine is not heavily loaded. Doesn't swap. It only receives 35 Mbps of mirrored traffic.
>>>> No spike or unusual activity during the 40 seconds of crazy numbers.
>>>>
>>>>
>>>> I can't find any pattern or reason for the sudden burst of crazy numbers.
>>>>
>>>> Other traffic flows show "0 0" gaps during the crazy periods, so not all flows are affected (not even all flows between 1.2.3.4 and 2.2.2.2, but many of them).
>>>>
>>>> /Elof
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 22 Oct 2014, Carter Bullard wrote:
>>>>
>>>>> Can you send the pcap file ?? Does argus generale the crazy
>>>>> numbers with this file ???
>>>>> Carter
>>>>>
>>>>>> On Oct 22, 2014, at 9:39 AM, elof2 at sentor.se wrote:
>>>>>>
>>>>>>
>>>>>> Hi Carter!
>>>>>>
>>>>>> FYI, the gap detection counters still show some wonky numbers in 3.0.8.
>>>>>>
>>>>>> ra -Zb -s flgs spkts dpkts state:13 sgap dgap -nr gaps.argus - tcp | grep g
>>>>>>
>>>>>> Normal and OK gaps show up like this:
>>>>>> Flgs SrcPkts DstPkts State SrcGap DstGap
>>>>>> e g 15 24 PA_PA 0 458
>>>>>> e g 3 3 PA_PA 0 284
>>>>>> e g 5 6 PA_PA 0 732
>>>>>> e g 129 284 PA_PA 0 1367
>>>>>> e g 66 94 PA_PA 0 1367
>>>>>> e g 2 2 PA_PA 0 801
>>>>>>
>>>>>> ...but here and there I get lines like this:
>>>>>> e g 7 6 FSPA_FSPA 3051561* 8894044*
>>>>>> e g 7 6 FSPA_FSPA 1142343* 8891853*
>>>>>> e g 7 6 FSPA_FSPA 98000397 7385371*
>>>>>> e g 6 4 FSPA_FSPA 59208794 6255514*
>>>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>>>> e g 5 6 SPA_SPA 2562525* 9087142*
>>>>>> e g 7 6 FSPA_FSPA 68629719 5826434*
>>>>>> e g 7 6 FSPA_FSPA -214748* 5815425*
>>>>>> e g 7 6 FSPA_FSPA -214748* 1919818*
>>>>>> e g 17 24 FSPA_FSPA 3167486* 2765698*
>>>>>>
>>>>>> This doesn't look as nice.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I tcpdump:ed traffic to pcap while argus created its logfile.
>>>>>> I analysed two flows showing gaps, one normal with "0 1367" gaps and one wonky with "11274535 1767532*" gaps.
>>>>>>
>>>>>> Wireshark analysis of the "normal flow" show identical numbers as argus; 0 gaps from src and 1367 bytes (one packet) missing (in mid-stream) from dst.
>>>>>> Good.
>>>>>>
>>>>>> Wireshark analysis of a "wonky" flow show no errors! No complaints at all from Wireshark (including its Expert Info). No "previous segment not captured" and no "ACKed unseen segment".
>>>>>> Everything looks good in the pcap.
>>>>>> I can't find any reason as to why argus create those wonky numbers.
>>>>>>
>>>>>>
>>>>>> Oh, well, I don't use the gaps fields very often, so for me this is not important. I just thought I'd let you know.
>>>>>>
>>>>>> /Elof
>>>>>>
>>>> <gaps.txt>
>>>
>
More information about the argus
mailing list