Errors in gap detection
elof2 at sentor.se
elof2 at sentor.se
Mon Oct 27 12:39:23 EDT 2014
On Mon, 27 Oct 2014, Carter Bullard wrote:
> Hmmm, a few things.
> Argus does not flush flow records because of the protocol state.
> This would be a very bad thing, especially in the presence of an
> attack that does weird things with the TCP flags.
Ah, ok. I just assumed it flushed it after the FIN.
> The default status interval is 5 seconds. Because you are reusing
> port numbers, we’ll just match all your packets into the same flow,
> until argus idle times out the cache.
Ok.
> The default idle time, where argus tosses a TCP cache is 60 seconds.
> You should be waiting 60 seconds between runs, or change the
> ARGUS_TCP_TIMEOUT to something shorter for your experiments.
Ok, so when I wait more than 60 seconds, the pkts counters are
almost correct, and the sequence number is always still showing the
incorrect value:
12:54:35.268526 e * 1087 -> 443 55 101
FSPA_FSPA 0 0 1 1.559693 1651741098 1545322196
12:55:11.691944 e * 1087 -> 443 55 101
FSPA_FSPA 0 0 1 1.559695 1651741098 1545322196
17:25:10.080434 e * 1087 -> 443 55 101
FSPA_FSPA 0 0 1 1.559707 1651741098 1545322196
17:29:15.374957 e * 1087 -> 443 55 101
FSPA_FSPA 0 0 1 1.559699 1651741098 1545322196
It says 55 spkts. It should be 56.
1651741098 should be 1651739529.
1545322196 should be 1545191373.
> Can you share the flow records that generate the bad gap counts ???
> We calculate gaps in the client, based on whats in the flow record.
> Maybe a bug in that algorithm ????
Yes, I'll filter out that particular flow, verify there's still crazy
numbers in the new file, and send it to you.
I'm on my way to the train home right now. I'll do it when I come home or
tomorrow.
/Elof
> On Oct 27, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>
>> On Sun, 26 Oct 2014, Carter Bullard wrote:
>>
>>> Idle field is not something you will want to use, in this case.
>>> That is intended for tools like ratop.1 so you can see how
>>> the idle time is growing as nothing is coming in .
>>
>> Ok. You asked "So what are the flow idle timeout values in your sensor(s)?", apparently you were looking for something else. :)
>>
>> I use this config:
>> # cat /etc/argus.conf
>> ARGUS_MONITOR_ID=1.2.3.4
>> ARGUS_INTERFACE=mon0
>> ARGUS_OUTPUT_FILE=/usr/foobar/log/out.log
>> ARGUS_DAEMON=yes
>> ARGUS_ACCESS_PORT=0
>> ARGUS_GENERATE_MAC_DATA=yes
>> ARGUS_CAPTURE_DATA_LEN=120
>> ARGUS_FILTER=""
>>
>> Everything else is using the defaults.
>>
>>
>>> Since both flows are reporting the same number, my guess is that its algorithmic,
>>> but there is more to this. So those flows are not the same set of packets, as
>>> the stime is off and the base sequence numbers are not the same ????
>>
>> Yes, they are the same.
>> 'stime' differ since I replayed the pcap using tcpreplay to my test/lab sensor.
>> Why the sequence numbers differ I don't know. That is probably the main issue here, and the reason for the crazy gap numbers.
>>
>>> Can you print the duration ??? Print out the ‘trans’ field.
>>
>> On the live sensor that logged the crazy gap numbers (argus has been running for days and it has previously seen flows for these same IPs and ports, so there could possibly be garbage still present in memory when the following flow gets logged):
>>
>> ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr gaps.argus - tcp and port 1087
>> StartTime Flgs Sport Dir Dport SrcPkts DstPkts State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
>> 13:42:43.511666 e g 1087 -> 443 56 101 FSPA_FSPA 11274535 1767532* 1 1.599528 1640464994 1368438167
>> 13:42:43.511743 e g 1087 -> 443 56 101 FSPA_FSPA 11274535 1767532* 1 1.599467 1640464994 1368438167
>>
>>
>> Same command on the new logfile where I replayed the pcap (freshly started argus daemon, so no garbage in memory):
>>
>> ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr out_tcpreplay_gaps_after_kill.log - tcp and port 1087
>> StartTime Flgs Sport Dir Dport SrcPkts DstPkts State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
>> 17:41:56.413382 e 1087 -> 443 56 101 FSPA_FSPA 0 0 1 1.631145 1651739529 1545191373
>> 17:41:56.413462 e 1087 -> 443 56 101 FSPA_FSPA 0 0 1 1.631081 1651739529 1545191373
>>
>> So, again, stime differ because this was a tcp-replay I did in the evening.
>>
>> The diff in duration could either be because of some old garbage numbers in argus memory, or more likely, it is simply the minor differences in time when the pcap was written to hdd plus minor differences in time when it was replayed on the network.
>>
>> The diff in sequence numbers however can't be explained. The problem must be here.
>> My guess is that when argus see reused IP+ports some garbage remain. (see the logs where I could see that ports are reused day after day)
>> OR
>> When argus see two almost identical flows happening at the same time, something is screwed up and tracked to the wrong flow.
>> Like when the SPAN is setup to mirror both the external link AND the DMZ-vlan.
>> Client x.x.x.x do a 3way handshake towards the NAT-IP y.y.y.y:443.
>> The NAT-box now do a 3way handshake of its own towards 10.10.10.10:443 (only dst-NAT, the src remain x.x.x.x).
>> The client send a request (PA) which is NATed into a new PA-packet on the DMZ.
>> Argus see both these PA-packets:
>> x.x.x.x:1087 <-> y.y.y.y:443
>> and
>> x.x.x.x:1087 <-> 10.10.10.10:443
>> The server ACKs this and sends data back, and so on.
>> The two sessions send data and they finally terminate with FIN.
>>
>> Looking at these two flows in the fresh argus daemon log, everything looks just fine.
>> Could something be left in memory after the flows terminate, so the next time they reuse the port numbers, we get crazy numbers?
>>
>>
>> I start a new argus daemon and replay the pcap with a single tcp flow a few times (at normal speed and sequentially):
>>
>> The pcap contain a single tcp connection. 56 packets from x.x.x.x and 101 response packets back from 10.10.10.10. (See the entire pcap below)
>>
>> # tcpreplay --preload-pcap --stats=1 -i mon0 no_gaps.pcap
>> I waited a few seconds and then ran it again, and again, and again...
>>
>> ra show no crazy gap numbers but it DOES show different sequence numbers the first run compared to the rest:
>>
>> # ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr out.log -
>> StartTime Flgs Sport Dir Dport SrcPkts DstPkts State SrcGap DstGap Trans Dur SrcTCPBase DstTCPBase
>> 12:02:15.837014 e * 1087 -> 443 112 202 FSPA_FSPA 0 0 1 4.726716 1651739529 1545191373
>> 12:02:21.147487 e * 1087 -> 443 166 302 FSPA_FSPA 0 0 1 4.717483 1651741098 1545322196
>> 12:02:26.874580 e * 1087 -> 443 135 274 FSPA_FSPA 0 0 1 4.996352 1651741098 1545322196
>> 12:02:31.928752 e * 1087 -> 443 94 134 FSPA_FSPA 0 0 1 5.069161 1651741098 1545322196
>> 12:02:37.055272 e * 1087 -> 443 52 98 FPA_FPA 0 0 1 1.440231 1651741098 1545322196
>> 12:03:18.478272 e * 1087 -> 443 174 307 FSPA_FSPA 0 0 1 4.948444 1651741098 1545322196
>> 12:03:23.556139 e * 1087 -> 443 202 399 FSPA_FSPA 0 0 1 4.989495 1651741098 1545322196
>> 12:03:28.595580 e * 1087 -> 443 182 303 FSPA_FSPA 0 0 1 4.829063 1651741098 1545322196
>> 12:03:34.434253 e * 1087 -> 443 176 308 FSPA_FSPA 0 0 1 4.988512 1651741098 1545322196
>> ...
>>
>> So the first flow show different sequence numbers compared to the second one. This is strange since I replay the exact same tcp flow.
>> pcap seq: 1651739529 and 1545191373
>> So the first flow gets logged correctly while all the others are incorrect.
>>
>> Also, all packet counters look strange! When the tcp connection in the replayed pcap is finished (FIN), argus don't seem to purge the flow from memory and start treating the new replayed flow as a new one. This next replay gets accumulated into the previous flow's counters.
>> All flows should have "56 101" packets, not "112 202", "166 302", "135 274", etc.
>>
>>
>>> What is your ARGUS_MAR_STAUS_INTERVAL ???
>>
>> Default (which I think is 5 min). See conf above.
>>
>>
>>> So are any of the packets out of order ???
>>
>> Wireshark doesn't complain and I've even manually checked all 56 and 101 packets in the flows. No out of order. Everything looks fine.
>>
>>
>>> And you aren’t aggregating any of these records, that could be cause some the problems ???
>>
>> Nope. I'm just running argus with the above config file and looking at the out-file with 'ra'.
>> Both daemon and client is 3.0.8.
>>
>>
>>> No way you can share a tcpdump of just one flows that has the gaps problem ???
>>
>> I don't have a pcap that generate the problem. The pcap I have generates correct values when replayed to a fresh argus daemon.
>> The only way to get a pcap that generate crazy numbers *might* be to capture *lots* of flows, perhaps over a day. Enough traffic to trigger the fault in argus... Then checking with ra if crazy numbers show up and terminate the tcpdump. This pcap would probaly be waaay bigger than my entire HDD, so I guess it' too hard to do. Plus the fact that this is possibly sensitive customer-data that I would need to anonymize before putting it on a FTP-server or simmilar.
>> Sorry.
>> (But see below for a txt version of my test-pcap with just one single flow)
>>
>>
>>> So this email has too many issues in it. We can talk about each independently.
>>> Argus should not be holding onto flows, it is not state based. And killing argus
>>> doesn’t cause argus to flush, it causes the kernel to flush its buffers to
>>> disk, because we close the file. We don’t flush the output socket to disk, as its very
>>> expensive.
>>
>> Ah, ok.
>>
>>
>>
>>
>>
>> The replayed file no_gaps.pcap contain these 56+101 packets. No more. All of them look sane. No out of order. The session is terminated with FIN:s:
>>
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [S], seq 1651739529, win 14600, options [mss 1380,sackOK,TS val 28452891 ecr 0,nop,wscale 6], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [S.], seq 1545191373, ack 1651739530, win 28960, options [mss 1460,sackOK,TS val 584372731 ecr 28452891,nop,wscale 7], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 28452897 ecr 584372731], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 1:439, ack 1, win 229, options [nop,nop,TS val 28452897 ecr 584372731], length 438
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 439, win 235, options [nop,nop,TS val 584372746 ecr 28452897], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 1:159, ack 439, win 235, options [nop,nop,TS val 584372746 ecr 28452897], length 158
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 439:567, ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 128
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 567:1569, ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 1002
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 1569, win 259, options [nop,nop,TS val 584372768 ecr 28452903], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 159:1527, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 1527:2895, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 2895:4263, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 4263:5631, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 5631:6999, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 6999:8367, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 8367:9735, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 9735:11103, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 11103:12471, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 12471:13839, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 2895, win 336, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 5631, win 426, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 13839:15207, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 15207:16575, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 16575:17943, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 17943:19311, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 19311:20679, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 20679:22047, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 22047:23415, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 23415:24783, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 8367, win 517, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 11103, win 607, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 13839, win 698, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 24783:26151, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 26151:27519, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 27519:28887, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 28887:30255, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 30255:31623, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 31623:32991, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 32991:34359, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 34359:35727, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 35727:37095, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 37095:38463, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 38463:39831, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 39831:41199, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 41199:42567, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 42567:43935, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 43935:45303, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 16575, win 788, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 19311, win 879, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 45303:46671, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 46671:48039, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 48039:49407, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 49407:50775, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 50775:52143, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 22047, win 969, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 24783, win 1060, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 27519, win 1150, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 30255, win 1241, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 32991, win 1331, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 52143:53511, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 53511:54879, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 54879:56247, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 56247:57615, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 57615:58983, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 58983:60351, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 60351:61719, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 61719:63087, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 63087:64455, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 64455:65823, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 65823:67191, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 67191:68559, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 68559:69927, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 69927:71295, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 71295:72663, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 72663:74031, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 74031:75399, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 75399:76767, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 76767:78135, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 78135:79503, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 35727, win 1422, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 38463, win 1512, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 41199, win 1603, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 79503:80871, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 80871:82239, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 82239:83607, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 83607:84975, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 84975:86343, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 86343:87711, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 87711:89079, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 89079:90447, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 90447:91815, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 43935, win 1693, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 46671, win 1784, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 91815:93183, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 93183:94551, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 94551:95919, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 95919:97287, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 49407, win 1874, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 52143, win 1965, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 54879, win 2055, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 57615, win 2146, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 60351, win 2236, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 97287:98655, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 98655:100023, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 100023:101391, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 101391:102759, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 102759:104127, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 104127:105495, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 105495:106863, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 106863:108231, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 108231:109599, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 109599:110967, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 63087, win 2327, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 65823, win 2417, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 68559, win 2508, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 71295, win 2598, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 74031, win 2689, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 110967:112335, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 112335:113703, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 113703:115071, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 115071:116439, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 116439:117807, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 117807:119175, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 119175:120543, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 120543:121911, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 121911:123279, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 123279:124647, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 76767, win 2779, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 79503, win 2870, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 82239, win 2960, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 84975, win 3051, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 124647:126015, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 126015:127383, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 127383:128751, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 128751:130119, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 130119:130823, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 704
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 87711, win 3141, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 90447, win 3232, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 93183, win 3322, options [nop,nop,TS val 28452941 ecr 584372826], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 95919, win 3413, options [nop,nop,TS val 28452942 ecr 584372840], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 98655, win 3503, options [nop,nop,TS val 28452943 ecr 584372840], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 101391, win 3594, options [nop,nop,TS val 28452944 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 104127, win 3684, options [nop,nop,TS val 28452944 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 106863, win 3775, options [nop,nop,TS val 28452945 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 109599, win 3865, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 112335, win 3956, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 115071, win 4046, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 117807, win 4137, options [nop,nop,TS val 28452947 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 120543, win 4227, options [nop,nop,TS val 28452947 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 123279, win 4318, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 126015, win 4408, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 128751, win 4499, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130823, win 4589, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [F.], seq 1569, ack 130823, win 4589, options [nop,nop,TS val 28453048 ecr 584372846], length 0
>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [F.], seq 130823, ack 1570, win 259, options [nop,nop,TS val 584373120 ecr 28453048], length 0
>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130824, win 4589, options [nop,nop,TS val 28453053 ecr 584373120], length 0
>>
>>
>> /Elof
>>
>>
>>>> On Oct 23, 2014, at 12:16 PM, elof2 at sentor.se wrote:
>>>>
>>>>
>>>> Hi!
>>>>
>>>> The crazy flow look like this with idle, stcpb and dtcpb:
>>>>
>>>> ra -Zb -s stime:9 flgs saddr sport dir:3 daddr dport spkts dpkts state:13 sgap dgap idle:14 stcpb dtcpb -nr gaps.argus - tcp
>>>> 13:42:43.511666 e g x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
>>>> 13:42:43.511743 e g x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
>>>>
>>>> Apparently the network SPAN is setup to both mirror the outside vlan (before NAT:ing y.y.y.y to 10.10.10.10) and the inside vlan. So argus see both flows on its sniffer interface.
>>>> Both flows were caught in my pcap. There are no gaps in either of them. I see how x.x.x.x do a threeway handshake towards y.y.y.y. Then I see how the inside if the NAT-fw send its SYN to 10.10.10.10. The two sessions are happening at the same time, not sequentially.
>>>> Ra show the crazy gap values for both flows.
>>>>
>>>>
>>>>
>>>> The same two flows looks like this when I replayed the full pcap (20 seconds of traffic):
>>>>
>>>> 17:41:56.413382 e x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
>>>> 17:41:56.413462 e x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
>>>>
>>>>
>>>> I restarted argus and replayed a filtered pcap which only contained one of the flows (all 157 packets).
>>>> out.log is only 128 bytes and growing 128 bytes every minute due to the MAR-status events.
>>>> I've waited more than 5 minutes and the single flow is still not flushed to the file.
>>>> I don't know if this is due to the fact that no more data at all is coming in on the sniffer interface, or if argus didn't realize that the connection has finished (the FIN packets are sent).
>>>> Anyhow, I kill the argus daemon to force it to flush the data into out.log.
>>>>
>>>> 17:52:04.561207 e x.x.x.x.1087 -> 10.10.10.10.443 55 101 FSPA_FSPA 0 0 1414080256.00* 1651739529 1545191373
>>>>
>>>>
>>>> So in both pcap cases, there are zero gaps reported, and the base sequence numbers are the same.
>>>> However, the sequence numbers do NOT match the ones from the long-running argus daemon.
>>>>
>>>>
>>>>
>>>>
>>>> Side-step:
>>>> Regarding the non-flushing of the single flow to out.log. Is that a bug or work as intended?
>>>>
>>>> /Elof
>>>>
>>>>
>>>>
>>>> On Thu, 23 Oct 2014, Carter Bullard wrote:
>>>>
>>>>> So what are the flow idle timeout values in your sensor(s) ???
>>>>> Could it be really long, leaving older base sequence numbers
>>>>> around, and we’re getting port reuse ??? Or it could be we’re
>>>>> seeing sequence number rollover ??? What are the stcpb and
>>>>> dtcpb for the flows that have crazy numbers ???
>>>>>
>>>>> Carter
>>>>>
>>>>> On Oct 23, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>>>>>
>>>>>>
>>>>>> Hi Carter!
>>>>>>
>>>>>> Sorry, but I can't give you a pcap or argus-logfile since they contain sensitive data.
>>>>>>
>>>>>>
>>>>>> I'm also sorry to say that this will probably be hard to debug.
>>>>>>
>>>>>> 'cause when I replayed the pcap on another sensor, ra showed "0 0"
>>>>>> gaps for this flow instead of "11274535 1767532*". This is correct, since I found no gaps in the flow in the pcap.
>>>>>>
>>>>>> (the other flow I analysed show the correct "0 1367" since one packet is really missing in this flow in the pcap)
>>>>>>
>>>>>> So a freshly started argus daemon seem to log correct values.
>>>>>>
>>>>>>
>>>>>> I found another sensor with lots of crazy numbers.
>>>>>> See attached logfile.
>>>>>>
>>>>>> In there you can see that 1.2.3.4 is running a continous web-spider towards a wiki on 2.2.2.2:80.
>>>>>> All the GET requests always generates flows with approximately 11 packes in each direction.
>>>>>> There are zero gaps for days, and then suddenly there is a burst of crazy numbers during a period of 40 seconds. Then everything is good for two hours and then another 40 second burst.
>>>>>>
>>>>>>
>>>>>> netstat -B show zero drops for the argus daemon.
>>>>>> My graphs for the cpu usage, swapping, memory usage, packets per second, etc show nothing out of the ordinary. The machine is not heavily loaded. Doesn't swap. It only receives 35 Mbps of mirrored traffic.
>>>>>> No spike or unusual activity during the 40 seconds of crazy numbers.
>>>>>>
>>>>>>
>>>>>> I can't find any pattern or reason for the sudden burst of crazy numbers.
>>>>>>
>>>>>> Other traffic flows show "0 0" gaps during the crazy periods, so not all flows are affected (not even all flows between 1.2.3.4 and 2.2.2.2, but many of them).
>>>>>>
>>>>>> /Elof
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 22 Oct 2014, Carter Bullard wrote:
>>>>>>
>>>>>>> Can you send the pcap file ?? Does argus generale the crazy
>>>>>>> numbers with this file ???
>>>>>>> Carter
>>>>>>>
>>>>>>>> On Oct 22, 2014, at 9:39 AM, elof2 at sentor.se wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Carter!
>>>>>>>>
>>>>>>>> FYI, the gap detection counters still show some wonky numbers in 3.0.8.
>>>>>>>>
>>>>>>>> ra -Zb -s flgs spkts dpkts state:13 sgap dgap -nr gaps.argus - tcp | grep g
>>>>>>>>
>>>>>>>> Normal and OK gaps show up like this:
>>>>>>>> Flgs SrcPkts DstPkts State SrcGap DstGap
>>>>>>>> e g 15 24 PA_PA 0 458
>>>>>>>> e g 3 3 PA_PA 0 284
>>>>>>>> e g 5 6 PA_PA 0 732
>>>>>>>> e g 129 284 PA_PA 0 1367
>>>>>>>> e g 66 94 PA_PA 0 1367
>>>>>>>> e g 2 2 PA_PA 0 801
>>>>>>>>
>>>>>>>> ...but here and there I get lines like this:
>>>>>>>> e g 7 6 FSPA_FSPA 3051561* 8894044*
>>>>>>>> e g 7 6 FSPA_FSPA 1142343* 8891853*
>>>>>>>> e g 7 6 FSPA_FSPA 98000397 7385371*
>>>>>>>> e g 6 4 FSPA_FSPA 59208794 6255514*
>>>>>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>>>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>>>>>> e g 5 6 SPA_SPA 2562525* 9087142*
>>>>>>>> e g 7 6 FSPA_FSPA 68629719 5826434*
>>>>>>>> e g 7 6 FSPA_FSPA -214748* 5815425*
>>>>>>>> e g 7 6 FSPA_FSPA -214748* 1919818*
>>>>>>>> e g 17 24 FSPA_FSPA 3167486* 2765698*
>>>>>>>>
>>>>>>>> This doesn't look as nice.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I tcpdump:ed traffic to pcap while argus created its logfile.
>>>>>>>> I analysed two flows showing gaps, one normal with "0 1367" gaps and one wonky with "11274535 1767532*" gaps.
>>>>>>>>
>>>>>>>> Wireshark analysis of the "normal flow" show identical numbers as argus; 0 gaps from src and 1367 bytes (one packet) missing (in mid-stream) from dst.
>>>>>>>> Good.
>>>>>>>>
>>>>>>>> Wireshark analysis of a "wonky" flow show no errors! No complaints at all from Wireshark (including its Expert Info). No "previous segment not captured" and no "ACKed unseen segment".
>>>>>>>> Everything looks good in the pcap.
>>>>>>>> I can't find any reason as to why argus create those wonky numbers.
>>>>>>>>
>>>>>>>>
>>>>>>>> Oh, well, I don't use the gaps fields very often, so for me this is not important. I just thought I'd let you know.
>>>>>>>>
>>>>>>>> /Elof
>>>>>>>>
>>>>>> <gaps.txt>
>>>>>
>
>
More information about the argus
mailing list