Errors in gap detection
elof2 at sentor.se
elof2 at sentor.se
Thu Oct 23 12:16:04 EDT 2014
Hi!
The crazy flow look like this with idle, stcpb and dtcpb:
ra -Zb -s stime:9 flgs saddr sport dir:3 daddr dport spkts dpkts state:13 sgap dgap idle:14 stcpb dtcpb -nr gaps.argus - tcp
13:42:43.511666 e g x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
13:42:43.511743 e g x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 11274535 1767532* 1414077184.00* 1640464994 1368438167
Apparently the network SPAN is setup to both mirror the outside vlan
(before NAT:ing y.y.y.y to 10.10.10.10) and the inside vlan.
So argus see both flows on its sniffer interface.
Both flows were caught in my pcap. There are no gaps in either of them. I
see how x.x.x.x do a threeway handshake towards y.y.y.y. Then I see how
the inside if the NAT-fw send its SYN to 10.10.10.10. The two sessions
are happening at the same time, not sequentially.
Ra show the crazy gap values for both flows.
The same two flows looks like this when I replayed the full pcap (20
seconds of traffic):
17:41:56.413382 e x.x.x.x.1087 -> y.y.y.y.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
17:41:56.413462 e x.x.x.x.1087 -> 10.10.10.10.443 56 101 FSPA_FSPA 0 0 1414079232.00* 1651739529 1545191373
I restarted argus and replayed a filtered pcap which only contained one of
the flows (all 157 packets).
out.log is only 128 bytes and growing 128 bytes every minute due to the
MAR-status events.
I've waited more than 5 minutes and the single flow is still not flushed
to the file.
I don't know if this is due to the fact that no more data at all is coming
in on the sniffer interface, or if argus didn't realize that the
connection has finished (the FIN packets are sent).
Anyhow, I kill the argus daemon to force it to flush the data into
out.log.
17:52:04.561207 e x.x.x.x.1087 -> 10.10.10.10.443 55 101 FSPA_FSPA 0 0 1414080256.00* 1651739529 1545191373
So in both pcap cases, there are zero gaps reported, and the base sequence
numbers are the same.
However, the sequence numbers do NOT match the ones from the
long-running argus daemon.
Side-step:
Regarding the non-flushing of the single flow to out.log. Is that a bug or
work as intended?
/Elof
On Thu, 23 Oct 2014, Carter Bullard wrote:
> So what are the flow idle timeout values in your sensor(s) ???
> Could it be really long, leaving older base sequence numbers
> around, and we’re getting port reuse ??? Or it could be we’re
> seeing sequence number rollover ??? What are the stcpb and
> dtcpb for the flows that have crazy numbers ???
>
> Carter
>
> On Oct 23, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>
>>
>> Hi Carter!
>>
>> Sorry, but I can't give you a pcap or argus-logfile since they contain sensitive data.
>>
>>
>> I'm also sorry to say that this will probably be hard to debug.
>>
>> 'cause when I replayed the pcap on another sensor, ra showed "0 0"
>> gaps for this flow instead of "11274535 1767532*". This is correct, since I found no gaps in the flow in the pcap.
>>
>> (the other flow I analysed show the correct "0 1367" since one packet is really missing in this flow in the pcap)
>>
>> So a freshly started argus daemon seem to log correct values.
>>
>>
>> I found another sensor with lots of crazy numbers.
>> See attached logfile.
>>
>> In there you can see that 1.2.3.4 is running a continous web-spider towards a wiki on 2.2.2.2:80.
>> All the GET requests always generates flows with approximately 11 packes in each direction.
>> There are zero gaps for days, and then suddenly there is a burst of crazy numbers during a period of 40 seconds. Then everything is good for two hours and then another 40 second burst.
>>
>>
>> netstat -B show zero drops for the argus daemon.
>> My graphs for the cpu usage, swapping, memory usage, packets per second, etc show nothing out of the ordinary. The machine is not heavily loaded. Doesn't swap. It only receives 35 Mbps of mirrored traffic.
>> No spike or unusual activity during the 40 seconds of crazy numbers.
>>
>>
>> I can't find any pattern or reason for the sudden burst of crazy numbers.
>>
>> Other traffic flows show "0 0" gaps during the crazy periods, so not all flows are affected (not even all flows between 1.2.3.4 and 2.2.2.2, but many of them).
>>
>> /Elof
>>
>>
>>
>>
>> On Wed, 22 Oct 2014, Carter Bullard wrote:
>>
>>> Can you send the pcap file ?? Does argus generale the crazy
>>> numbers with this file ???
>>> Carter
>>>
>>>> On Oct 22, 2014, at 9:39 AM, elof2 at sentor.se wrote:
>>>>
>>>>
>>>> Hi Carter!
>>>>
>>>> FYI, the gap detection counters still show some wonky numbers in 3.0.8.
>>>>
>>>> ra -Zb -s flgs spkts dpkts state:13 sgap dgap -nr gaps.argus - tcp | grep g
>>>>
>>>> Normal and OK gaps show up like this:
>>>> Flgs SrcPkts DstPkts State SrcGap DstGap
>>>> e g 15 24 PA_PA 0 458
>>>> e g 3 3 PA_PA 0 284
>>>> e g 5 6 PA_PA 0 732
>>>> e g 129 284 PA_PA 0 1367
>>>> e g 66 94 PA_PA 0 1367
>>>> e g 2 2 PA_PA 0 801
>>>>
>>>> ...but here and there I get lines like this:
>>>> e g 7 6 FSPA_FSPA 3051561* 8894044*
>>>> e g 7 6 FSPA_FSPA 1142343* 8891853*
>>>> e g 7 6 FSPA_FSPA 98000397 7385371*
>>>> e g 6 4 FSPA_FSPA 59208794 6255514*
>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>> e g 20 20 FSPA_FSPA 3468512* 65538
>>>> e g 5 6 SPA_SPA 2562525* 9087142*
>>>> e g 7 6 FSPA_FSPA 68629719 5826434*
>>>> e g 7 6 FSPA_FSPA -214748* 5815425*
>>>> e g 7 6 FSPA_FSPA -214748* 1919818*
>>>> e g 17 24 FSPA_FSPA 3167486* 2765698*
>>>>
>>>> This doesn't look as nice.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I tcpdump:ed traffic to pcap while argus created its logfile.
>>>> I analysed two flows showing gaps, one normal with "0 1367" gaps and one wonky with "11274535 1767532*" gaps.
>>>>
>>>> Wireshark analysis of the "normal flow" show identical numbers as argus; 0 gaps from src and 1367 bytes (one packet) missing (in mid-stream) from dst.
>>>> Good.
>>>>
>>>> Wireshark analysis of a "wonky" flow show no errors! No complaints at all from Wireshark (including its Expert Info). No "previous segment not captured" and no "ACKed unseen segment".
>>>> Everything looks good in the pcap.
>>>> I can't find any reason as to why argus create those wonky numbers.
>>>>
>>>>
>>>> Oh, well, I don't use the gaps fields very often, so for me this is not important. I just thought I'd let you know.
>>>>
>>>> /Elof
>>>>
>> <gaps.txt>
>
>
More information about the argus
mailing list