Errors in gap detection

elof2 at sentor.se elof2 at sentor.se
Tue Jun 2 07:18:41 EDT 2015


Hi Carter!


Huh?!?! Not responded?

My last email was not a bug report in itself, it was merely a reminder 
that the problem reported in 2014 still exist.
The pasted data was just to illustrate that 3.0.8.1 behaves exactly as 
reported in 2014.

For full and detailed reports, see our discussions in October last year.

See the subject "[ARGUS] Errors in gap detection" on dates 22, 23, 26, 27, 
28 and 29 Oct 2014.
On Oct 30 2014 I sent you the pcap (subject "Pcap") which generates the 
problem if you replay it twice.

After that you were busy, on the road and distracted for months, asking me 
to bother you about this issue from time to time, which I have done 
approximately once every month since I think this is a major bug in argus 
and need to be fixed. :)



So, I've sent you the argus log, the pcap, instructions how to reproduce, 
full bug reports and reminders.
I don't know what more I can do.
I'm no programmer, so I can't fix a patch myself. :-/

Please re-read the discussion from the start.




Summary:

The pcap I sent you contain a single tcp flow, 157 packets in total, 
starting with SYN and finishing with FIN. There are no gaps or errors in 
the flow.


<I added ARGUS_TCP_TIMEOUT=10 to /etc/argus.conf>
# argus -S 0.0000001
# tcpreplay --stats=1 -i mon0 /root/no_gaps.pcap
   <waited a few minutes>
   'ra' now show 156 lines.
   The next line, for the last _FA, is not yet flushed to file.
I now replay the same flow again:
# tcpreplay --stats=1 -i mon0 /root/no_gaps.pcap
   <waited a few minutes>
   Now line 157 has appeared in the ra output together with the new
   packets from the second flow, all lines except the last one (313).
# killall argus
   Now line 313 appeared in ra.

We can see that in a freshly started argus, the first time it see the SYN, 
the following is recorded:
             Flgs  Sport Dir  Dport  SrcPkts  DstPkts State   SrcGap 
DstGap  Trans        Dur   SrcTCPBase   DstTCPBase
    001  e        1087    -> 443           1        0 S_        0        0 
1   0.000000   1651739529
    002  e        1087    -> 443           0        1 _SA        0        0 
1   0.000000   1651739529   1545191373
    003  e        1087    -> 443           1        0 A_        0        0 
1   0.000000   1651739529   1545191373
    004  e        1087    -> 443           1        0 PA_        0        0 
1   0.000000   1651739529   1545191373
    005  e        1087    -> 443           0        1 _A        0        0 
1   0.000000   1651739968   1545191373
    006  e        1087    -> 443           0        1 _PA        0        0 
1   0.000000   1651739968   1545191373
    007  e        1087    -> 443           1        0 A_        0        0 
1   0.000000   1651739968   1545191532
    008  e        1087    -> 443           1        0 PA_        0        0 
1   0.000000   1651739968   1545191532
    009  e        1087    -> 443           1        0 PA_        0        0 
1   0.000000   1651740096   1545191532
    010  e        1087    -> 443           0        1 _A        0        0 
1   0.000000   1651741098   1545191532
    011  e        1087    -> 443           0        1 _A        0        0 
1   0.000000   1651741098   1545191532
    012  e        1087    -> 443           0        1 _A        0        0 
1   0.000000   1651741098   1545192900
    013  e        1087    -> 443           0        1 _A        0        0 
1   0.000000   1651741098   1545194268
    ...
    ...
    155  e        1087    -> 443           1        0 A_        0        0 
1   0.000000   1651741098   1545322196
    156  e        1087    -> 443           1        0 FA_        0        0 
1   0.000000   1651741098   1545322196
    157  e        1087    -> 443           0        1 _FA        0        0 
1   0.000000   1651741098   1545322196

See how the seq starts off correctly with 1651739529 and moves to 
1651741098.
So far everything is good.

But...
The new SYN in line 158 show 1651741098 instead of the expected 
1651739529.
This flow reuse some cached state even though it has been minutes between 
the flows.

    ...
    155  e        1087    -> 443           1        0 A_        0        0 
1   0.000000   1651741098   1545322196
    156  e        1087    -> 443           1        0 FA_        0        0 
1   0.000000   1651741098   1545322196
    157  e        1087    -> 443           0        1 _FA        0        0 
1   0.000000   1651741098   1545322196

    158  e        1087    -> 443           1        0 S_        0        0 
1   0.000000   1651741098   1545322196
    159  e        1087    -> 443           0        1 _SA        0        0 
1   0.000000   1651741098   1545322196
    160  e        1087    -> 443           1        0 A_        0        0 
1   0.000000   1651741098   1545322196
    ...





Another problem:
The src packet counter is wrong.

If I use the default resolution, not -S 0.0000001, I see this in 'ra':
   e *      1087    -> 443          55      101 FSPA_FSPA

It says 55 spkts. It should be 56.
55 spkts + 101 dpkts = 156 pkts. The flow in the pcap is 157 pkts.


/Elof



On Tue, 2 Jun 2015, Carter Bullard wrote:

> I am sorry, but this is not a reasonable bug report.
> How did you generate the out.log file ???
>
> Are you still writing the same packet file to an interface multiple times with long delays and then attempting to read valid data ??
> Does it generate gaps when you simply read the packet file directly with argus ??
>
> Print out the flgs field.  Is there any loss in these flows ???
> Why haven’t you responded to any of my private email to you ??
>
> Carter
>
>> On May 27, 2015, at 9:03 AM, elof2 at sentor.se wrote:
>>
>>
>> (replying to an old mail)
>>
>>
>> Hi Carter!
>>
>> Sorry to say, you didn't solve this error in the 3.0.8.1 release. :-/
>>
>>
>> I still get weird gap values. Example:
>>
>> # ra -Zb -s stime:9 proto sport dir:3 dport spkts dpkts sbytes dbytes state:13 sgap dgap -nr out.log - tcp | grep '*'
>> 14:42:57.060893    tcp 17978   -> 443          69      124         5475 169883     FSPA_FSPA        0 -214748*
>> 14:42:57.158769    tcp 17978   -> 443          69      124         5461 169883     FSPA_FSPA -214748* 98808124
>> 14:42:57.258804    tcp 17986   -> 443          11        8         1271 5131     FSPA_FSPA        0 1469744*
>> 14:42:57.406707    tcp 17986   -> 443          11        8         1257 5131     FSPA_FSPA 4791073* 3632332*
>> 14:43:00.177403    tcp 18093   -> 443         116      195        11319 258305     FSPA_FSPA -214748* 66628995
>> 14:43:13.721668    tcp 18051   -> 80           94      133         7869 178011    FSRPA_FSPA -214748*        0
>> 14:43:20.359447    tcp 1059    -> 80          234      246        17756 334441     FSPA_FSPA 9902869*        0
>> 14:43:22.063313    tcp 1175    -> 80           48       53         3626 68754     FSPA_FSPA 9850686*        0
>> 14:43:24.154002    tcp 1392    -> 443          23       17         3931 11578     FSPA_FSPA 1010202*        0
>> 14:43:26.428696    tcp 1230    -> 443          51       84         9501 106639     FSPA_FSPA 7095679*        0
>> 14:43:26.477505    tcp 1230    -> 443          51       83         9487 105205     FSPA_FSPA        0 6225909*
>> 14:43:26.882867    tcp 1269    -> 443          38       56         4866 66173     FSPA_FSPA        0 5309608*
>> 14:43:27.773332    tcp 1269    -> 443          38       56         4848 66173     FSPA_FSPA        0 3003745*
>> 14:43:29.768670    tcp 17938   -> 443          24       19         4758 11368     FSPA_FSPA        0 1790099*
>> 14:43:31.563272    tcp 6560    -> 443           8        7         1873 795     FSPA_FSPA        0 -214748*
>> 14:43:31.639145    tcp 6560    -> 443           8        7         1859 795     FSPA_FSPA        0 1069055*
>> 14:43:43.257623    tcp 1400    -> 443          14       11         3061 6262     FSPA_FSPA 1751030*        0
>> 14:43:45.376017    tcp 18232   -> 443          35       61         3373 79954     FSPA_FSPA        0 3313109*
>> 14:44:06.413124    tcp 64311   -> 80            5        5          934 3337       SPA_SPA        0 9516513*
>> 14:44:23.016703    tcp 29000   -> 80           10        8         1895 4521     FSPA_FSPA        0 4607406*
>> 14:44:38.375909    tcp 1044    -> 443          62       98         7551 117176    FSRPA_FSPA 1437146*        0
>> 14:44:48.359264    tcp 18271   -> 80            7        6         1065 4136     FSPA_FSPA        0 1440882*
>> 14:44:54.380581    tcp 18077   -> 80            9        7         2061 4374     FSPA_FSPA        0 2947889*
>> 14:44:54.380618    tcp 18116   -> 80            8        5         2020 752     FSPA_FSPA 5651196* 2945763*
>> 14:45:03.632559    tcp 17925   -> 443          89      186         9322 245670     FSPA_FSPA        0 1496437*
>> 14:45:04.040171    tcp 18043   -> 443          46       93         4751 123879     FSPA_FSPA -214748* 2366603*
>> 14:45:04.884425    tcp 18078   -> 443          44       93         4645 123879     FSPA_FSPA 9885776*        0
>> 14:45:04.912696    tcp 18078   -> 443          44       93         4631 123879     FSPA_FSPA        0 2502969*
>>
>>
>> Please see our private email discussion about this in the thread with subject "Pcap" and let me know if I should send you the pcap again.
>>
>> Your last thought some months ago was:
>> "OK, I think the idle timeout queues are not being timed out in your experiments, possibly because you aren't sending any packets between runs. Just a guess, but it's possible that the global clock is being used to time out the idle queues, which is updated by packet header time stamps. No packets, no change in global time.  Again that is just a guess."
>>
>>
>> note: See also the separate thread "Bugs in gap detection" with three other gap-related issues. Perhaps they are related...
>>
>> /Elof
>>
>>
>>
>>
>> On Wed, 29 Oct 2014, Carter Bullard wrote:
>>
>>> So the cache is still sitting around, (FINs for all reports after first close) and that can explain all the gap issues.  It doesn't affect flow reporting, just the packet dynamics metrics, like loss, gaps,  etc ...
>>>
>>> So I can't replicate here, so if you test a few more things tomorrow that would be very helpful !!!
>>>
>>> Sorry for the inconvenience !!!
>>> Carter
>>>
>>>> On Oct 28, 2014, at 12:25 PM, elof2 at sentor.se wrote:
>>>>
>>>>
>>>> Ok, removing -Zb from the commandline.
>>>>
>>>> # ra -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr out.log | cat -n
>>>>    1        StartTime      Flgs  Sport Dir  Dport  SrcPkts  DstPkts State   SrcGap   DstGap  Trans        Dur   SrcTCPBase   DstTCPBase
>>>>    2  10:44:32.046526  e        1087    -> 443           1        0 REQ        0        0      1   0.000000   1651739529
>>>>    3  10:44:32.046617  e        1087    -> 443           0        1 ACC        0        0      1   0.000000   1651739529   1545191373
>>>>    4  10:44:32.087006  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651739529   1545191373
>>>>    5  10:44:32.108056  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651739529   1545191373
>>>>    6  10:44:32.108137  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651739968   1545191373
>>>>    7  10:44:32.108628  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651739968   1545191373
>>>>    8  10:44:32.165988  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651739968   1545191532
>>>>    9  10:44:32.188005  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651739968   1545191532
>>>>   10  10:44:32.196067  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651740096   1545191532
>>>>   11  10:44:32.196145  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545191532
>>>>   12  10:44:32.252113  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545191532
>>>>   13  10:44:32.252126  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545192900
>>>>   14  10:44:32.252191  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545194268
>>>>   15  10:44:32.252193  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545195636
>>>>   16  10:44:32.252375  e        1087    -> 443           0        1 CON        0        0      1   0.000000   1651741098   1545197004
>>>> ...
>>>>  150  10:44:32.595137  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  151  10:44:32.595160  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  152  10:44:32.595171  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  153  10:44:32.595173  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  154  10:44:32.596473  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  155  10:44:32.596484  e        1087    -> 443           1        0 CON        0        0      1   0.000000   1651741098   1545322196
>>>>  156  10:44:33.606094  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  157  10:44:33.606267  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  158  10:50:31.330213  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  159  10:50:31.330304  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  160  10:50:31.370692  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  161  10:50:31.391743  e s      1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  162  10:50:31.391824  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  163  10:50:31.392315  e d      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  164  10:50:31.449675  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  165  10:50:31.471692  e s      1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  166  10:50:31.479754  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  167  10:50:31.479832  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  168  10:50:31.535800  e d      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  169  10:50:31.535813  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  170  10:50:31.535878  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  171  10:50:31.535880  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  172  10:50:31.536062  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  173  10:50:31.536065  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  174  10:50:31.536068  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  175  10:50:31.536070  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  176  10:50:31.536072  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  177  10:50:31.536249  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  178  10:50:31.609256  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  179  10:50:31.609266  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  180  10:50:31.609330  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  181  10:50:31.609341  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  182  10:50:31.609353  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  183  10:50:31.609536  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  184  10:50:31.609539  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  185  10:50:31.609541  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  186  10:50:31.609544  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  187  10:50:31.609546  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  188  10:50:31.630738  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>> ...
>>>>  292  10:50:31.792880  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  293  10:50:31.792882  e r      1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  294  10:50:31.793043  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  295  10:50:31.799943  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  296  10:50:31.799953  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  297  10:50:31.799964  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  298  10:50:31.849911  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  299  10:50:31.870814  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  300  10:50:31.870825  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  301  10:50:31.870836  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  302  10:50:31.870848  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  303  10:50:31.871776  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  304  10:50:31.871786  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  305  10:50:31.871788  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  306  10:50:31.878820  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  307  10:50:31.878843  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  308  10:50:31.878854  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  309  10:50:31.878856  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  310  10:50:31.880156  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  311  10:50:31.880167  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  312  10:50:32.889777  e        1087    -> 443           1        0 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>  313  10:50:32.889948  e        1087    -> 443           0        1 FIN        0        0      1   0.000000   1651741098   1545322196
>>>>
>>>> /Elof
>>>>
>>>>
>>>>> On Tue, 28 Oct 2014, Carter Bullard wrote:
>>>>>
>>>>> Sample ./support/Config/argus.conf file has lots of
>>>>> timer values in it.  Sorry it didn’t make it in the man page.
>>>>>
>>>>> So print out the state field.  It should indicate if this is
>>>>> INT or a CON state, which reflects whether the cache needed
>>>>> to be generated or whether it was in memory.
>>>>>
>>>>> Carter
>>>>>
>>>>> Print out the
>>>>>> On Oct 28, 2014, at 5:59 AM, elof2 at sentor.se wrote:
>>>>>>
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> 1)
>>>>>> ARGUS_TCP_TIMEOUT is undocumented. I can't find anything about it in argus.conf(5), but I assume its value is in seconds, so I set it to
>>>>>> ARGUS_TCP_TIMEOUT=10
>>>>>>
>>>>>> 2)
>>>>>> # killall argus
>>>>>> # <add ARGUS_TCP_TIMEOUT=10 to /etc/argus.conf>
>>>>>> # argus -S 0.0000001
>>>>>> # tcpreplay --stats=1 -i mon0 /root/no_gaps.pcap
>>>>>> <waited a few minutes>
>>>>>> So far, out.log looks identical to last days run.
>>>>>> Line 157, the last _FA, is not yet flushed to file.
>>>>>> # tcpreplay --stats=1 -i mon0 /root/no_gaps.pcap
>>>>>> <waited a few minutes>
>>>>>> Now line 157 appeared in ra together with the new flows, all lines except the last one (313).
>>>>>> # killall argus
>>>>>> Now line 313 appeared in ra.
>>>>>>
>>>>>> Everything looks the same. The new SYN in line 158 still show 1651741098 instead of the expected 1651739529.
>>>>>> So either the ARGUS_TCP_TIMEOUT=10 had no effect, or there's something else going on:
>>>>>>
>>>>>> ...
>>>>>> 155  10:44:32.596484  e        1087    -> 443           1        0 A_        0        0      1   0.000000   1651741098   1545322196
>>>>>> 156  10:44:33.606094  e        1087    -> 443           1        0 FA_        0        0      1   0.000000   1651741098   1545322196
>>>>>> 157  10:44:33.606267  e        1087    -> 443           0        1 _FA        0        0      1   0.000000   1651741098   1545322196
>>>>>> 158  10:50:31.330213  e        1087    -> 443           1        0 S_        0        0      1   0.000000   1651741098   1545322196
>>>>>> 159  10:50:31.330304  e        1087    -> 443           0        1 _SA        0        0      1   0.000000   1651741098   1545322196
>>>>>> 160  10:50:31.370692  e        1087    -> 443           1        0 A_        0        0      1   0.000000   1651741098   1545322196
>>>>>> ...
>>>>>>
>>>>>> /Elof
>>>>>>
>>>>>>
>>>>>>> On Mon, 27 Oct 2014, Carter Bullard wrote:
>>>>>>>
>>>>>>> So all of this suggests that the idle timeout queue isn’t being processed.
>>>>>>> Caches stay in the argus until they are timed out, and it looks like these
>>>>>>> multiple runs are finding the prior flow cache, and just reusing them.
>>>>>>>
>>>>>>> Can you comment out the ARGUS_TCP_TIMEOUT value in your argus.conf file,
>>>>>>> and set it to something like 10 seconds, and then rerun ???  Should do
>>>>>>> something different ???
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>>
>>>>>>>> On Oct 27, 2014, at 3:55 PM, elof2 at sentor.se wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Carter!
>>>>>>>>
>>>>>>>> Here's an update to my last email, now with tcpreplay and not only a test with 'argus -r'.
>>>>>>>>
>>>>>>>> See attached textfile for all details.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Summary:
>>>>>>>> we see:
>>>>>>>> 156  20:14:50.741605  e        1087    -> 443           1        0 FA_        0        0      1   0.000000   1651741098   1545322196
>>>>>>>> 157  20:14:50.741776  e        1087    -> 443           0        1 _FA        0        0      1   0.000000   1651741098   1545322196
>>>>>>>> 158  20:18:54.945126  e        1087    -> 443           1        0 S_        0        0      1   0.000000   1651741098   1545322196
>>>>>>>> 159  20:18:54.945217  e        1087    -> 443           0        1 _SA        0        0      1   0.000000   1651741098   1545322196
>>>>>>>> When the new flow is replayed at 20:18:54, its sequence numbers show a static 1651741098, so argus seems to be matching/reusing some old data already in memory instead of logging a new fresh flow with sequence number 1651739529.
>>>>>>>>
>>>>>>>> /Elof
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mon, 27 Oct 2014, Carter Bullard wrote:
>>>>>>>>>
>>>>>>>>> Another thing to try in your experiments, is to use a very short
>>>>>>>>> status interval.  This may show us which packet we’re unhappy with.
>>>>>>>>>
>>>>>>>>> argus -S 0.001 -r ………..
>>>>>>>>>
>>>>>>>>> This should generate maybe 156 flow records, but we’ll see something ???
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Oct 27, 2014, at 12:39 PM, elof2 at sentor.se wrote:
>>>>>>>>>>
>>>>>>>>>>> On Mon, 27 Oct 2014, Carter Bullard wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hmmm, a few things.
>>>>>>>>>>> Argus does not flush flow records because of the protocol state.
>>>>>>>>>>> This would be a very bad thing, especially in the presence of an
>>>>>>>>>>> attack that does weird things with the TCP flags.
>>>>>>>>>>
>>>>>>>>>> Ah, ok. I just assumed it flushed it after the FIN.
>>>>>>>>>>
>>>>>>>>>>> The default status interval is 5 seconds. Because you are reusing
>>>>>>>>>>> port numbers, we’ll just match all your packets into the same flow,
>>>>>>>>>>> until argus idle times out the cache.
>>>>>>>>>>
>>>>>>>>>> Ok.
>>>>>>>>>>
>>>>>>>>>>> The default idle time, where argus tosses a TCP cache is 60 seconds.
>>>>>>>>>>> You should be waiting 60 seconds between runs, or change the
>>>>>>>>>>> ARGUS_TCP_TIMEOUT to something shorter for your experiments.
>>>>>>>>>>
>>>>>>>>>> Ok, so when I wait more than 60 seconds, the pkts counters are almost correct, and the sequence number is always still showing the incorrect value:
>>>>>>>>>> 12:54:35.268526  e *      1087    -> 443          55      101 FSPA_FSPA        0        0      1   1.559693   1651741098   1545322196
>>>>>>>>>> 12:55:11.691944  e *      1087    -> 443          55      101 FSPA_FSPA        0        0      1   1.559695   1651741098   1545322196
>>>>>>>>>> 17:25:10.080434  e *      1087    -> 443          55      101 FSPA_FSPA        0        0      1   1.559707   1651741098   1545322196
>>>>>>>>>> 17:29:15.374957  e *      1087    -> 443          55      101 FSPA_FSPA        0        0      1   1.559699   1651741098   1545322196
>>>>>>>>>>
>>>>>>>>>> It says 55 spkts. It should be 56.
>>>>>>>>>> 1651741098 should be 1651739529.
>>>>>>>>>> 1545322196 should be 1545191373.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Can you share the flow records that generate the bad gap counts ???
>>>>>>>>>>> We calculate gaps in the client, based on whats in the flow record.
>>>>>>>>>>> Maybe a bug in that algorithm ????
>>>>>>>>>>
>>>>>>>>>> Yes, I'll filter out that particular flow, verify there's still crazy numbers in the new file, and send it to you.
>>>>>>>>>>
>>>>>>>>>> I'm on my way to the train home right now. I'll do it when I come home or tomorrow.
>>>>>>>>>>
>>>>>>>>>> /Elof
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> On Oct 27, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, 26 Oct 2014, Carter Bullard wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Idle field is not something you will want to use, in this case.
>>>>>>>>>>>>> That is intended for tools like ratop.1 so you can see how
>>>>>>>>>>>>> the idle time is growing as nothing is coming in .
>>>>>>>>>>>>
>>>>>>>>>>>> Ok. You asked "So what are the flow idle timeout values in your sensor(s)?", apparently you were looking for something else. :)
>>>>>>>>>>>>
>>>>>>>>>>>> I use this config:
>>>>>>>>>>>> # cat /etc/argus.conf
>>>>>>>>>>>> ARGUS_MONITOR_ID=1.2.3.4
>>>>>>>>>>>> ARGUS_INTERFACE=mon0
>>>>>>>>>>>> ARGUS_OUTPUT_FILE=/usr/foobar/log/out.log
>>>>>>>>>>>> ARGUS_DAEMON=yes
>>>>>>>>>>>> ARGUS_ACCESS_PORT=0
>>>>>>>>>>>> ARGUS_GENERATE_MAC_DATA=yes
>>>>>>>>>>>> ARGUS_CAPTURE_DATA_LEN=120
>>>>>>>>>>>> ARGUS_FILTER=""
>>>>>>>>>>>>
>>>>>>>>>>>> Everything else is using the defaults.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Since both flows are reporting the same number, my guess is that its algorithmic,
>>>>>>>>>>>>> but there is more to this.  So those flows are not the same set of packets, as
>>>>>>>>>>>>> the stime is off and the base sequence numbers are not the same ????
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, they are the same.
>>>>>>>>>>>> 'stime' differ since I replayed the pcap using tcpreplay to my test/lab sensor.
>>>>>>>>>>>> Why the sequence numbers differ I don't know. That is probably the main issue here, and the reason for the crazy gap numbers.
>>>>>>>>>>>>
>>>>>>>>>>>>> Can you print the duration ??? Print out the ‘trans’ field.
>>>>>>>>>>>>
>>>>>>>>>>>> On the live sensor that logged the crazy gap numbers (argus has been running for days and it has previously seen flows for these same IPs and ports, so there could possibly be garbage still present in memory when the following flow gets logged):
>>>>>>>>>>>>
>>>>>>>>>>>> ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr gaps.argus - tcp and port 1087
>>>>>>>>>>>> StartTime      Flgs  Sport Dir  Dport  SrcPkts  DstPkts State   SrcGap   DstGap  Trans        Dur   SrcTCPBase   DstTCPBase
>>>>>>>>>>>> 13:42:43.511666  e g      1087    -> 443          56      101 FSPA_FSPA 11274535 1767532*      1   1.599528   1640464994   1368438167
>>>>>>>>>>>> 13:42:43.511743  e g      1087    -> 443          56      101 FSPA_FSPA 11274535 1767532*      1   1.599467   1640464994   1368438167
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Same command on the new logfile where I replayed the pcap (freshly started argus daemon, so no garbage in memory):
>>>>>>>>>>>>
>>>>>>>>>>>> ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr out_tcpreplay_gaps_after_kill.log - tcp and port 1087
>>>>>>>>>>>> StartTime      Flgs  Sport Dir  Dport  SrcPkts  DstPkts State   SrcGap   DstGap  Trans        Dur   SrcTCPBase   DstTCPBase
>>>>>>>>>>>> 17:41:56.413382  e        1087    -> 443          56      101 FSPA_FSPA        0        0      1   1.631145   1651739529   1545191373
>>>>>>>>>>>> 17:41:56.413462  e        1087    -> 443          56      101 FSPA_FSPA        0        0      1   1.631081   1651739529   1545191373
>>>>>>>>>>>>
>>>>>>>>>>>> So, again, stime differ because this was a tcp-replay I did in the evening.
>>>>>>>>>>>>
>>>>>>>>>>>> The diff in duration could either be because of some old garbage numbers in argus memory, or more likely, it is simply the minor differences in time when the pcap was written to hdd plus minor differences in time when it was replayed on the network.
>>>>>>>>>>>>
>>>>>>>>>>>> The diff in sequence numbers however can't be explained. The problem must be here.
>>>>>>>>>>>> My guess is that when argus see reused IP+ports some garbage remain. (see the logs where I could see that ports are reused day after day)
>>>>>>>>>>>> OR
>>>>>>>>>>>> When argus see two almost identical flows happening at the same time, something is screwed up and tracked to the wrong flow.
>>>>>>>>>>>> Like when the SPAN is setup to mirror both the external link AND the DMZ-vlan.
>>>>>>>>>>>> Client x.x.x.x do a 3way handshake towards the NAT-IP y.y.y.y:443.
>>>>>>>>>>>> The NAT-box now do a 3way handshake of its own towards 10.10.10.10:443 (only dst-NAT, the src remain x.x.x.x).
>>>>>>>>>>>> The client send a request (PA) which is NATed into a new PA-packet on the DMZ.
>>>>>>>>>>>> Argus see both these PA-packets:
>>>>>>>>>>>> x.x.x.x:1087 <-> y.y.y.y:443
>>>>>>>>>>>> and
>>>>>>>>>>>> x.x.x.x:1087 <-> 10.10.10.10:443
>>>>>>>>>>>> The server ACKs this and sends data back, and so on.
>>>>>>>>>>>> The two sessions send data and they finally terminate with FIN.
>>>>>>>>>>>>
>>>>>>>>>>>> Looking at these two flows in the fresh argus daemon log, everything looks just fine.
>>>>>>>>>>>> Could something be left in memory after the flows terminate, so the next time they reuse the port numbers, we get crazy numbers?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I start a new argus daemon and replay the pcap with a single tcp flow a few times (at normal speed and sequentially):
>>>>>>>>>>>>
>>>>>>>>>>>> The pcap contain a single tcp connection. 56 packets from x.x.x.x and 101 response packets back from 10.10.10.10. (See the entire pcap below)
>>>>>>>>>>>>
>>>>>>>>>>>> # tcpreplay --preload-pcap --stats=1 -i mon0 no_gaps.pcap
>>>>>>>>>>>> I waited a few seconds and then ran it again, and again, and again...
>>>>>>>>>>>>
>>>>>>>>>>>> ra show no crazy gap numbers but it DOES show different sequence numbers the first run compared to the rest:
>>>>>>>>>>>>
>>>>>>>>>>>> # ra -Zb -s stime:9 flgs sport dir:3 dport spkts dpkts state:13 sgap dgap trans dur stcpb dtcpb -nr out.log -
>>>>>>>>>>>> StartTime      Flgs  Sport Dir  Dport  SrcPkts  DstPkts State   SrcGap   DstGap  Trans        Dur   SrcTCPBase   DstTCPBase
>>>>>>>>>>>> 12:02:15.837014  e *      1087    -> 443         112      202 FSPA_FSPA        0        0      1   4.726716   1651739529   1545191373
>>>>>>>>>>>> 12:02:21.147487  e *      1087    -> 443         166      302 FSPA_FSPA        0        0      1   4.717483   1651741098   1545322196
>>>>>>>>>>>> 12:02:26.874580  e *      1087    -> 443         135      274 FSPA_FSPA        0        0      1   4.996352   1651741098   1545322196
>>>>>>>>>>>> 12:02:31.928752  e *      1087    -> 443          94      134 FSPA_FSPA        0        0      1   5.069161   1651741098   1545322196
>>>>>>>>>>>> 12:02:37.055272  e *      1087    -> 443          52       98 FPA_FPA        0        0      1   1.440231   1651741098   1545322196
>>>>>>>>>>>> 12:03:18.478272  e *      1087    -> 443         174      307 FSPA_FSPA        0        0      1   4.948444   1651741098   1545322196
>>>>>>>>>>>> 12:03:23.556139  e *      1087    -> 443         202      399 FSPA_FSPA        0        0      1   4.989495   1651741098   1545322196
>>>>>>>>>>>> 12:03:28.595580  e *      1087    -> 443         182      303 FSPA_FSPA        0        0      1   4.829063   1651741098   1545322196
>>>>>>>>>>>> 12:03:34.434253  e *      1087    -> 443         176      308 FSPA_FSPA        0        0      1   4.988512   1651741098   1545322196
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>> So the first flow show different sequence numbers compared to the second one. This is strange since I replay the exact same tcp flow.
>>>>>>>>>>>> pcap seq: 1651739529 and 1545191373
>>>>>>>>>>>> So the first flow gets logged correctly while all the others are incorrect.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, all packet counters look strange! When the tcp connection in the replayed pcap is finished (FIN), argus don't seem to purge the flow from memory and start treating the new replayed flow as a new one. This next replay gets accumulated into the previous flow's counters.
>>>>>>>>>>>> All flows should have "56   101" packets, not "112      202", "166 302", "135      274", etc.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> What is your ARGUS_MAR_STAUS_INTERVAL ???
>>>>>>>>>>>>
>>>>>>>>>>>> Default (which I think is 5 min). See conf above.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> So are any of the packets out of order ???
>>>>>>>>>>>>
>>>>>>>>>>>> Wireshark doesn't complain and I've even manually checked all 56 and 101 packets in the flows. No out of order. Everything looks fine.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> And you aren’t aggregating any of these records, that could be cause some the problems ???
>>>>>>>>>>>>
>>>>>>>>>>>> Nope. I'm just running argus with the above config file and looking at the out-file with 'ra'.
>>>>>>>>>>>> Both daemon and client is 3.0.8.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> No way you can share a tcpdump of just one flows that has the gaps problem ???
>>>>>>>>>>>>
>>>>>>>>>>>> I don't have a pcap that generate the problem. The pcap I have generates correct values when replayed to a fresh argus daemon.
>>>>>>>>>>>> The only way to get a pcap that generate crazy numbers *might* be to capture *lots* of flows, perhaps over a day. Enough traffic to trigger the fault in argus... Then checking with ra if crazy numbers show up and terminate the tcpdump. This pcap would probaly be waaay bigger than my entire HDD, so I guess it' too hard to do. Plus the fact that this is possibly sensitive customer-data that I would need to anonymize before putting it on a FTP-server or simmilar.
>>>>>>>>>>>> Sorry.
>>>>>>>>>>>> (But see below for a txt version of my test-pcap with just one single flow)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> So this email has too many issues in it.  We can talk about each independently.
>>>>>>>>>>>>> Argus should not be holding onto flows, it is not state based.  And killing argus
>>>>>>>>>>>>> doesn’t cause argus to flush, it causes the kernel to flush its buffers to
>>>>>>>>>>>>> disk, because we close the file.  We don’t flush the output socket to disk, as its very
>>>>>>>>>>>>> expensive.
>>>>>>>>>>>>
>>>>>>>>>>>> Ah, ok.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The replayed file no_gaps.pcap contain these 56+101 packets. No more. All of them look sane. No out of order. The session is terminated with FIN:s:
>>>>>>>>>>>>
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [S], seq 1651739529, win 14600, options [mss 1380,sackOK,TS val 28452891 ecr 0,nop,wscale 6], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [S.], seq 1545191373, ack 1651739530, win 28960, options [mss 1460,sackOK,TS val 584372731 ecr 28452891,nop,wscale 7], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 28452897 ecr 584372731], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 1:439, ack 1, win 229, options [nop,nop,TS val 28452897 ecr 584372731], length 438
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 439, win 235, options [nop,nop,TS val 584372746 ecr 28452897], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 1:159, ack 439, win 235, options [nop,nop,TS val 584372746 ecr 28452897], length 158
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 439:567, ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 128
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [P.], seq 567:1569, ack 159, win 245, options [nop,nop,TS val 28452903 ecr 584372746], length 1002
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], ack 1569, win 259, options [nop,nop,TS val 584372768 ecr 28452903], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 159:1527, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 1527:2895, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 2895:4263, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 4263:5631, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 5631:6999, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 6999:8367, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 8367:9735, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 9735:11103, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 11103:12471, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 12471:13839, ack 1569, win 259, options [nop,nop,TS val 584372782 ecr 28452903], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 2895, win 336, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 5631, win 426, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 13839:15207, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 15207:16575, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 16575:17943, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 17943:19311, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 19311:20679, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 20679:22047, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 22047:23415, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 23415:24783, ack 1569, win 259, options [nop,nop,TS val 584372800 ecr 28452920], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 8367, win 517, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 11103, win 607, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 13839, win 698, options [nop,nop,TS val 28452920 ecr 584372782], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 24783:26151, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 26151:27519, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 27519:28887, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 28887:30255, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 30255:31623, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 31623:32991, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 32991:34359, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 34359:35727, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 35727:37095, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 37095:38463, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 38463:39831, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 39831:41199, ack 1569, win 259, options [nop,nop,TS val 584372806 ecr 28452920], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 41199:42567, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 42567:43935, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 43935:45303, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 16575, win 788, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 19311, win 879, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 45303:46671, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 46671:48039, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 48039:49407, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 49407:50775, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 50775:52143, ack 1569, win 259, options [nop,nop,TS val 584372820 ecr 28452927], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 22047, win 969, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 24783, win 1060, options [nop,nop,TS val 28452927 ecr 584372800], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 27519, win 1150, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 30255, win 1241, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 32991, win 1331, options [nop,nop,TS val 28452927 ecr 584372806], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 52143:53511, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 53511:54879, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 54879:56247, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 56247:57615, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 57615:58983, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 58983:60351, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 60351:61719, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 61719:63087, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 63087:64455, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 64455:65823, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 65823:67191, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 67191:68559, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 68559:69927, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 69927:71295, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 71295:72663, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 72663:74031, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 74031:75399, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 75399:76767, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 76767:78135, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 78135:79503, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452927], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 35727, win 1422, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 38463, win 1512, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 41199, win 1603, options [nop,nop,TS val 28452928 ecr 584372806], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 79503:80871, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 80871:82239, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 82239:83607, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 83607:84975, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 84975:86343, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 86343:87711, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 87711:89079, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 89079:90447, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 90447:91815, ack 1569, win 259, options [nop,nop,TS val 584372826 ecr 28452928], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 43935, win 1693, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 46671, win 1784, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 91815:93183, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 93183:94551, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 94551:95919, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 95919:97287, ack 1569, win 259, options [nop,nop,TS val 584372840 ecr 28452934], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 49407, win 1874, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 52143, win 1965, options [nop,nop,TS val 28452934 ecr 584372820], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 54879, win 2055, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 57615, win 2146, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 60351, win 2236, options [nop,nop,TS val 28452935 ecr 584372826], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 97287:98655, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 98655:100023, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 100023:101391, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 101391:102759, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452934], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 102759:104127, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 104127:105495, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 105495:106863, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 106863:108231, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 108231:109599, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 109599:110967, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452935], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 63087, win 2327, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 65823, win 2417, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 68559, win 2508, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 71295, win 2598, options [nop,nop,TS val 28452936 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 74031, win 2689, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 110967:112335, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 112335:113703, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 113703:115071, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 115071:116439, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 116439:117807, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 117807:119175, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 119175:120543, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 120543:121911, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452936], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 121911:123279, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 123279:124647, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 76767, win 2779, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 79503, win 2870, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 82239, win 2960, options [nop,nop,TS val 28452938 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 84975, win 3051, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 124647:126015, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 126015:127383, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 127383:128751, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [.], seq 128751:130119, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 1368
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [P.], seq 130119:130823, ack 1569, win 259, options [nop,nop,TS val 584372846 ecr 28452938], length 704
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 87711, win 3141, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 90447, win 3232, options [nop,nop,TS val 28452939 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 93183, win 3322, options [nop,nop,TS val 28452941 ecr 584372826], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 95919, win 3413, options [nop,nop,TS val 28452942 ecr 584372840], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 98655, win 3503, options [nop,nop,TS val 28452943 ecr 584372840], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 101391, win 3594, options [nop,nop,TS val 28452944 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 104127, win 3684, options [nop,nop,TS val 28452944 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 106863, win 3775, options [nop,nop,TS val 28452945 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 109599, win 3865, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 112335, win 3956, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 115071, win 4046, options [nop,nop,TS val 28452946 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 117807, win 4137, options [nop,nop,TS val 28452947 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 120543, win 4227, options [nop,nop,TS val 28452947 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 123279, win 4318, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 126015, win 4408, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 128751, win 4499, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130823, win 4589, options [nop,nop,TS val 28452948 ecr 584372846], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [F.], seq 1569, ack 130823, win 4589, options [nop,nop,TS val 28453048 ecr 584372846], length 0
>>>>>>>>>>>> IP 10.10.10.10.443 > x.x.x.x.1087: Flags [F.], seq 130823, ack 1570, win 259, options [nop,nop,TS val 584373120 ecr 28453048], length 0
>>>>>>>>>>>> IP x.x.x.x.1087 > 10.10.10.10.443: Flags [.], ack 130824, win 4589, options [nop,nop,TS val 28453053 ecr 584373120], length 0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /Elof
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> On Oct 23, 2014, at 12:16 PM, elof2 at sentor.se wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The crazy flow look like this with idle, stcpb and dtcpb:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ra -Zb -s stime:9 flgs saddr sport dir:3 daddr dport spkts dpkts state:13 sgap dgap idle:14 stcpb dtcpb -nr gaps.argus - tcp
>>>>>>>>>>>>>> 13:42:43.511666  e g          x.x.x.x.1087    ->     y.y.y.y.443          56      101     FSPA_FSPA 11274535 1767532*  1414077184.00*   1640464994   1368438167
>>>>>>>>>>>>>> 13:42:43.511743  e g          x.x.x.x.1087    -> 10.10.10.10.443          56      101     FSPA_FSPA 11274535 1767532*  1414077184.00*   1640464994   1368438167
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Apparently the network SPAN is setup to both mirror the outside vlan (before NAT:ing y.y.y.y to 10.10.10.10) and the inside vlan. So argus see both flows on its sniffer interface.
>>>>>>>>>>>>>> Both flows were caught in my pcap. There are no gaps in either of them. I see how x.x.x.x do a threeway handshake towards y.y.y.y. Then I see how the inside if the NAT-fw send its SYN to 10.10.10.10. The two sessions are happening at the same time, not sequentially.
>>>>>>>>>>>>>> Ra show the crazy gap values for both flows.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The same two flows looks like this when I replayed the full pcap (20 seconds of traffic):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 17:41:56.413382  e            x.x.x.x.1087    ->     y.y.y.y.443          56      101     FSPA_FSPA        0        0  1414079232.00*   1651739529   1545191373
>>>>>>>>>>>>>> 17:41:56.413462  e            x.x.x.x.1087    -> 10.10.10.10.443          56      101     FSPA_FSPA        0        0  1414079232.00*   1651739529   1545191373
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I restarted argus and replayed a filtered pcap which only contained one of the flows (all 157 packets).
>>>>>>>>>>>>>> out.log is only 128 bytes and growing 128 bytes every minute due to the MAR-status events.
>>>>>>>>>>>>>> I've waited more than 5 minutes and the single flow is still not flushed to the file.
>>>>>>>>>>>>>> I don't know if this is due to the fact that no more data at all is coming in on the sniffer interface, or if argus didn't realize that the connection has finished (the FIN packets are sent).
>>>>>>>>>>>>>> Anyhow, I kill the argus daemon to force it to flush the data into out.log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 17:52:04.561207  e            x.x.x.x.1087    -> 10.10.10.10.443          55      101     FSPA_FSPA        0        0  1414080256.00*   1651739529   1545191373
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So in both pcap cases, there are zero gaps reported, and the base sequence numbers are the same.
>>>>>>>>>>>>>> However, the sequence numbers do NOT match the ones from the long-running argus daemon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Side-step:
>>>>>>>>>>>>>> Regarding the non-flushing of the single flow to out.log. Is that a bug or work as intended?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /Elof
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, 23 Oct 2014, Carter Bullard wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So what are the flow idle timeout values in your sensor(s) ???
>>>>>>>>>>>>>>> Could it be really long, leaving older base sequence numbers
>>>>>>>>>>>>>>> around, and we’re getting port reuse ???  Or it could be we’re
>>>>>>>>>>>>>>> seeing sequence number rollover ???  What are the stcpb and
>>>>>>>>>>>>>>> dtcpb for the flows that have crazy numbers ???
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Oct 23, 2014, at 9:41 AM, elof2 at sentor.se wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Carter!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry, but I can't give you a pcap or argus-logfile since they contain sensitive data.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm also sorry to say that this will probably be hard to debug.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 'cause when I replayed the pcap on another sensor, ra showed "0   0"
>>>>>>>>>>>>>>>> gaps for this flow instead of "11274535 1767532*". This is correct, since I found no gaps in the flow in the pcap.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (the other flow I analysed show the correct "0     1367" since one packet is really missing in this flow in the pcap)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So a freshly started argus daemon seem to log correct values.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I found another sensor with lots of crazy numbers.
>>>>>>>>>>>>>>>> See attached logfile.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In there you can see that 1.2.3.4 is running a continous web-spider towards a wiki on 2.2.2.2:80.
>>>>>>>>>>>>>>>> All the GET requests always generates flows with approximately 11 packes in each direction.
>>>>>>>>>>>>>>>> There are zero gaps for days, and then suddenly there is a burst of crazy numbers during a period of 40 seconds. Then everything is good for two hours and then another 40 second burst.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> netstat -B show zero drops for the argus daemon.
>>>>>>>>>>>>>>>> My graphs for the cpu usage, swapping, memory usage, packets per second, etc show nothing out of the ordinary. The machine is not heavily loaded. Doesn't swap. It only receives 35 Mbps of mirrored traffic.
>>>>>>>>>>>>>>>> No spike or unusual activity during the 40 seconds of crazy numbers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I can't find any pattern or reason for the sudden burst of crazy numbers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Other traffic flows show "0   0" gaps during the crazy periods, so not all flows are affected (not even all flows between 1.2.3.4 and 2.2.2.2, but many of them).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /Elof
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, 22 Oct 2014, Carter Bullard wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you send the pcap file ??  Does argus generale the crazy
>>>>>>>>>>>>>>>>> numbers with this file ???
>>>>>>>>>>>>>>>>> Carter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Oct 22, 2014, at 9:39 AM, elof2 at sentor.se wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Carter!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> FYI, the gap detection counters still show some wonky numbers in 3.0.8.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ra -Zb -s flgs spkts dpkts state:13 sgap dgap -nr gaps.argus - tcp | grep g
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Normal and OK gaps show up like this:
>>>>>>>>>>>>>>>>>> Flgs  SrcPkts  DstPkts         State   SrcGap   DstGap
>>>>>>>>>>>>>>>>>> e g            15       24         PA_PA        0      458
>>>>>>>>>>>>>>>>>> e g             3        3         PA_PA        0      284
>>>>>>>>>>>>>>>>>> e g             5        6         PA_PA        0      732
>>>>>>>>>>>>>>>>>> e g           129      284         PA_PA        0     1367
>>>>>>>>>>>>>>>>>> e g            66       94         PA_PA        0     1367
>>>>>>>>>>>>>>>>>> e g             2        2         PA_PA        0      801
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ...but here and there I get lines like this:
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA 3051561* 8894044*
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA 1142343* 8891853*
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA 98000397 7385371*
>>>>>>>>>>>>>>>>>> e g             6        4     FSPA_FSPA 59208794 6255514*
>>>>>>>>>>>>>>>>>> e g            20       20     FSPA_FSPA 3468512*    65538
>>>>>>>>>>>>>>>>>> e g            20       20     FSPA_FSPA 3468512*    65538
>>>>>>>>>>>>>>>>>> e g             5        6       SPA_SPA 2562525* 9087142*
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA 68629719 5826434*
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA -214748* 5815425*
>>>>>>>>>>>>>>>>>> e g             7        6     FSPA_FSPA -214748* 1919818*
>>>>>>>>>>>>>>>>>> e g            17       24     FSPA_FSPA 3167486* 2765698*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This doesn't look as nice.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I tcpdump:ed traffic to pcap while argus created its logfile.
>>>>>>>>>>>>>>>>>> I analysed two flows showing gaps, one normal with "0     1367" gaps and one wonky with "11274535 1767532*" gaps.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Wireshark analysis of the "normal flow" show identical numbers as argus; 0 gaps from src and 1367 bytes (one packet) missing (in mid-stream) from dst.
>>>>>>>>>>>>>>>>>> Good.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Wireshark analysis of a "wonky" flow show no errors! No complaints at all from Wireshark (including its Expert Info). No "previous segment not captured" and no "ACKed unseen segment".
>>>>>>>>>>>>>>>>>> Everything looks good in the pcap.
>>>>>>>>>>>>>>>>>> I can't find any reason as to why argus create those wonky numbers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Oh, well, I don't use the gaps fields very often, so for me this is not important. I just thought I'd let you know.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> /Elof
>>>>>>>>>>>>>>>> <gaps.txt>
>>>>>>>> <replay_seqs.txt>
>
>


More information about the argus mailing list