man records in 3.0

[Peter Van Epp]

On Tue, Sep 23, 2008 at 01:51:42PM -0400, Carter Bullard wrote:
> Hey Peter,
> I think the most important thing is to make the changes so the script is
> logical and intelligible.  So, if we are having direction problems, we
> really should fix them.  Between v2 and v3, I took out the cleverness
> that we had to 'correct' the direction, as it was slightly flawed (scans
> especially), and it hid the "ground truth" of the packets on the wire.
> The direction had been a complex set of conditions in the probe
> and in the client reading the data stream, so the work to clean that
> up should have worked.  If not, definitely need to fix it.
> 

	Turns out I probably don't need version after all. I don't know what 
I was trying to fix in the scan code with the direction flags, but I don't
think it works quite correctly (although that said its been in production 
that way for at least 4 years since I last modified it in 2004 and it works
fine). It looks like you are correct the problem is actually 2.0.6 getting 
things inconsistant:

2.0.6 sensor and ra:

24 Sep 08 06:07:06           udp  114.102.16.155.25150 <->   206.12.16.155.16881 1        1         104          104         CON
24 Sep 08 06:27:11           udp   206.12.16.154.16881 <->  114.102.16.155.25150 1        1         104          104         CON


3.0 sensor and ra:

   06:07:06.841485  e          17     114.102.16.155.25150    <->      206.12.16.155.16881         2        208   CON
   06:27:11.982042  e          17     114.102.16.155.25150    <->      206.12.16.154.16881         2        208   CON

since neither is a priv port the 2.0.6 data doesn't get corrected and if that
was a scan, would be under reported. One option will be to keep history
and when we see a new flow in 2.0.6 see if there is a reversed flow already
present and invert one or the other (preferably marking the common port as
the dest port so we pick up scans.) Inversion due to src being a priv port 
seems to still be needed as I have lots of these (which should be the other
way) but that is easy:

   06:01:43.300635  e           6      67.228.81.217.80        ->      142.58.212.39.47599         1         60   ACC
   06:01:43.328636  eI          6      67.228.81.217.80        ->      142.58.168.83.59321         1         60   ACC

	As noted the traffic counts between between 2.0.6 and 3.0 are at least
close (and some of the errors are bad classification in 2.0.6 which isn't 
worth worrying about at this point) but the output of the scan code is still
radically different:

2.0.6 data proceessed by ra from 2.0.6:

Our hosts scanning:

   206.12.16.154            885,148            533,536
   206.12.16.155            633,724            382,360
   142.58.186.69            481,860            475,970
  142.58.115.217            429,803            416,725
  142.58.117.141            375,487            276,707
...

same 2.0.6 data but processed by ra from 3.0:

Our hosts scanning:

   206.12.16.154            885,946            533,890
   206.12.16.155            635,793            382,357
   142.58.186.69            481,906            476,003
  142.58.115.217            429,846            416,743
  142.58.117.141            375,490            276,708

close enough to identical for government work :-)

3.0 data from a 3.0 sensor on the same link at the same time for 24 hours 
through a regen tap and processed by ra3:

Our hosts scanning:

   206.12.16.154            438,382             85,200
  142.58.117.141            329,581            202,306
   206.12.16.155            296,095             39,660
   142.58.199.44            239,997            125,814
    142.58.199.7            237,548            150,629

not very identical. Some of this (perhaps all of it) is 2.0.6 getting the
direction wrong and 3.0 getting it right. If I figure a way to correct the
2.0.6 data the issue may go away. The next step is probably to dump the raw
records for that one IP in 2.0.6 and 3.0 and see (manually) how they differ
and what needs to be done to get them to agree and go from there. 

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada