man records in 3.0

Peter Van Epp vanepp at
Fri Sep 26 16:05:28 EDT 2008

On Tue, Sep 23, 2008 at 01:51:42PM -0400, Carter Bullard wrote:
> Hey Peter,
> I think the most important thing is to make the changes so the script is
> logical and intelligible.  So, if we are having direction problems, we
> really should fix them.  Between v2 and v3, I took out the cleverness
> that we had to 'correct' the direction, as it was slightly flawed (scans
> especially), and it hid the "ground truth" of the packets on the wire.
> The direction had been a complex set of conditions in the probe
> and in the client reading the data stream, so the work to clean that
> up should have worked.  If not, definitely need to fix it.

	Turns out I probably don't need version after all. I don't know what 
I was trying to fix in the scan code with the direction flags, but I don't
think it works quite correctly (although that said its been in production 
that way for at least 4 years since I last modified it in 2004 and it works
fine). It looks like you are correct the problem is actually 2.0.6 getting 
things inconsistant:

2.0.6 sensor and ra:

24 Sep 08 06:07:06           udp <-> 1        1         104          104         CON
24 Sep 08 06:27:11           udp <-> 1        1         104          104         CON

3.0 sensor and ra:

   06:07:06.841485  e          17    <->         2        208   CON
   06:27:11.982042  e          17    <->         2        208   CON

since neither is a priv port the 2.0.6 data doesn't get corrected and if that
was a scan, would be under reported. One option will be to keep history
and when we see a new flow in 2.0.6 see if there is a reversed flow already
present and invert one or the other (preferably marking the common port as
the dest port so we pick up scans.) Inversion due to src being a priv port 
seems to still be needed as I have lots of these (which should be the other
way) but that is easy:

   06:01:43.300635  e           6        ->         1         60   ACC
   06:01:43.328636  eI          6        ->         1         60   ACC

	As noted the traffic counts between between 2.0.6 and 3.0 are at least
close (and some of the errors are bad classification in 2.0.6 which isn't 
worth worrying about at this point) but the output of the scan code is still
radically different:

2.0.6 data proceessed by ra from 2.0.6:

Our hosts scanning:            885,148            533,536            633,724            382,360            481,860            475,970            429,803            416,725            375,487            276,707

same 2.0.6 data but processed by ra from 3.0:

Our hosts scanning:            885,946            533,890            635,793            382,357            481,906            476,003            429,846            416,743            375,490            276,708

close enough to identical for government work :-)

3.0 data from a 3.0 sensor on the same link at the same time for 24 hours 
through a regen tap and processed by ra3:

Our hosts scanning:            438,382             85,200            329,581            202,306            296,095             39,660            239,997            125,814            237,548            150,629

not very identical. Some of this (perhaps all of it) is 2.0.6 getting the
direction wrong and 3.0 getting it right. If I figure a way to correct the
2.0.6 data the issue may go away. The next step is probably to dump the raw
records for that one IP in 2.0.6 and 3.0 and see (manually) how they differ
and what needs to be done to get them to agree and go from there. 

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

More information about the argus mailing list